Apache Airflow

Name:flytekitplugins-airflow
Version:0.0.0+develop
Author:admin@flyte.org
Provides: flytekitplugins.airflow
Requires: apache-airflow
apache-airflow-providers-google<12.0.0
flytekit>1.10.7
flyteidl>1.10.7
Python:>=3.9
License:apache2
Source Code: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-airflow
  • Intended Audience :: Science/Research
  • Intended Audience :: Developers
  • License :: OSI Approved :: Apache Software License
  • Programming Language :: Python :: 3.9
  • Programming Language :: Python :: 3.10
  • Programming Language :: Python :: 3.11
  • Topic :: Scientific/Engineering
  • Topic :: Scientific/Engineering :: Artificial Intelligence
  • Topic :: Software Development
  • Topic :: Software Development :: Libraries
  • Topic :: Software Development :: Libraries :: Python Modules

Airflow plugin allows you to seamlessly run Airflow tasks in the Flyte workflow without changing any code.

  • Compile Airflow tasks to Flyte tasks
  • Use Airflow sensors/operators in Flyte workflows
  • Add support running Airflow tasks locally without running a cluster

Example

from airflow.sensors.filesystem import FileSensor
from flytekit import task, workflow

@task()
def t1():
    print("flyte")


@workflow
def wf():
    sensor = FileSensor(task_id="id", filepath="/tmp/1234")
    sensor >> t1()


if __name__ == '__main__':
    wf()

To install the plugin, run the following command:

pip install flytekitplugins-airflow