Airflow agent example usage

Once you have a Union account, install union:

pip install union

Export the following environment variable to build and push images to your own container registry:

        
# replace with your registry name
export IMAGE_SPEC_REGISTRY="<your-container-registry>"

Then run the following commands to run the workflow:

        
    
git clone https://github.com/unionai/unionai-examples
cd unionai-examples
union run --remote tutorials/sentiment_classifier/sentiment_classifier.py main --model distilbert-base-uncased

The source code for this tutorial can be found here {octicon}mark-github.

Apache Airflow is a widely used open source platform for managing workflows with a robust ecosystem. Flyte provides an Airflow plugin that allows you to run Airflow tasks as Flyte tasks. This allows you to use the Airflow plugin ecosystem in conjunction with Flyte’s powerful task execution and orchestration capabilities.

        
    
from airflow.operators.bash import BashOperator
from airflow.sensors.filesystem import FileSensor
from flytekit import task, workflow


@task()
def t1():
    print("success")

Use the Airflow FileSensor to wait for a file to appear before running the task.

        
    
@workflow
def file_sensor():
    sensor = FileSensor(task_id="id", filepath="/tmp/1234")
    sensor >> t1()

Use the Airflow BashOperator to run a bash command.

        
    
@workflow
def bash_sensor():
    op = BashOperator(task_id="airflow_bash_operator", bash_command="echo hello")
    op >> t1()