flytekitplugins.spark.agent

Directory

Classes

Methods

Variables

Methods

get_header()

def get_header()

result_state_is_available()

def result_state_is_available(
    life_cycle_state: str,
) -> bool

flytekitplugins.spark.agent.DatabricksAgent

This is the base class for all async agents. It defines the interface that all agents must implement. The agent service is responsible for invoking agents. The propeller will communicate with the agent service to create tasks, get the status of tasks, and delete tasks.

All the agents should be registered in the AgentRegistry. Agent Service will look up the agent based on the task type. Every task type can only have one agent.

def DatabricksAgent()

Methods

create()

def create(
    task_template: flytekit.models.task.TaskTemplate,
    inputs: typing.Optional[flytekit.models.literals.LiteralMap],
    kwargs,
) -> flytekitplugins.spark.agent.DatabricksJobMetadata

Return a resource meta that can be used to get the status of the task.

delete()

def delete(
    resource_meta: flytekitplugins.spark.agent.DatabricksJobMetadata,
    kwargs,
)

Delete the task. This call should be idempotent. It should raise an error if fails to delete the task.

get()

def get(
    resource_meta: flytekitplugins.spark.agent.DatabricksJobMetadata,
    kwargs,
) -> flytekit.extend.backend.base_agent.Resource

Return the status of the task, and return the outputs in some cases. For example, bigquery job can’t write the structured dataset to the output location, so it returns the output literals to the propeller, and the propeller will write the structured dataset to the blob store.

Properties

flytekitplugins.spark.agent.DatabricksAgentV2

Add DatabricksAgentV2 to support running the k8s spark and databricks spark together in the same workflow. This is necessary because one task type can only be handled by a single backend plugin.

spark -> k8s spark plugin databricks -> databricks agent

def DatabricksAgentV2()

Methods

create()

def create(
    task_template: flytekit.models.task.TaskTemplate,
    inputs: typing.Optional[flytekit.models.literals.LiteralMap],
    kwargs,
) -> flytekitplugins.spark.agent.DatabricksJobMetadata

Return a resource meta that can be used to get the status of the task.

delete()

def delete(
    resource_meta: flytekitplugins.spark.agent.DatabricksJobMetadata,
    kwargs,
)

Delete the task. This call should be idempotent. It should raise an error if fails to delete the task.

get()

def get(
    resource_meta: flytekitplugins.spark.agent.DatabricksJobMetadata,
    kwargs,
) -> flytekit.extend.backend.base_agent.Resource

Return the status of the task, and return the outputs in some cases. For example, bigquery job can’t write the structured dataset to the output location, so it returns the output literals to the propeller, and the propeller will write the structured dataset to the blob store.

Properties

flytekitplugins.spark.agent.DatabricksJobMetadata

class DatabricksJobMetadata(
    databricks_instance: str,
    run_id: str,
)

Methods

decode()

def decode(
    data: bytes,
) -> ResourceMeta

Decode the resource meta from bytes.

encode()

def encode()

Encode the resource meta to bytes.