Dask
| Name: | flytekitplugins-dask |
| Version: | 0.0.0+develop |
| Author: | admin@flyte.org |
| Provides: |
flytekitplugins.dask |
| Requires: |
flyteidl>=1.3.2 flytekit>=1.3.0b2,<2.0.0 dask[distributed]>=2022.10.2 |
| Python: | >=3.9 |
| License: | apache2 |
| Source Code: | https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-dask |
- Intended Audience :: Science/Research
- Intended Audience :: Developers
- License :: OSI Approved :: Apache Software License
- Programming Language :: Python :: 3.9
- Programming Language :: Python :: 3.10
- Topic :: Scientific/Engineering
- Topic :: Scientific/Engineering :: Artificial Intelligence
- Topic :: Software Development
- Topic :: Software Development :: Libraries
- Topic :: Software Development :: Libraries :: Python Modules
Flyte can execute dask jobs natively on a Kubernetes Cluster, which manages the virtual dask cluster’s lifecycle
(spin-up and tear down). It leverages the open-source Kubernetes Dask Operator and can be enabled without signing up
for any service. This is like running a transient (ephemeral) dask cluster - a type of cluster spun up for a specific
task and torn down after completion. This helps in making sure that the Python environment is the same on the job-runner
(driver), scheduler and the workers.
To install the plugin, run the following command:
pip install flytekitplugins-daskTo configure Dask in the Flyte deployment’s backed, follow these directionsernetes/k8s_dask/index.html#step-2-environment-setup)
An usage example can be found in the documentation.