Async FSSpec
| Name: | flytekitplugins-async-fsspec |
| Version: | 0.0.0+develop |
| Author: | admin@flyte.org |
| Provides: |
flytekitplugins.async_fsspec flytekitplugins.async_fsspec.s3fs |
| Requires: |
flytekit |
| Python: | >=3.9 |
| License: | apache2 |
| Source Code: | https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-async-fsspec |
- Intended Audience :: Science/Research
- Intended Audience :: Developers
- License :: OSI Approved :: Apache Software License
- Programming Language :: Python :: 3.9
- Programming Language :: Python :: 3.10
- Programming Language :: Python :: 3.11
- Topic :: Scientific/Engineering
- Topic :: Scientific/Engineering :: Artificial Intelligence
- Topic :: Software Development
- Topic :: Software Development :: Libraries
- Topic :: Software Development :: Libraries :: Python Modules
The Flyte async fsspec plugin is a powerful addition to the Flyte ecosystem designed to optimize the performance of object transmission. This plugin focuses on overriding key methods of the file systems in fsspec to introduce efficiency improvements, resulting in accelerated data transfers between Flyte workflows and object storage.
Currently, the async fsspec plugin improves the following file systems:
- s3fs
To install the plugin, run the following command:
pip install flytekitplugins-async-fsspecOnce installed, the plugin will automatically override the original file system and register optimized ones, seamlessly integrating with your Flyte workflows.