0.0.0+develop

flytekitplugins.inference.vllm.serve

Directory

Classes

Class Description
HFSecret .
VLLM

flytekitplugins.inference.vllm.serve.HFSecret

class HFSecret(
    secrets_prefix: str,
    hf_token_key: str,
    hf_token_group: typing.Optional[str],
)
Parameter Type
secrets_prefix str
hf_token_key str
hf_token_group typing.Optional[str]

flytekitplugins.inference.vllm.serve.VLLM

class VLLM(
    hf_secret: flytekitplugins.inference.vllm.serve.HFSecret,
    arg_dict: typing.Optional[dict],
    image: str,
    health_endpoint: str,
    port: int,
    cpu: int,
    gpu: int,
    mem: str,
)

Initialize NIM class for managing a Kubernetes pod template.

Parameter Type
hf_secret flytekitplugins.inference.vllm.serve.HFSecret
arg_dict typing.Optional[dict]
image str
health_endpoint str
port int
cpu int
gpu int
mem str

Methods

Method Description
build_vllm_args()
setup_vllm_pod_template()

build_vllm_args()

def build_vllm_args()

setup_vllm_pod_template()

def setup_vllm_pod_template()

Properties

Property Type Description
base_url
pod_template