Objects

A detailed description of the objects used in the client library

Instance

The Instance class represents an instance managed by the library. It contains the instance's metadata, arguments, and status.

Attributes

instance_id (str)

• Description: A unique identifier for the instance.

• Example: instance-id-1234

---

base_url (str | None)

• Description: The base URL for accessing the instance. This may be None if an instance is not finished initializing.

• Example: https://my-base-url.eu/v1

---

instance_args (InstanceArgs)

• Description: The arguments with which the instance was initialized.

---

instance_status (InstanceStatus)

• Description: Contains information about the current status of the instance.

---

worker_statuses: (list[WorkerStatus])

• Description: A list of statuses of individual workers.

InstanceArgs

The InstanceArgs encapsulate the arguments of an instance.

Attributes

model_id (str):

• Description: The internal identifier for the model used in the instance. Equals the HuggingFace model name with the "/" replaced by "--".

• Example: "neuralmagic--Meta-Llama-3.1-8B-Instruct-FP8

---

hardware_type_id (str):

• Description: The identifier for the hardware type.

• Example: NVIDIA_L4_1

---

context_length (int):

• Description: The maximum context length supported by the hardware_type. If None, a default value applies.

• Example: 16000

---

billing_interval (str):

• Description: The billing interval for the instance.

• Example: per_minute or per_hour.

---

num_workers (int):

• Description: A worker is a processing unit within an instance. For example, each worker with hardware_type_id=NVIDIA_L40S_2 has 2 GPUs, so an instance with num_workers=2 contains 4 GPUs in total and is billed accordingly. Can be a number between 1 and 8.

• Example: 4

InstanceStatus

The InstanceStatus represents the current status and lifecycle information of an instance.

Attributes

status (str):

• Description: The current status of the instance.

• Example: "running", "pending" or "stopped"

---

started_at (datetime | None):

• Description: The timestamp when the instance was started. None if the instance hasn't been started yet (eg. it's still initializing).

• Example: 2024-11-29T09:05:47.788Z

---

stopped_at (datetime | None):

• Description: The timestamp when the instance was stopped. None if the instance is still running.

• Example: 2024-11-29T09:05:47.788Z

WorkerStatus

The WorkerStatus represents the status of an individual worker. As soon as at least one worker is running, the instance is usable and can be queried.

Attributes

init_progress (dict[str, Any] | None):

• Description: Progress information for worker initialization. Useful for tracking step-by-step initialization updates. None if the worker is stopped.

• Properties:

  • current_step (int)

    • Can be a number between 1 and 5.

  • num_steps (int)

    • Defines the total number of steps (5).

  • description (str)

    • A description of the current initialization step.

---

ModelPreview

The ModelPreviewobject provides basic information about a model, including its model_id, creator details, configuration options, etc.

Properties

model_id (str)

• Description: The unique internal identifier for the model.

• Example: "neuralmagic--Meta-Llama-3.1-8B-Instruct-FP8"

---

hf_name (str)

• Description: The Hugging Face model name corresponding to themodel_id with the '--' replaced by '/'.

• Example: "neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8"

---

instant_provisioned (bool)

• Description: Indicates whether the model is available for instant provisioning.

• Example: true

---

creator (dict[str, str])

• Description: A dictionary containing information about the model's creator. Keys include "name" and "url".

• Example:

  "creator": {
      "name": "Meta",
      "url": "https://ai.meta.com"
  }

---

description (str)

• Description: A brief description of the model.

• Example: "A state-of-the-art language model for natural language understanding."

---

screen_name (str)

• Description: An alternate display name for the model.

• Example: "Llama 3.1 8B"

---

size (int)

• Description: The size of the model, in parameter count.

• Example: 70600000000

---

hardware_configs (list[str])

• Description: A list of hardware type identifiers compatible with the model.

• Example: ["NVIDIA_L4_1", "NVIDIA_L40S_1", "NVIDIA_H100_1"]

---

HardwareType

The HardwareType object defines the specifications and pricing details for a hardware type available for instances. It includes information about the id, pricing, and technical specifications.

Properties

hardware_type_id (str)

• Description: A unique identifier for the hardware type.

• Example: "NVIDIA_L4_1"

---

price_info (dict)

• Description: Contains pricing details for the hardware type.

• Properties:

  • price_per_hour (float):

    • The hourly cost of using the hardware.

    • Example: 1.2

  • price_per_minute (float):

    • The per-minute cost of using the hardware.

    • Example: 0.02

  • currency(str)

    • The currency used for the pricing information.

    • Example: "EUR"

---

hardware_info (dict)

• Description: Provides technical details about the hardware.

• Properties:

  • gpu_type (str):

    • The type of GPU available in the hardware.

    • Example:"NVIDIA L4"

  • num_gpus (int):

    • The number of GPUs.

    • Example: 2

  • gpu_ram_gb (float):

    • The amount of GPU RAM, in gigabytes.

    • Example: 24