cortecs.aiModels

Objects

A detailed description of the objects used in the client library

Instance

The Instance class represents an instance managed by the library. It contains the instance's metadata, arguments, and status, while also providing a utility method to facilitate integration with the ChatOpenAI wrapper from LangChain.

Attributes

instance_id (str)

  • Description: A unique identifier for the instance.

  • Example: instance-id-1234

base_url (str | None)

  • Description: The base URL for accessing the instance. This may be None if an instance is not finished initializing.

  • Example: https://my-base-url.eu/v1

instance_args (InstanceArgs)

  • Description: The arguments with which the instance was initialized.

instance_status (InstanceStatus)

  • Description: Contains information about the current status of the instance.

Methods

chat_openai_config(**kwargs: dict[str, Any]) -> dict[str, Any]

Generates a dictionary of arguments for initializing a ChatOpenAI client. This method includes both the instance's predefined configuration and any additional arguments provided.

  • Parameters: **kwargs (dict[str, Any]): Optional key-value pairs to include in the returned configuration (eg. temperature).

  • Returns: A dictionary containing the following keys:

    • "model_name": The model name, equivalent to the HuggingFace model name or the model_id with "--" replaced by "/".

    • "base_url": The base URL of the instance.

    • Additional key-value pairs passed via **kwargs.

InstanceArgs

The InstanceArgs encapsulate the arguments of an instance.

Attributes

model_id (str):

  • Description: The internal identifier for the model used in the instance. Equals the HuggingFace model name with the "/" replaced by "--".

  • Example: "neuralmagic--Meta-Llama-3.1-8B-Instruct-FP8

hardware_type_id (str):

  • Description: The identifier for the hardware type.

  • Example: NVIDIA_L4_1

context_length (int):

  • Description: The maximum context length supported by the hardware_type. If None, a default value applies.

  • Example: 16000

billing_interval (str):

  • Description: The billing interval for the instance.

  • Example: per_minute or per_hour.

InstanceStatus

The InstanceStatus represents the current status and lifecycle information of an instance.

Attributes

status (str):

  • Description: The current status of the instance.

  • Example: "running", "pending" or "stopped"

started_at (datetime | None):

  • Description: The timestamp when the instance was started. None if the instance hasn't been started yet (eg. it's still initializing).

  • Example: 2024-11-29T09:05:47.788Z

stopped_at (datetime | None):

  • Description: The timestamp when the instance was stopped. None if the instance is still running.

  • Example: 2024-11-29T09:05:47.788Z

init_progress (dict[str, Any] | None):

  • Description: Progress information for instance initialization. Useful for tracking step-by-step initialization updates. None if the instance is stopped.

  • Properties:

    • current_step (int)

      • Can be a number between 1 and 5.

    • num_steps (int)

      • Defines the total number of steps (5).

    • description (str)

      • A description of the current initialization step.

ModelPreview

The ModelPreviewobject provides basic information about a model, including its model_id, creator details, configuration options, etc.

Properties

model_id (str)

  • Description: The unique internal identifier for the model.

  • Example: "neuralmagic--Meta-Llama-3.1-8B-Instruct-FP8"

hf_name (str)

  • Description: The Hugging Face model name corresponding to themodel_id with the '--' replaced by '/'.

  • Example: "neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8"

instant_provisioned (bool)

  • Description: Indicates whether the model is available for instant provisioning.

  • Example: true

creator (dict[str, str])

  • Description: A dictionary containing information about the model's creator. Keys include "name" and "url".

  • Example:

    "creator": {
    "name": "Meta",
    "url": "https://ai.meta.com"
}

description (str)

  • Description: A brief description of the model.

  • Example: "A state-of-the-art language model for natural language understanding."

screen_name (str)

  • Description: An alternate display name for the model.

  • Example: "Llama 3.1 8B"

size (int)

  • Description: The size of the model, in parameter count.

  • Example: 70600000000

hardware_configs (list[str])

  • Description: A list of hardware type identifiers compatible with the model.

  • Example: ["NVIDIA_L4_1", "NVIDIA_L40S_1", "NVIDIA_H100_1"]

HardwareType

The HardwareType object defines the specifications and pricing details for a hardware type available for instances. It includes information about the id, pricing, and technical specifications.

Properties

hardware_type_id (str)

  • Description: A unique identifier for the hardware type.

  • Example: "NVIDIA_L4_1"

price_info (dict)

  • Description: Contains pricing details for the hardware type.

  • Properties:

    • price_per_hour (float):

      • The hourly cost of using the hardware.

      • Example: 1.2

    • price_per_minute (float):

      • The per-minute cost of using the hardware.

      • Example: 0.02

    • currency(str)

      • The currency used for the pricing information.

      • Example: "EUR"

hardware_info (dict)

  • Description: Provides technical details about the hardware.

  • Properties:

    • gpu_type (str):

      • The type of GPU available in the hardware.

      • Example:"NVIDIA L4"

    • num_gpus (int):

      • The number of GPUs.

      • Example: 2

    • gpu_ram_gb (float):

      • The amount of GPU RAM, in gigabytes.

      • Example: 24

PreviousPython clientNextIntegrations

Last updated