Python client
Start, manage and stop instances
Cortecs-py is a ligthweight Python wrapper of our REST API. It provides you with the necessary tools to dynamically manage your instances directly from your workflow.
Setup
In order to use the API you need to create your access credentials in your profile page first. Before accessing the API make sure your environment variables are set:
OPENAI_API_KEY
-> use your cortecs api keyCORTECS_CLIENT_ID
CORTECS_CLIENT_SECRET
Methods Overview
The client helps you start, manage and stop your models.
Starts an instance.
An Instance
object*.
Restarts a stopped instance by the given instance_id
.
An Instance
object*.
If an instance with the same InstanceArgs
is running it returns that one; if it's stopped, it restarts it; otherwise it starts a new instance.
An Instance
object*.
Polls an instance until it is running.
An Instance
object.
Retrieves an Instance
by the instance_id
.
An Instance
object.
Retrieves only the InstanceStatus
by the instance_id
.
An InstanceStatus
object.
Retrieves a list of all instances (both running and stopped).
A list of Instance
objects.
Retrieves a list of all running instances.
A list of Instance
objects.
Stops an instance by its instance_id
.
Instance
object of the stopped instance.
Stops all running instances.
A list of Instance
objects of the stopped instances.
Deletes an instance by its instance_id
. The instance must first be stopped to be deleted.
The instance_id
of the deleted instance.
Deletes all instances. They must first be stopped to be deleted.
A list of instance_id
s of the deleted instances.
*If not using poll=True, the Instance object won't be complete. For more information visit the Objects page.
Additionally, the client can be used to retrieve information about models and hardware types.
get_all_models
Retrieve a list of all supported Model
s.
A list of Model
objects.
get_all_hardware_types
Retrieve a list of all supported HardwareType
s.
A list of HardwareType
objects.
get_available_hardware_types
Retrieve a list of the HardawareType
s which are currently available.
A list of HardwareType
objects.
Starting instances
The client offers several methods for starting an instance: start
, ensure_instance
, and restart
. Given that model startup times can take up to a few minutes (unless using Instant Provisioning), users have the option to wait for the instance to become ready by setting the poll
argument to True
. Alternatively, users can set the poll
argument to False
and use the poll_instance
method separately for more control.
start
start
Start an instance with the given instance arguments. It accepts the same arguments as ensure_instance
.
ensure_instance
ensure_instance
Checks if an instance with the same arguments is already running, in which case that one is returned. If there is an equivalent pending instance, that one is returned. If there is an equivalent stopped instance, it's restarted and returned. Otherwise, a new instance with the given arguments is started.
Both start
and ensure_instance
accept the following arguments:
model_name
: str
The model name (equivalent to HuggingFace name, eg. neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8'
).
Required
hardware_type_id
: str
The id of the HardwareType to use, eg.NVIDIA_L40S_1
.
The recommended hardware configuration
context_length
: int
The maximum context length the model should be initialized with. A larger context length slows down inference, so it's good practice to limit it according to your use case.
32k tokens or the maximum context length of the corresponding hardware configuration (if it is smaller than 32k)
billing_interval
: str
The interval in which the instance should be billed. Can be per_minute
or per_hour
.
per_minute
poll
: bool
If true
, the method will wait until the Instance
object is fully available. Otherwise it returns the partial Instance
object with some fields set to null.
True
restart
restart
Restart an instance that has already been started and stopped by its instance_id
.
instance_id
: str
The id of the instance.
Required
poll
: bool
If true
, the method will wait until the Instance
object is fully available. Otherwise it returns the partial Instance
object with some fields set to null.
True
poll_instance
poll_instance
Poll an instance until it is running.
instance_id
: str
The id of the instance.
Required
poll_interval
: int
The interval in seconds between each status check.
5
max_retries
: int
The maximum number of retries before raising an error.
150
Example
Managing instances
get_instance
get_instance
Get an instance by its id.
instance_id
: str
The id of the instance.
Required
get_instance_status
get_instance_status
Get the instance status by its id.
instance_id
: str
The id of the instance.
Required
get_all_instances
get_all_instances
Get all instances.
get_running_instances
get_running_instances
Get running instances.
Example
Stopping instances
Stopping an instance lets the user halt an instance as soon as a job is complete, avoiding additional costs while preserving the instance setup for future convenience.
stop
stop
Stop a specific instance by its id.
instance_id
: str
The id of the instance.
Required
stop_all
stop_all
Stop all running instances.
Example
Deleting instances
When a setup is no longer needed, the instance can be deleted from the user's console. Note that
delete
delete
Delete a stopped instance. Note that the instance must be stopped to be deleted, otherwise the method produces an error.
instance_id
: str
The id of the instance.
Required
delete_all
delete_all
Delete all instances.
force
: bool
If set to true
all instances will be deleted, regardless of their status. Otherwise, only stopped instances will be deleted.
False
Example
Last updated