Quantization

Quantization reduces the numerical precision used by a model. Quantized endpoints can provide faster inference and lower costs, with a possible reduction in output quality.

Use the quantizations request parameter to restrict routing to endpoints that use an approved quantization method. The parameter acts as a whitelist: an endpoint is eligible only when its quantization matches one of the supplied values.

Quantization routing can currently be configured only through the API. It is not available in the Web Console.

Configure the whitelist

Add quantizations to an inference request as an array of accepted methods:

Python

import requests

api_key = "<CORTECS_API_KEY>"

response = requests.post(
    "https://api.cortecs.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    json={
        "model": "devstral-2512",
        "messages": [
            {
                "role": "user",
                "content": "Explain quantization in one paragraph.",
            }
        ],
        "quantizations": ["fp8", "bf16", "unknown"],
    },
)

response.raise_for_status()
result = response.json()

print(result["choices"][0]["message"]["content"])

Node.js

const apiKey = "<CORTECS_API_KEY>";

const response = await fetch("https://api.cortecs.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "devstral-2512",
    messages: [
      {
        role: "user",
        content: "Explain quantization in one paragraph.",
      },
    ],
    quantizations: ["fp8", "bf16", "unknown"],
  }),
});

if (!response.ok) {
  throw new Error(
    `Completion request failed: ${response.status} ${await response.text()}`,
  );
}

const result = await response.json();

console.log(result.choices[0].message.content);

Curl

curl 'https://api.cortecs.ai/v1/chat/completions' \
  -H 'Authorization: Bearer <CORTECS_API_KEY>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "devstral-2512",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantization in one paragraph."
      }
    ],
    "quantizations": ["fp8", "bf16", "unknown"]
  }'

In this example, the router considers FP8-family, BF16, and endpoints without a known quantization value.

The supported values are:

Value	Matching endpoint quantizations
int4	int4
int8	int8
fp4	fp4, mxfp4, nvfp4
mxfp4	mxfp4
nvfp4	nvfp4
fp6	fp6
fp8	fp8, mxfp8
mxfp8	mxfp8
fp16	fp16
bf16	bf16
fp32	fp32
unknown	Endpoints whose quantization is not specified

fp4 and fp8 are group values. They include their listed variants, while a specific variant such as mxfp4 matches only that variant.

If quantizations is omitted or set to null, the whitelist is not applied. Unsupported values do not match any endpoints.

allow_quantization is deprecated and remains available only for backward compatibility. Use quantizations for new integrations.

When to allow quantized endpoints

• Reduce latency for real-time or high-throughput workloads.

• Lower inference costs at scale.

• Accept a potential quality trade-off in exchange for performance gains.