> For the complete documentation index, see [llms.txt](https://docs.cortecs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cortecs.ai/examples/quantization.md).

# Quantization

Quantization reduces the numerical precision used by a model. Quantized endpoints can provide faster inference and lower costs, with a possible reduction in output quality.

Use the `quantizations` request parameter to restrict routing to endpoints that use an approved quantization method. The parameter acts as a whitelist: an endpoint is eligible only when its quantization matches one of the supplied values.

{% hint style="warning" %}
Quantization routing can currently be configured only through the API. It is not available in the Web Console.
{% endhint %}

## Configure the whitelist

Add `quantizations` to an inference request as an array of accepted methods:

{% tabs %}
{% tab title="Python" %}

```python
import requests

api_key = "<CORTECS_API_KEY>"

response = requests.post(
    "https://api.cortecs.ai/v1/chat/completions",
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    json={
        "model": "devstral-2512",
        "messages": [
            {
                "role": "user",
                "content": "Explain quantization in one paragraph.",
            }
        ],
        "quantizations": ["fp8", "bf16", "unknown"],
    },
)

response.raise_for_status()
result = response.json()

print(result["choices"][0]["message"]["content"])
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
const apiKey = "<CORTECS_API_KEY>";

const response = await fetch("https://api.cortecs.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiKey}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "devstral-2512",
    messages: [
      {
        role: "user",
        content: "Explain quantization in one paragraph.",
      },
    ],
    quantizations: ["fp8", "bf16", "unknown"],
  }),
});

if (!response.ok) {
  throw new Error(
    `Completion request failed: ${response.status} ${await response.text()}`,
  );
}

const result = await response.json();

console.log(result.choices[0].message.content);
```

{% endtab %}

{% tab title="Curl" %}

```bash
curl 'https://api.cortecs.ai/v1/chat/completions' \
  -H 'Authorization: Bearer <CORTECS_API_KEY>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "devstral-2512",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantization in one paragraph."
      }
    ],
    "quantizations": ["fp8", "bf16", "unknown"]
  }'
```

{% endtab %}
{% endtabs %}

In this example, the router considers FP8-family, BF16, and endpoints without a known quantization value.

The supported values are:

| Value     | Matching endpoint quantizations               |
| --------- | --------------------------------------------- |
| `int4`    | `int4`                                        |
| `int8`    | `int8`                                        |
| `fp4`     | `fp4`, `mxfp4`, `nvfp4`                       |
| `mxfp4`   | `mxfp4`                                       |
| `nvfp4`   | `nvfp4`                                       |
| `fp6`     | `fp6`                                         |
| `fp8`     | `fp8`, `mxfp8`                                |
| `mxfp8`   | `mxfp8`                                       |
| `fp16`    | `fp16`                                        |
| `bf16`    | `bf16`                                        |
| `fp32`    | `fp32`                                        |
| `unknown` | Endpoints whose quantization is not specified |

`fp4` and `fp8` are group values. They include their listed variants, while a specific variant such as `mxfp4` matches only that variant.

If `quantizations` is omitted or set to `null`, the whitelist is not applied. Unsupported values do not match any endpoints.

{% hint style="warning" %}
`allow_quantization` is deprecated and remains available only for backward compatibility. Use `quantizations` for new integrations.
{% endhint %}

## When to allow quantized endpoints

* Reduce latency for real-time or high-throughput workloads.
* Lower inference costs at scale.
* Accept a potential quality trade-off in exchange for performance gains.