Routing Metadata

Routing metadata provides detailed insight into the router's decision-making process when selecting an endpoint from multiple eligible candidates.

To include routing metadata in your response, set include_metadata: true in your request body.

Python

from openai import OpenAI

client = OpenAI(
  base_url="https://cortecs.ai/api/v1/models/serverless",
  api_key="<API_KEY>",
)

completion = client.chat.completions.create(
  model="<MODEL_NAME>",
  messages=[
    {
      "role": "user",
      "content": "Tell me a joke."
    }
  ],
  extra_body={
    "include_metadata": True
  }
)

print(completion.metadata)

Node.js

import OpenAI from "openai";

const openai = new OpenAI({
    baseURL: 'https://cortecs.ai/api/v1/models/serverless',
    apiKey: '<API_KEY>'
});

const completion = await openai.chat.completions.create({
  model: "<MODEL_NAME>",
  messages: [
    {
      role: "user",
      content: "Tell me a joke.",
    }
  ],
  extra_body: {
    "include_metadata": true
  }
});

console.log(completion.metadata);

Curl

curl 'https://cortecs.ai/api/v1/models/serverless/chat/completions' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <API_KEY>' \
    -d '{
      "model": "<MODEL_NAME>",
      "messages": [
        { "role": "user", "content": "Tell me a joke." }
      ],
      "include_metadata": true
    }'

Note: If using streaming mode, the metadata is included with the final chunk of the response.

Metadata Structure

The metadata object contains the following sections:

routing_results

• selected_provider: The provider endpoint chosen by the router.

• applied_filters: A list of all filters applied (both implicit and explicit).

• cost:

  • prompt_cost: Cost associated with the prompt.

  • completion_cost: Cost associated with the completion.

  • total_cost: Total cost.

  • currency: Currency used for cost calculations.

• latency_ms: Latency in milliseconds from the selected provider.

routing_trace

• initial_candidates: List of all providers that support the requested model.

• filtered_out_candidates: Providers removed during filtering, each with:

  • provider: Name of the provider.

  • filter_failed: The name of the filter that failed.

  • reason: Explanation for the filter failure.

• scored_candidates: Providers that passed filtering, along with scoring details:

  • provider: Name of the provider.

  • score: The computed score.

  • scoring_context: Data used during scoring.

• scoring_function:

  • function: Name of the scoring function used.

  • sort: Indicates whether the router sorted by maximum or minimum score.