Batch jobs

Everything All at Once

Dedicated inference is the best way to process massive workloads. It enables parallel processing without restricting you to rate limits.

Example

DedicatedLLM returns the ChatOpenAI object from LangChain. As LangChain supports batched inference with its Langchain Expression Language, it is easy to execute batch jobs using the batch(.) method.

with DedicatedLLM(client=cortecs, model_name='<MDOEL_NAME>') as llm:
    chain = ... | llm
    summaries = chain.batch([{...} for doc in docs])

This simple example showcases the power of dynamic provisioning. We summarized 224.2k input tokens into 12.9k output tokens in 55 seconds.

from langchain_community.document_loaders import ArxivLoader
from langchain_core.prompts import ChatPromptTemplate

from cortecs_py.client import Cortecs
from cortecs_py.integrations.langchain import DedicatedLLM

cortecs = Cortecs()
loader = ArxivLoader(
    query="reasoning",
    load_max_docs=40,
    get_ful_documents=True,
    doc_content_chars_max=25000,  # ~6.25k tokens, make sure the models supports that context length
    load_all_available_meta=False
)

prompt = ChatPromptTemplate.from_template("{text}\n\n Explain to me like I'm five:")
docs = loader.load()

with DedicatedLLM(client=cortecs, model_name='cortecs/phi-4-FP8-Dynamic') as llm:
    chain = prompt | llm

    print("Processing data batch-wise ...")
    summaries = chain.batch([{"text": doc.page_content} for doc in docs])
    for summary in summaries:
        print(summary.content + '-------\n\n\n')

The llm can be fully utilized in those 55 seconds enabling superior cost efficiency without rate limits.

Price Comparison (USD)