API reference

Reference for the Evaluation SDK's core types and the main evaluation.run() entry point. For the method-level reference of the advanced operations, see Retry failed records, Rescore persisted runs, and Optimize prompts and parameters.

The main entry point for running evaluations.

run = await client.evaluation.run(
    dataset=...,
    task=...,
    evaluators=...,
    # optional parameters below
)

run = await client.evaluation.run(
    dataset=...,
    task=...,
    evaluators=...,
    # optional parameters below
)

It returns an EvaluationRun:

run() marks the run "running" while it executes, then "completed". If the run raises, it's marked "failed"; if it's interrupted, "cancelled". While the run executes, the SDK sends a heartbeat to Studio, so a run whose process dies is marked "cancelled" instead of staying in progress.

A project groups related evaluations together. Think of it as a folder — for example, "Chatbot QA" or "RAG Pipeline".

Project(name="My Project")       # create or get by name
Project(slug="my-project")       # get by slug

Project(name="My Project")       # create or get by name
Project(slug="my-project")       # get by slug

At least one of name or slug must be provided. A project selected by name is created if it doesn't exist; a project selected by slug must already exist.

Selects a Studio dataset by slug or UUID. The SDK fetches its records before the run starts. See Use a Studio dataset.

from mistralai.evaluations import Dataset

Dataset(slug="customer-support-golden-set")
Dataset(id="018f879d-20cd-7e9f-a1bc-2f4f08b6f170")

from mistralai.evaluations import Dataset

Dataset(slug="customer-support-golden-set")
Dataset(id="018f879d-20cd-7e9f-a1bc-2f4f08b6f170")

Exactly one of id and slug must be provided, and id must be a valid UUID. The dataset must already exist: selecting it by slug never creates it.

An evaluation is a named test you run repeatedly over time — for example, "Accuracy on French prompts". Each call to evaluation.run() creates a new run under it, so you can track how scores evolve across runs.

Evaluation(name="My Eval")       # create or get by name
Evaluation(slug="my-eval")       # get by slug

Evaluation(name="My Eval")       # create or get by name
Evaluation(slug="my-eval")       # get by slug

At least one of name or slug must be provided. An evaluation selected by name is created under the given project if it doesn't exist; an evaluation selected by slug must already exist.

An evaluator defines how to score each individual record. It pairs a name with a scorer function that receives a ScorerContext and returns a score.

Evaluator(
    name="accuracy",
    scorer=my_scorer,
    description="Optional description",
    tags=["tag1"],
    num_scores=1,
    goal=Goal.gte(0.8),
)

Evaluator(
    name="accuracy",
    scorer=my_scorer,
    description="Optional description",
    tags=["tag1"],
    num_scores=1,
    goal=Goal.gte(0.8),
)

A scorer can return:

• int or float — numeric score (statistics: avg, min, max, std, count).
• str or bool — categorical score (statistics: frequencies, mode).
• Score(value=..., rationale=..., metadata=...) — rich score with an explanation and extra data.

A run evaluator operates on the full set of results after all records have been processed. Use it for aggregate metrics that can't be expressed per-record — like F1 score, global pass/fail gates, or cross-record analysis.

RunEvaluator(
    name="accuracy_gate",
    scorer=my_run_scorer,
)

RunEvaluator(
    name="accuracy_gate",
    scorer=my_run_scorer,
)

The scorer receives a RunEvaluatorContext with access to all records, their scores, aggregate statistics, and system config. See the run evaluators guide for examples.

Factory for creating goal specifications. See Set goals for the full guide.

from mistralai.evaluations import Goal

Goal.gte(0.8)             # gate: score >= 0.8
Goal.lte(0.1)             # gate: score <= 0.1
Goal.between(0.2, 0.8)    # gate: 0.2 <= score <= 0.8

from mistralai.evaluations import Goal

Goal.gte(0.8)             # gate: score >= 0.8
Goal.lte(0.1)             # gate: score <= 0.1
Goal.between(0.2, 0.8)    # gate: 0.2 <= score <= 0.8

A Goal is only a gate. To declare which way is better, set direction="maximize" / "minimize" on the evaluator (it is also inferred from a gte/lte goal).

A system captures the configuration that drives your task — model name, temperature, system prompt, tool definitions, and so on. Storing these as params (instead of hardcoding them) makes them visible in Studio so you can compare runs across different configs. See Configure system params for details.

from mistralai.evaluations import System

system = System(name="small-t0", params={"model": "mistral-small-latest", "temperature": 0})

from mistralai.evaluations import System

system = System(name="small-t0", params={"model": "mistral-small-latest", "temperature": 0})

When system is provided, it is available via ctx.system in tasks and scorers:

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        temperature=float(ctx.system.params["temperature"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        temperature=float(ctx.system.params["temperature"]),
        messages=[{"role": "user", "content": ctx.input_record["prompt"]}],
    )
    return str(response.choices[0].message.content)

See Use context objects for the full context reference.