> ## Documentation Index
> Fetch the complete documentation index at: https://enrolla-nk-hub-guardrails.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Create custom evaluator

> A custom evaluator requires three main components: a prompt, an input schema, and an output schema.

**Prompt**

The prompt serves as an LLM-as-a-judge, instructing the model to evaluate your data against the criteria you define. Use Jinja2 templating syntax `{{var_name}}` to reference variables from your input schema.

**Input Schema**

Defines the variables available to your prompt template. Each variable declared here can be referenced in your prompt using `{{var_name}}`.

**Output Schema**

Defines the structure of the evaluation result. The model returns its assessment as structured output matching this schema.

Additionally, you'll specify an LLM provider, model, and any provider-specific settings.



## OpenAPI

````yaml post /v2/evaluators/custom-evaluator
openapi: 3.0.0
info:
  title: Traceloop API
  version: 1.0.0
  contact: {}
servers:
  - url: https://api.traceloop.com
security: []
paths:
  /v2/evaluators/custom-evaluator:
    post:
      tags:
        - evaluators
      summary: Create custom evaluator
      description: >-
        A custom evaluator requires three main components: a prompt, an input
        schema, and an output schema.


        **Prompt**


        The prompt serves as an LLM-as-a-judge, instructing the model to
        evaluate your data against the criteria you define. Use Jinja2
        templating syntax `{{var_name}}` to reference variables from your input
        schema.


        **Input Schema**


        Defines the variables available to your prompt template. Each variable
        declared here can be referenced in your prompt using `{{var_name}}`.


        **Output Schema**


        Defines the structure of the evaluation result. The model returns its
        assessment as structured output matching this schema.


        Additionally, you'll specify an LLM provider, model, and any
        provider-specific settings.
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/request.CreateCustomEvaluatorInput'
        description: Custom evaluator configuration
        required: true
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/response.CreateEvaluatorResponse'
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/response.ErrorResponse'
        '500':
          description: Internal Server Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/response.ErrorResponse'
      security:
        - BearerAuth: []
components:
  schemas:
    request.CreateCustomEvaluatorInput:
      properties:
        name:
          type: string
          description: Display name of the evaluator
        slug:
          type: string
          description: URL-safe identifier (auto-generated if not provided)
        description:
          type: string
          description: Description of what the evaluator does
        provider:
          type: string
          description: LLM provider (e.g., openai, anthropic)
        model:
          type: string
          description: Model to use for evaluation
        messages:
          type: array
          description: Prompt messages for the LLM judge
          items:
            $ref: '#/components/schemas/model.LLMMessage'
        input_schema:
          type: array
          description: Schema defining evaluator inputs
          items:
            $ref: '#/components/schemas/evaluator.Property'
        output_schema:
          type: array
          description: Schema defining evaluator outputs
          items:
            $ref: '#/components/schemas/evaluator.Property'
        temperature:
          type: number
          description: Temperature setting for the LLM
        top_p:
          type: number
          description: Top P (nucleus sampling) setting
        max_tokens:
          type: integer
          description: Maximum tokens in the response
        frequency_penalty:
          type: number
          description: Frequency penalty to reduce repetition
        presence_penalty:
          type: number
          description: Presence penalty for topic diversity
        stop:
          type: array
          description: Stop sequences
          items:
            type: string
      required:
        - name
        - provider
        - model
        - messages
        - input_schema
        - output_schema
      type: object
    response.CreateEvaluatorResponse:
      properties:
        evaluator_id:
          type: string
          description: Unique identifier for the created evaluator
        version_id:
          type: string
          description: Version identifier
        slug:
          type: string
          description: URL-safe slug for the evaluator
        type:
          type: string
          description: Type of the evaluator
      type: object
    response.ErrorResponse:
      description: Standard error response structure
      properties:
        error:
          example: error message
          type: string
      type: object
    model.LLMMessage:
      properties:
        role:
          type: string
          description: Message role (system, user, assistant)
        content:
          type: string
          description: Message content
      required:
        - role
        - content
      type: object
    evaluator.Property:
      properties:
        description:
          type: string
          description: Description of the property
        label:
          type: string
          description: Display label for the property
        name:
          type: string
          description: Property name (used as variable in templates)
        type:
          type: string
          description: 'Property type: string, int, float, boolean, or enum'
          enum:
            - string
            - int
            - float
            - boolean
            - enum
        enum_values:
          type: array
          description: Allowed values if type is enum
          items:
            type: string
      required:
        - name
        - type
      type: object
  securitySchemes:
    BearerAuth:
      description: Type "Bearer" followed by a space and JWT token.
      in: header
      name: Authorization
      type: apiKey

````