Class Judge

Judge implementation that handles evaluation functionality and conversation management.

According to the AIEval spec, judges are AI Configs with mode: "judge" that evaluate other AI Configs using structured output.

Hierarchy

  • Judge

Constructors

constructor

Properties

_aiConfig _aiConfigTracker _aiProvider _evaluationResponseStructure _logger?

Methods

_constructEvaluationMessages _getEvaluationMetricKey _interpolateMessage _parseEvaluationResponse evaluate evaluateMessages getAIConfig getProvider getTracker

Constructors

constructor

Properties

Private Readonly _aiConfig

_aiConfig: LDAIJudgeConfig

Private Readonly _aiConfigTracker

_aiConfigTracker: LDAIConfigTracker

Private Readonly _aiProvider

_aiProvider: AIProvider

Private Readonly _evaluationResponseStructure

_evaluationResponseStructure: Record<string, unknown>

Private Optional Readonly _logger

_logger?: LDLogger

Methods

Private _constructEvaluationMessages

  • _constructEvaluationMessages(input, output): LDMessage[]

  • Constructs evaluation messages by combining judge's config messages with input/output.

    Parameters

    • input: string
    • output: string

    Returns LDMessage[]

Private _getEvaluationMetricKey

  • _getEvaluationMetricKey(): undefined | string

  • Gets the evaluation metric key, prioritizing evaluationMetricKey over evaluationMetricKeys. Falls back to the first valid (non-empty, non-whitespace) value in evaluationMetricKeys if evaluationMetricKey is not provided. Treats empty strings and whitespace-only strings as invalid.

    Returns undefined | string

    The evaluation metric key, or undefined if not available

Private _interpolateMessage

  • _interpolateMessage(content, variables): string

  • Interpolates message content with variables using Mustache templating.

    Parameters

    • content: string
    • variables: Record<string, string>

    Returns string

Private _parseEvaluationResponse

  • _parseEvaluationResponse(data, evaluationMetricKey): Record<string, EvalScore>

  • Parses the structured evaluation response from the AI provider.

    Parameters

    • data: Record<string, unknown>
    • evaluationMetricKey: string

    Returns Record<string, EvalScore>

evaluate

  • Evaluates an AI response using the judge's configuration.

    Parameters

    • input: string

      The input prompt or question that was provided to the AI

    • output: string

      The AI-generated response to be evaluated

    • samplingRate: number = 1

      Sampling rate (0-1) to determine if evaluation should be processed (defaults to 1)

    Returns Promise<undefined | JudgeResponse>

    Promise that resolves to evaluation results or undefined if not sampled

evaluateMessages

  • Evaluates an AI response from chat messages and response.

    Parameters

    • messages: LDMessage[]

      Array of messages representing the conversation history

    • response: ChatResponse

      The AI response to be evaluated

    • samplingRatio: number = 1

      Sampling ratio (0-1) to determine if evaluation should be processed (defaults to 1)

    Returns Promise<undefined | JudgeResponse>

    Promise that resolves to evaluation results or undefined if not sampled

getAIConfig

getProvider

getTracker

Settings

Member Visibility

Theme

On This Page

Generated using TypeDoc