Class: LaunchDarkly::Server::AI::AIConfigTracker

Inherits:
Object
  • Object
show all
Defined in:
lib/server/ai/ai_config_tracker.rb

Overview

The AIConfigTracker records metrics for a single AI run. Unless otherwise noted, the tracker's methods are not safe for concurrent use.

All events a tracker emits share a runId (a UUIDv4) so LaunchDarkly can correlate them in metrics views. See individual track methods for their specific semantics. Call create_tracker on the AI Config to start a new run. A resumption token preserves the runId, so events emitted by a tracker reconstructed in another process share the original runId.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(ld_client:, run_id:, config_key:, variation_key:, version:, context:, model_name:, provider_name:, model_key: nil, model_version: 1) ⇒ AIConfigTracker

Initialize a new AIConfigTracker instance.

Parameters:

  • ld_client (LDClient) —

    The LaunchDarkly client instance

  • variation_key (String) —

    The variation key from the flag evaluation

  • config_key (String) —

    The configuration key

  • version (Integer) —

    The version number

  • model_name (String) —

    The name of the AI model being used

  • provider_name (String) —

    The name of the AI provider

  • model_key (String, nil) (defaults to: nil) —

    The stable, unique key of the model used

  • model_version (Integer) (defaults to: 1) —

    The pinned version of the model used

  • context (LDContext) —

    The context used for the flag evaluation



70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/server/ai/ai_config_tracker.rb', line 70

def initialize(ld_client:, run_id:, config_key:, variation_key:, version:, context:, model_name:, provider_name:,
               model_key: nil, model_version: 1)
  @ld_client = ld_client
  @variation_key = variation_key
  @config_key = config_key
  @version = version
  @model_name = model_name
  @provider_name = provider_name
  @model_key = model_key
  @model_version = model_version
  @context = context
  @summary = MetricSummary.new
  @run_id = run_id
  @logger = LaunchDarkly::Server::AI.default_logger
end

Instance Attribute Details

#config_key ⇒ Object (readonly)

Returns the value of attribute config_key.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def config_key
  @config_key
end

#context ⇒ Object (readonly)

Returns the value of attribute context.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def context
  @context
end

#ld_client ⇒ Object (readonly)

Returns the value of attribute ld_client.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def ld_client
  @ld_client
end

#model_key ⇒ Object (readonly)

Returns the value of attribute model_key.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def model_key
  @model_key
end

#model_name ⇒ Object (readonly)

Returns the value of attribute model_name.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def model_name
  @model_name
end

#model_version ⇒ Object (readonly)

Returns the value of attribute model_version.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def model_version
  @model_version
end

#provider_name ⇒ Object (readonly)

Returns the value of attribute provider_name.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def provider_name
  @provider_name
end

#summary ⇒ Object (readonly)

Returns the value of attribute summary.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def summary
  @summary
end

#variation_key ⇒ Object (readonly)

Returns the value of attribute variation_key.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def variation_key
  @variation_key
end

#version ⇒ Object (readonly)

Returns the value of attribute version.



54
55
56
 
# File 'lib/server/ai/ai_config_tracker.rb', line 54

def version
  @version
end

Class Method Details

.from_resumption_token(token:, ld_client:, context:) ⇒ AIConfigTracker

Reconstructs a tracker from a resumption token.

Parameters:

  • token (String) —

    A URL-safe Base64-encoded JSON resumption token

  • ld_client (LDClient) —

    The LaunchDarkly client instance

  • context (LDContext) —

    The context for track events

Returns:



110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
 
# File 'lib/server/ai/ai_config_tracker.rb', line 110

def self.from_resumption_token(token:, ld_client:, context:)
  json = Base64.urlsafe_decode64(token)
  payload = JSON.parse(json)

  new(
    ld_client: ld_client,
    run_id: payload['runId'],
    config_key: payload['configKey'],
    variation_key: payload.fetch('variationKey', ''),
    version: payload['version'],
    context: context,
    model_name: '',
    provider_name: ''
  )
end

Instance Method Details

#resumption_token ⇒ String

Returns a URL-safe Base64-encoded JSON token that can be used to reconstruct a tracker in a different process (e.g. for deferred feedback).

The token contains: runId, configKey, variationKey, version. modelName, providerName, modelKey, and modelVersion are NOT included.

Returns:

  • (String) —

    the resumption token



95
96
97
98
99
100
 
# File 'lib/server/ai/ai_config_tracker.rb', line 95

def resumption_token
  payload = { runId: @run_id, configKey: @config_key }
  payload[:variationKey] = @variation_key if @variation_key && !@variation_key.empty?
  payload[:version] = @version
  Base64.urlsafe_encode64(JSON.generate(payload), padding: false)
end

#track_bedrock_converse_metrics { ... } ⇒ Hash

Track AWS Bedrock conversation operations. This method tracks the duration, token usage, and success/error status.

Subsequent calls re-run the inner block but emit only metrics not already recorded on this Tracker. Call create_tracker on the AI Config to start a new run.

Yields:

  • The block to track.

Returns:

  • (Hash) —

    The original response hash.



320
321
322
323
324
325
326
327
328
 
# File 'lib/server/ai/ai_config_tracker.rb', line 320

def track_bedrock_converse_metrics(&block)
  result = track_duration_of(&block)
  track_success
  track_tokens(bedrock_to_token_usage(result[:usage])) if result[:usage]
  result
rescue StandardError
  track_error
  raise
end

#track_duration(duration) ⇒ Object

Track the duration of an AI run.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • duration (Integer) —

    The duration in milliseconds



133
134
135
136
137
138
139
140
141
142
143
144
145
 
# File 'lib/server/ai/ai_config_tracker.rb', line 133

def track_duration(duration)
  unless @summary.duration.nil?
    @logger&.warn("Skipping track_duration: duration already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.duration = duration
  @ld_client.track(
    '$ld:ai:duration:total',
    @context,
    flag_data,
    duration
  )
end

#track_duration_of { ... } ⇒ Object

Track the duration of a block of code

Yields:

  • The block to measure

Returns:

  • The result of the block



153
154
155
156
157
158
159
 
# File 'lib/server/ai/ai_config_tracker.rb', line 153

def track_duration_of(&block)
  start_time = Time.now
  yield
ensure
  duration = ((Time.now - start_time) * 1000).to_i
  track_duration(duration)
end

#track_error ⇒ Object

Track an error in AI generation.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.



233
234
235
236
237
238
239
240
241
242
243
244
245
 
# File 'lib/server/ai/ai_config_tracker.rb', line 233

def track_error
  unless @summary.success.nil?
    @logger&.warn("Skipping track_error: success/error already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.success = false
  @ld_client.track(
    '$ld:ai:generation:error',
    @context,
    flag_data,
    1
  )
end

#track_feedback(kind:) ⇒ Object

Track user feedback.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • kind (Symbol) —

    The kind of feedback (:positive or :negative)



190
191
192
193
194
195
196
197
198
199
200
201
202
203
 
# File 'lib/server/ai/ai_config_tracker.rb', line 190

def track_feedback(kind:)
  unless @summary.feedback.nil?
    @logger&.warn("Skipping track_feedback: feedback already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.feedback = kind
  event_name = kind == :positive ? '$ld:ai:feedback:user:positive' : '$ld:ai:feedback:user:negative'
  @ld_client.track(
    event_name,
    @context,
    flag_data,
    1
  )
end

#track_openai_metrics { ... } ⇒ Object

Track OpenAI-specific operations. This method tracks the duration, token usage, and success/error status. If the provided block raises, this method will also raise. A failed operation will not have any token usage data.

Subsequent calls re-run the inner block but emit only metrics not already recorded on this Tracker. Call create_tracker on the AI Config to start a new run.

Yields:

  • The block to track.

Returns:

  • The result of the tracked block.



299
300
301
302
303
304
305
306
307
 
# File 'lib/server/ai/ai_config_tracker.rb', line 299

def track_openai_metrics(&block)
  result = track_duration_of(&block)
  track_success
  track_tokens(openai_to_token_usage(result[:usage])) if result[:usage]
  result
rescue StandardError
  track_error
  raise
end

#track_success ⇒ Object

Track a successful AI generation.

Records at most once per Tracker. track_success and track_error share state; only one of the two can record per Tracker, and subsequent calls are ignored.



212
213
214
215
216
217
218
219
220
221
222
223
224
 
# File 'lib/server/ai/ai_config_tracker.rb', line 212

def track_success
  unless @summary.success.nil?
    @logger&.warn("Skipping track_success: success/error already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.success = true
  @ld_client.track(
    '$ld:ai:generation:success',
    @context,
    flag_data,
    1
  )
end

#track_time_to_first_token(time_to_first_token) ⇒ Object

Track time to first token.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • duration (Integer) —

    The duration in milliseconds



168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/server/ai/ai_config_tracker.rb', line 168

def track_time_to_first_token(time_to_first_token)
  unless @summary.time_to_first_token.nil?
    @logger&.warn("Skipping track_time_to_first_token: time-to-first-token already recorded on this tracker. " \
                  "Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.time_to_first_token = time_to_first_token
  @ld_client.track(
    '$ld:ai:tokens:ttf',
    @context,
    flag_data,
    time_to_first_token
  )
end

#track_tokens(token_usage) ⇒ Object

Track token usage.

Records at most once per Tracker; further calls are ignored.

Parameters:

  • token_usage (TokenUsage) —

    An object containing token usage details



254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
 
# File 'lib/server/ai/ai_config_tracker.rb', line 254

def track_tokens(token_usage)
  unless @summary.usage.nil?
    @logger&.warn("Skipping track_tokens: token usage already recorded on this tracker. Call create_tracker on the AI Config for a new run. #{flag_data}")
    return
  end
  @summary.usage = token_usage
  if token_usage.total.positive?
    @ld_client.track(
      '$ld:ai:tokens:total',
      @context,
      flag_data,
      token_usage.total
    )
  end
  if token_usage.input.positive?
    @ld_client.track(
      '$ld:ai:tokens:input',
      @context,
      flag_data,
      token_usage.input
    )
  end
  return unless token_usage.output.positive?

  @ld_client.track(
    '$ld:ai:tokens:output',
    @context,
    flag_data,
    token_usage.output
  )
end