Documentation Index
Fetch the complete documentation index at: https://braintrust.dev/docs/llms.txt
Use this file to discover all available pages before exploring further.
Summary
Goal: Configure pass/fail thresholds for scorers in Ruby using the Braintrust API.
Features: Function endpoint, scorer metadata, pass threshold configuration.
Configuration Steps
Step 1: Create the scorer registration function
POST to the /v1/function endpoint with scorer metadata.
def register_pass_threshold(project_id, scorer_name, pass_threshold)
return unless project_id
api_state = Braintrust::State.global
return unless api_state&.logged_in
payload = {
project_id: project_id,
name: scorer_name,
slug: scorer_name.to_s.downcase.gsub(/[^a-z0-9]+/, "-"),
function_type: "scorer",
metadata: { "__pass_threshold" => pass_threshold }
}
uri = URI("#{api_state.api_url}/v1/function")
request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{api_state.api_key}"
request["Content-Type"] = "application/json"
request.body = JSON.dump(payload)
response = Braintrust::Internal::Http.with_redirects(uri, request)
if response.code.to_i < 400
puts "[Scorer] Registered #{scorer_name} with pass threshold: #{pass_threshold}"
end
rescue StandardError => e
puts "[Warning] Failed to register pass threshold for #{scorer_name}: #{e.message}"
end
Step 2: Call the function with your scorer details
Pass the project ID, scorer name, and desired threshold value.
register_pass_threshold("your_project_id", "accuracy", 0.8)
Step 3: Verify in the Braintrust UI
The pass threshold will appear in the eval results, marking scores as pass/fail based on the threshold.
Notes
- The
__pass_threshold feature is not directly supported in the Ruby SDK yet
- This workaround uses the same API endpoint that Python and TypeScript SDKs use internally
- The slug field must be lowercase alphanumeric with hyphens only
- Registration errors won’t fail your eval; they’re logged as warnings