12-02-03 — Arize Phoenix e open-source

Arize Phoenix

Phoenix e open-source (BSD-3-Clause), criado pela Arize AI. Roda como servidor local ou em container e oferece:

• Trace viewer com suporte completo a OpenTelemetry GenAI
• Analise de embeddings (UMAP visualization)
• Eval suite built-in (relevance, hallucination, toxicity)
• Dataset management para fine-tuning e avaliacao
• Drift detection sobre distribuicao de queries

Iniciando o Phoenix localmente

import phoenix as px

# Inicia servidor Phoenix em http://localhost:6006
session = px.launch_app()

# Configurar OpenTelemetry para enviar para Phoenix
from phoenix.otel import register

tracer_provider = register(
    project_name="sales-agent",
    endpoint="http://localhost:6006/v1/traces",
)

# Auto-instrumentacao
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

# Agora todas as chamadas OpenAI aparecem no Phoenix UI
from openai import AzureOpenAI
client = AzureOpenAI(...)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Analise vendas Q1"}]
)

# Ver traces em http://localhost:6006
print(session.url)

Phoenix via Docker (producao)

# docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
# 
# Variaveis de ambiente para persistencia:
# PHOENIX_WORKING_DIR=/data
# 
# Docker Compose com PostgreSQL para persistencia:
# services:
#   phoenix:
#     image: arizephoenix/phoenix:latest
#     ports: ["6006:6006", "4317:4317"]
#     environment:
#       - PHOENIX_SQL_DATABASE_URL=postgresql://user:pass@db/phoenix
#   db:
#     image: postgres:15

Evals com Phoenix (LLM-as-judge)

import phoenix as px
from phoenix.evals import (
    HallucinationEvaluator,
    RelevanceEvaluator,
    run_evals
)
from phoenix.evals.models import AzureOpenAIModel

# Modelo para avaliacao (pode ser diferente do modelo em producao)
eval_model = AzureOpenAIModel(
    model="gpt-4o",
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"]
)

# Buscar traces do Phoenix para avaliar
client = px.Client()
traces_df = client.get_spans_dataframe(project_name="sales-agent")

# Executar evals
results = run_evals(
    dataframe=traces_df,
    evaluators=[
        HallucinationEvaluator(eval_model),
        RelevanceEvaluator(eval_model),
    ],
    provide_explanation=True,  # LLM explica o score
)

print(results[["input", "output", "hallucination_score", "relevance_score"]].head())

Outras opcoes open-source

Helicone — proxy simples

Helicone e um proxy HTTP que fica entre seu app e a OpenAI/Azure OpenAI. Sem mudanca de SDK — voce apenas troca o endpoint base.

from openai import AzureOpenAI

# Trocar endpoint para Helicone (self-hosted ou cloud)
client = AzureOpenAI(
    azure_endpoint="https://oai.hconeai.com/openai/deployments",  # self-hosted: sua URL
    api_key=os.environ["AZURE_OPENAI_KEY"],
    default_headers={
        "Helicone-Auth": f"Bearer {os.environ['HELICONE_API_KEY']}",
        "Helicone-Cache-Enabled": "true",  # cache de respostas identicas
        "Helicone-User-Id": user_id,
    }
)
# Todos os logs aparecem no Helicone dashboard sem mudar mais nada

Weave (Weights & Biases)

W&B Weave e a plataforma de MLOps da Weights & Biases adaptada para LLMs. Forte em versionamento de prompts e tracking de experimentos.

import weave

weave.init("sales-agent")

@weave.op()  # decorator: qualquer funcao trackeada automaticamente
def chamar_agente(query: str) -> str:
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": query}]
    )
    return response.choices[0].message.content

# Chamadas aparecem no W&B Weave com inputs, outputs e latencia
resultado = chamar_agente("Quais clientes abaixo da meta?")

Panorama geral das opcoes open-source