12-02-01 — LangSmith e Langfuse

LangSmith

LangSmith e a plataforma de observabilidade da LangChain. Criada primariamente para aplicacoes LangChain/LangGraph, mas funciona com qualquer framework via SDK.

Setup em 5 minutos

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "lsv2_..."
os.environ["LANGCHAIN_PROJECT"] = "sales-agent-prod"

# Pronto. Qualquer chain LangChain agora e trackeada automaticamente.
from langchain_openai import AzureChatOpenAI
from langchain_core.messages import HumanMessage

llm = AzureChatOpenAI(
    azure_deployment="gpt-4o",
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_KEY"],
)

# Esta chamada aparece no LangSmith com trace completo
response = llm.invoke([HumanMessage(content="Analise as vendas do Q1")])

Funcionalidades principais do LangSmith

• Run tree: Visualizacao hierarquica de cada execucao com latencia por etapa
• Datasets: Curadoria de exemplos (input/output) para evals
• Evaluators: Executar LLM-as-judge ou criterios customizados sobre runs
• Prompt Hub: Versionamento e gerenciamento de prompts com pull/push via SDK
• Monitoring: Dashboards de latencia, custo e taxa de erro em tempo real

LangSmith para avaliacao

from langsmith import Client
from langsmith.evaluation import evaluate, LangChainStringEvaluator

client = Client()

# Criar dataset de teste
dataset = client.create_dataset("vendas-qa-v1")
client.create_examples(
    inputs=[
        {"question": "Qual o ticket medio em SP no Q1?"},
        {"question": "Clientes abaixo de 70% da meta?"},
    ],
    outputs=[
        {"answer": "R$ 45.230,00"},
        {"answer": "3 clientes: ACME, Globo, Vale"},
    ],
    dataset_id=dataset.id
)

# Executar avaliacao
results = evaluate(
    lambda inputs: meu_agente(inputs["question"]),
    data="vendas-qa-v1",
    evaluators=[
        LangChainStringEvaluator("qa", config={"llm": llm}),
        LangChainStringEvaluator("criteria", config={"criteria": "conciseness"}),
    ],
    experiment_prefix="gpt4o-v2",
)

Langfuse

Langfuse e open-source (MIT), self-hostavel via Docker, e framework-agnostico. E a escolha para times que precisam de controle total sobre os dados de observabilidade ou que nao usam LangChain.

Setup (Python SDK)

from langfuse import Langfuse
from langfuse.decorators import observe, langfuse_context

# Configuracao via env vars
# LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST

langfuse = Langfuse()

@observe()  # decorator: tracka automaticamente a funcao como um span
def retrieve_docs(query: str) -> list:
    # Logica de retrieval
    docs = vector_store.similarity_search(query, k=5)
    
    # Adicionar metadados ao span atual
    langfuse_context.update_current_observation(
        metadata={"docs_retrieved": len(docs)},
        input=query,
        output=[d.page_content[:100] for d in docs]
    )
    return docs

@observe()
def gerar_resposta(query: str, docs: list) -> str:
    langfuse_context.update_current_trace(
        user_id="user_123",
        session_id="session_abc",
        tags=["producao", "sales-agent"]
    )
    
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Voce e um assistente de vendas..."},
            {"role": "user", "content": f"Contexto: {docs}\n\nPergunta: {query}"}
        ]
    )
    
    # Registrar uso de tokens manualmente se necessario
    langfuse_context.update_current_observation(
        usage={"input": response.usage.prompt_tokens, 
               "output": response.usage.completion_tokens}
    )
    
    return response.choices[0].message.content

Self-host com Docker Compose

# docker-compose.yml para Langfuse self-hosted
# version: '3'
# services:
#   langfuse-server:
#     image: langfuse/langfuse:latest
#     ports: ["3000:3000"]
#     environment:
#       - DATABASE_URL=postgresql://postgres:password@db:5432/langfuse
#       - NEXTAUTH_SECRET=your_secret
#       - SALT=your_salt
#   db:
#     image: postgres:15
#     environment:
#       POSTGRES_PASSWORD: password

# Alternativa: deploy no Azure Container Apps (recomendado para producao)

Comparativo: LangSmith vs Langfuse