12-02-02 — Azure Monitor + App Insights + Foundry Tracing

Arquitetura da stack Microsoft de observabilidade

Application Insights: configuracao para LLMs

App Insights ja instrumenta automaticamente chamadas HTTP (incluindo Azure OpenAI) via SDK. Com algumas customizacoes, voce adiciona o contexto de IA.

Setup .NET (ASP.NET Core com Semantic Kernel)

// Program.cs
using Microsoft.ApplicationInsights.Extensibility;
using OpenTelemetry;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Adicionar Application Insights
builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.ConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
});

// Configurar OpenTelemetry para capturar spans do Semantic Kernel
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("Microsoft.SemanticKernel*")
        .AddAzureMonitorTraceExporter()  // envia para App Insights
    );

// Habilitar dados sensiveis nos spans (apenas dev/staging)
AppContext.SetSwitch(
    "Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnosticsSensitive", 
    builder.Environment.IsDevelopment()
);

// Adicionar Semantic Kernel
builder.Services.AddKernel()
    .AddAzureOpenAIChatCompletion(
        deploymentName: builder.Configuration["AZURE_OPENAI_DEPLOYMENT"],
        endpoint: builder.Configuration["AZURE_OPENAI_ENDPOINT"],
        apiKey: builder.Configuration["AZURE_OPENAI_KEY"]
    );

Metricas customizadas de IA via TelemetryClient

using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;

public class AgentMetricsService
{
    private readonly TelemetryClient _telemetry;

    public AgentMetricsService(TelemetryClient telemetry)
    {
        _telemetry = telemetry;
    }

    public void TrackAgentRun(AgentRunResult result)
    {
        // Metrica de custo por run
        _telemetry.TrackMetric("agent.tokens.input", result.InputTokens);
        _telemetry.TrackMetric("agent.tokens.output", result.OutputTokens);
        _telemetry.TrackMetric("agent.cost_usd", result.CostUsd);
        _telemetry.TrackMetric("agent.latency_ms", result.LatencyMs);

        // Evento customizado para analise posterior
        _telemetry.TrackEvent("AgentRunCompleted", new Dictionary<string, string>
        {
            ["userId"] = result.UserId,
            ["sessionId"] = result.SessionId,
            ["toolsUsed"] = string.Join(",", result.ToolsCalled),
            ["qualityScore"] = result.QualityScore.ToString("F2")
        });

        // Dependency tracking para tool calls
        foreach (var tool in result.ToolExecutions)
        {
            var dep = new DependencyTelemetry
            {
                Name = $"Tool: {tool.Name}",
                Duration = tool.Duration,
                Success = tool.Success,
                Type = "AI Tool"
            };
            _telemetry.TrackDependency(dep);
        }
    }
}

Azure AI Foundry Tracing

O Azure AI Foundry tem um sistema de tracing proprio integrado ao portal. Qualquer agente criado via Foundry SDK tem tracing automatico que aparece no portal em Foundry → Tracing.

import os
from azure.ai.projects import AIProjectClient
from azure.monitor.opentelemetry import configure_azure_monitor

# Configurar Azure Monitor como destino dos traces
configure_azure_monitor(
    connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

# O Foundry SDK automaticamente envia traces para o App Insights configurado
client = AIProjectClient.from_connection_string(
    conn_str=os.environ["AIPROJECT_CONNECTION_STRING"]
)

# Habilitar tracing no projeto Foundry
client.telemetry.enable()

# A partir daqui, todos os agents criados tem tracing automatico
agent = client.agents.create_agent(
    model="gpt-4o",
    name="sales-agent",
    instructions="Voce e um assistente de vendas..."
)

# Runs aparecem automaticamente em:
# 1. Azure AI Foundry portal → Tracing
# 2. Application Insights → Transaction search
# 3. Log Analytics → customEvents

Queries KQL para dashboard de IA

Com os dados no Log Analytics, use KQL para criar dashboards especificos de IA:

// KQL: Custo diario por usuario (ultimos 7 dias)
customMetrics
| where name == "agent.cost_usd"
| summarize daily_cost = sum(value) by bin(timestamp, 1d), tostring(customDimensions.userId)
| order by timestamp desc

// KQL: Latencia P50/P95/P99 do agente
customMetrics
| where name == "agent.latency_ms"
| summarize 
    p50 = percentile(value, 50),
    p95 = percentile(value, 95),
    p99 = percentile(value, 99)
  by bin(timestamp, 1h)
| render timechart

// KQL: Tool calls mais frequentes
customEvents
| where name == "AgentRunCompleted"
| extend tools = split(tostring(customDimensions.toolsUsed), ",")
| mv-expand tools
| summarize count() by tostring(tools)
| order by count_ desc
| take 10

Alertas no Azure Monitor

Configure alertas para situacoes criticas:

• Custo acima do threshold: Se agent.cost_usd diario superar USD 50
• Latencia degradada: Se P95 de latencia superar 5 segundos
• Taxa de erro elevada: Se mais de 5% das runs falharem em 15 minutos
• Volume anomalo: Se o numero de requisicoes triplicar em 10 minutos (possivel abuso)