Observability Stack#

Added in v0.5. A one-command opt-in profile in docker-compose.yml spins up the full observability trio pre-wired to InfraGuard’s /metrics endpoint and OTEL span export.

Enable#

docker compose --profile observability up -d

Services started:

ServicePortPurpose
otel4318 (HTTP), 4317 (gRPC)OpenTelemetry collector; receives spans from the proxy/dashboard and forwards
prometheus9091Scrapes InfraGuard’s /metrics every 15s
grafana3000Auto-provisioned with the InfraGuard overview dashboard

Admin password for Grafana defaults to admin; override with GRAFANA_ADMIN_PASSWORD in .env.

Config files (under deploy/)#

  • deploy/otel/config.yaml - collector pipeline (OTLP receivers, batch + memory-limiter processors, debug exporter by default). Uncomment the OTLP exporter block to forward to Tempo, Honeycomb, Datadog Agent, etc.
  • deploy/prometheus/prometheus.yml - scrape jobs for the dashboard’s /metrics. A commented-out job template for scaled proxy replicas is included.
  • deploy/grafana/dashboards.yaml - provisioning descriptor; Grafana imports every *.json in the directory as a dashboard.
  • deploy/grafana/infraguard.json - the shipped overview dashboard (request rate, block rate, active beacons, upstream latency histogram, per-filter block rate, circuit-breaker transitions, top blocked IPs, per-domain burn score).

Wire InfraGuard to the collector#

Add to config.yaml:

observability:
  otel:
    enabled: true
    service_name: infraguard-proxy
    endpoint: http://otel:4318
    resource_attrs:
      deployment.env: production

The proxy’s app lifespan calls setup_otel on start and shutdown_otel on stop. Spans emit for router.handle, pipeline.evaluate, filter.<name>, upstream.forward, and drop.handle. See opentelemetry.

Custom metrics via prom_custom plugin#

Emit operator-defined counters:

plugins:
  - infraguard.plugins.builtin.prom_custom

plugin_settings:
  prom_custom:
    options:
      counters:
        - name: infraguard_requests_by_country_total
          labels: ["country"]
          from: "metadata.geo.country"
        - name: infraguard_requests_by_campaign_total
          labels: ["campaign"]
          from: "metadata.beacon_labels.campaign"

Then add a Grafana panel referencing the new series.

HPA input (Kubernetes)#

The infraguard_active_beacons gauge is the HPA input for the Helm chart’s proxy Deployment. See kubernetes for the prometheus-adapter recipe that publishes the gauge through the Custom Metrics API.