OTel Integration

Send OpenTelemetry metrics from your applications to Chamber via the Standalone Agent

The Standalone Agent is in beta. OTLP integration behavior may change.

The Standalone Agent includes a built-in OpenTelemetry (OTLP) gRPC receiver. Any application instrumented with the OpenTelemetry SDK can send metrics directly to the agent, which forwards them to the Chamber dashboard alongside GPU and host metrics.

This lets you correlate application-level metrics (training loss, throughput, queue depth, request latency) with infrastructure metrics in a single view.

How It Works

1. Your application exports OTLP metrics over gRPC to localhost:4317
2. The agent receives, transforms, and buffers the metrics
3. Metrics are uploaded to Chamber in batches every 60 seconds
4. They appear on the Services dashboard alongside infrastructure metrics

Configuration

The OTLP receiver is enabled by default on port 4317. No additional configuration is required for most setups.

Variable	Default	Description
CHAMBER_OTLP_GRPC_ENABLED	true	Enable or disable the OTLP gRPC receiver
CHAMBER_OTLP_GRPC_PORT	4317	Port the gRPC server listens on
CHAMBER_OTLP_GRPC_HOST	0.0.0.0	Bind address for the gRPC server
CHAMBER_OTLP_METRICS_QUEUE_SIZE	10000	Maximum queued metric batches before dropping

To disable the receiver, set CHAMBER_OTLP_GRPC_ENABLED=false in /etc/chamber/agent.env and restart the agent.

Sending Metrics from Your Application

Point your application’s OTLP exporter to the agent’s gRPC endpoint. Here is an example using the Python OpenTelemetry SDK:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource

# Configure exporter to send to the local Chamber agent
exporter = OTLPMetricExporter(
    endpoint="localhost:4317",
    insecure=True,  # Local connection, no TLS needed
)

reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30000)
provider = MeterProvider(
    resource=Resource.create({"service.name": "my-training-job"}),
    metric_readers=[reader],
)
metrics.set_meter_provider(provider)

meter = metrics.get_meter("my-training-job")

# Create metrics
loss_gauge = meter.create_gauge("training.loss", description="Current training loss")
throughput_counter = meter.create_counter(
    "training.samples.processed",
    unit="1",
    description="Total training samples processed",
)

# Record metrics in your training loop
loss_gauge.set(0.42)
throughput_counter.add(batch_size)

Set service.name in your resource attributes — it appears as the service name label on all metrics in the Chamber dashboard.

Metric Name Transformation

The agent transforms OTLP metric names to a Prometheus-compatible format before uploading:

Transformation	Example
Dots and hyphens become underscores	training.loss → training_loss
Unit suffixes are appended	request_duration (unit: s) → request_duration_seconds
Monotonic sums get _total suffix	samples.processed (sum, monotonic) → samples_processed_total
Consecutive underscores are collapsed	my__metric → my_metric

Unit Suffix Mapping

OTel Unit	Suffix
s	_seconds
ms	_milliseconds
By	_bytes
KBy	_kilobytes
MBy	_megabytes
GBy	_gigabytes
1	_ratio

Histogram Support

OTLP histograms are decomposed into Prometheus-style metrics:

Generated Metric	Description
{name}_bucket	Cumulative bucket counts with le (less-than-or-equal) labels
{name}_count	Total number of observations
{name}_sum	Sum of all observed values

Supported Metric Types

OTel Type	Supported
Gauge	Yes
Sum (monotonic and non-monotonic)	Yes
Histogram	Yes
ExponentialHistogram	No
Summary	No

Verifying Metrics Are Flowing

After configuring your application, check the agent logs:

sudo journalctl -u chamber-agent-standalone -f | grep -i otlp

You should see log lines indicating metrics received. Then check the Services tab in the dashboard — your application metrics will appear with the service_name label you configured.

Next Steps

Metrics Reference

Full list of infrastructure metrics collected

Configuration

All environment variables and settings