Supported data types: Logs • Metrics • Traces
Collector-level ingestion
Promtail (Logs)
Prometheus (Metrics)
ConfigurationAdd the Phoebe endpoint to the clients section of your Promtail configuration file (typically config.yaml or promtail.yaml).Single destinationIf you’re only sending logs to Phoebe: clients:
- url: https://ingest.phoebe.ai/loki/api/v1/push
headers:
X-API-Key: ${PHOEBE_INGEST_KEY}
clients:
# Your primary Loki instance - processed first
- url: http://your-loki-instance:3100/loki/api/v1/push
# Phoebe - processed last to avoid blocking your primary destination
- url: https://ingest.phoebe.ai/loki/api/v1/push
headers:
X-API-Key: ${PHOEBE_INGEST_KEY}
Client ordering matters: Promtail sends logs to clients sequentially on a single thread. List your primary logging destination first to ensure it always receives logs with the lowest latency.
Headers set by Promtail itself (such as X-Scope-OrgID) cannot be overwritten using the headers configuration.
- Restart Promtail to apply the changes.
- Check Promtail logs for any connection errors.
- Contact Phoebe to confirm logs are being received.
Three options for sending Prometheus metrics to Phoebe. Choose based on your deployment and scale.Option 1: Remote Write
The remote write protocol reliably propagates samples in real-time from a sender to a receiver without loss.With the remote write option, you can send data directly to ingest.phoebe.ai or deploy our Phoebe Agent in your environment. The receiver then forwards the data to us.Enable metadata WAL recordsPhoebe requires metric metadata (type, unit, help) to translate remote write to OTLP. Enable the feature flag: ./prometheus --config.file config.yml --enable-feature=metadata-wal-records
remote_write:
- url: "https://ingest.phoebe.ai"
protobuf_message: io.prometheus.write.v2.Request
headers:
X-API-Key: ${PHOEBE_INGEST_KEY}
Option 2: Phoebe Agent as Prometheus Server
Phoebe Agent scrapes your instrumented services directly. Deploy with a scrape_config: docker run \
-e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
-e SCRAPE_CONFIG_FILE=/etc/phoebe/scrape.yaml \
-v /path/to/your/scrape.yaml:/etc/phoebe/scrape.yaml \
phoebe-agent --config /otel/configs/prometheus/server-config.yaml
scrape_config options.Running multiple replicas with the same config will scrape targets multiple times. Configure each replica with different settings.
For Kubernetes with Target Allocator, contact support for setup guidance.
Option 3: Federation
Phoebe Agent scrapes time series from your existing Prometheus server. Minimal setup—just provide the URL to scrape from. docker run -d \
-e FEDERATE_TARGET=${PROMETHEUS_TARGET} \
-e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
phoebe-agent \
--config /otel/configs/prometheus/federate-otel-config.yaml
{job=~".+"} scrapes all metrics excluding Prometheus internals (prometheus_*, go_*, scrape_*, promhttp_*).To target specific metrics (e.g., recording rules): docker run -d \
-e FEDERATE_TARGET=${PROMETHEUS_TARGET} \
-e FEDERATE_MATCH='{__name__=~"aggregated:.*"}' \
-e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
phoebe-agent \
--config /otel/configs/prometheus/federate-otel-config.yaml
Federation can be resource-intensive on the source Prometheus. Use recording rules to pre-aggregate metrics or filter to specific jobs. For high-volume environments, consider Remote Write instead.
API access
-
Create service account - In Grafana, go to Administration > Service accounts, click Add service account, name it (e.g.,
phoebe-integration), and assign Viewer role
-
Generate token - Click on the service account, go to Tokens tab, click Add service account token, and copy immediately (it won’t be shown again)
- Ensure the token has the required permissions for read access to Loki (logs), Prometheus/Mimir (metrics) and Tempo (traces)
-
Find service identifiers - In Grafana Explore, select Loki, expand “Label filters” and look for labels that identify your services
Service identifier examples:
| Environment | Identifiers |
|---|
| Standard | ["service_name"] |
| Kubernetes | ["app_kubernetes_io_name", "app"] |
| Docker Compose | ["container_name", "compose_service"] |
| Custom | ["application_name", "service_id"] |
Without correct service identifiers, Phoebe cannot group logs by service. Test in Grafana Explore first with a query like {your_service_identifier="your-service-name"}.
-
Enter credentials in the Integrations UI
Troubleshooting
| Issue | Fix |
|---|
| No services discovered | Check identifiers match exactly (case-sensitive) with Loki labels |
| Missing data | Verify token permissions by testing queries in Grafana |
| Connection errors | Ensure BASE_URL is accessible and includes protocol (https://) |
