Observability

Phoebe uses two integration approaches: Context Lake ingestion and vendor API access. We recommend using both - ingestion enables better evidence retrieval, while API access fetches additional context such as dashboards.

Default data retention is 30 days. Contact support to request a different retention period.

Before you begin

Ingestion key from Phoebe (provided during onboarding)
Outbound HTTPS (443) allowed to ingest.phoebe.ai
(Recommended) A secrets manager or environment variables - avoid hard-coding keys in configs

OpenTelemetry (Recommended)

Supported data types: Logs • Metrics • Traces

Recommended approach for comprehensive coverage and lowest operational overhead.

If your stack supports OTel, use this for Context Lake ingestion. Still configure API access for your provider below to enable dashboards and additional context.

OpenTelemetry works with any system that already uses an OTel Collector/Agent. Add Phoebe as an additional exporter so you keep your existing destinations and send a copy to Phoebe. Add a Phoebe OTLP/HTTP exporter and reference it in your pipelines. Using environment variables for the key is strongly encouraged.

# --- opentelemetry-collector.yaml ---

exporters:
  # Your existing exporters (keep as-is)
  otlp/primary:
    endpoint: your-existing-endpoint

  # Phoebe exporter (OTLP over HTTP)
  otlphttp/phoebe:
    endpoint: https://ingest.phoebe.ai/otel
    headers:
      X-API-Key: ${PHOEBE_INGEST_KEY}
    compression: gzip
    timeout: 10s
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 1024
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 5m

service:
  pipelines:
    logs:
      receivers: [your-existing-receivers]
      exporters: [otlp/primary, otlphttp/phoebe]
    metrics:
      receivers: [your-existing-receivers]
      exporters: [otlp/primary, otlphttp/phoebe]
    traces:
      receivers: [your-existing-receivers]
      exporters: [otlp/primary, otlphttp/phoebe]

Set ${PHOEBE_INGEST_KEY} securely via environment variables or your secrets manager. Avoid hard-coding secrets into config files.

Important: Add the Phoebe exporter in addition to your current exporters for each pipeline so data keeps flowing to existing systems.

Verify

Restart/reload your Collector
Check Collector logs for export activity or errors:

# Example for a containerized Collector
docker logs <collector-container> | grep -i "otlphttp/phoebe"

Generate test telemetry (e.g., a sample trace or log event) and review for errors in the Collector logs. Contact support to confirm the records are visible in your Context Lake

Amazon Web Services

Supported data types: Logs • Metrics

Context Lake ingestion

Stream CloudWatch data to Phoebe using Amazon Data Firehose with an HTTP endpoint destination.

CloudWatch Logs
CloudWatch Metrics

Prerequisites

Permissions to create IAM roles/policies, Firehose delivery streams, subscription filters, and (optionally) S3 buckets
Phoebe ingestion key
CloudWatch Log Groups to stream

Setup

Create (optional) S3 bucket for failures

Use this bucket for failed deliveries to aid troubleshooting.Example name: phoebe-log-stream-failures

Create a minimal-scope IAM role for Firehose

Firehose can auto-create a role, but a custom role limits permissions and improves security.

Policy (replace the bucket name if different):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::phoebe-log-stream-failures",
        "arn:aws:s3:::phoebe-log-stream-failures/*"
      ]
    }
  ]
}

Trust policy for Firehose:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "firehose.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create a Firehose delivery stream (HTTP endpoint)

Source: Direct PUT
Destination: HTTP Endpoint
HTTP endpoint URL: https://ingest.phoebe.ai/aws/firehose/logs
Access key: your Phoebe ingestion key

Prefer storing the key in AWS Secrets Manager; grant Firehose read access.

Content encoding: GZIP
Retry duration: ~300 seconds (tune as needed)

S3 backup:

Mode: Failed data only
Bucket: the one you created for failures

Buffering:

Size: 1–5 MB (smaller = lower latency)
Interval: 60–300 s

Service access:

Choose the IAM role you created for Firehose.

Allow CloudWatch Logs to put records into Firehose

Create an IAM role for logs.amazonaws.com to call firehose:PutRecord* on your stream.Policy (replace REGION, ACCOUNT_ID, and stream name):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:PutRecord","firehose:PutRecordBatch"],
      "Resource": "arn:aws:firehose:REGION:ACCOUNT_ID:deliverystream/phoebe-firehose-stream"
    }
  ]
}

Trust policy (CloudWatch Logs):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "logs.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create CloudWatch Logs subscription filter(s)

For each Log Group you want to stream:

Destination: Amazon Data Firehose
Delivery stream: your Firehose stream
Role: the CloudWatch-to-Firehose role
Filter pattern: empty for all logs (or set a pattern to restrict)

Verify delivery

Generate a test event:

aws logs put-log-events \
  --log-group-name /your/log/group \
  --log-stream-name test-stream \
  --log-events timestamp=$(date +%s)000,message='{"test":"phoebe-firehose"}'

In Firehose → your stream → Monitoring, verify successful deliveries increase and failures remain zero.
Check the failure S3 bucket is empty (or inspect objects for errors).

Metric Streams provide near real-time CloudWatch metrics (2-3 minute latency) for services like EC2, RDS, Lambda, ECS, ALB, WAF, and 200+ other AWS services.

Setup

Create a Firehose delivery stream for metrics

Create a new Firehose delivery stream (separate from logs):

Source: Direct PUT
Destination: HTTP Endpoint
HTTP endpoint URL: https://ingest.phoebe.ai/aws/firehose/metrics
Access key: your Phoebe ingestion key
Content encoding: GZIP
Retry duration: 300 seconds

S3 backup:

Mode: Failed data only
Bucket: your failure bucket (can reuse the one created for logs)

Buffering:

Size: 1–5 MB
Interval: 60 s (lower for near real-time)

Service access:

Use the same IAM role created for logs, or create a dedicated one.

Create a CloudWatch Metric Stream

In CloudWatch → Metrics → Streams, create a new Metric Stream:

Destination: Amazon Data Firehose (in same account)
Firehose stream: select your metrics Firehose stream
Output format: JSON

Use JSON format, not OpenTelemetry.

Metrics to stream: Choose either:
- All metrics — streams all CloudWatch namespaces
- Selected namespaces — choose specific services (e.g., AWS/EC2, AWS/RDS, AWS/Lambda, AWS/ApplicationELB, AWS/WAFV2)

Start with specific namespaces to control costs, then expand as needed. High-cardinality namespaces like AWS/EC2 can generate significant volume.

Additional statistics: Optionally include percentiles (p50, p90, p99) for supported metrics.

Configure IAM permissions

CloudWatch Metric Streams needs permission to write to Firehose. Create or update an IAM role:Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:PutRecord", "firehose:PutRecordBatch"],
      "Resource": "arn:aws:firehose:REGION:ACCOUNT_ID:deliverystream/your-metrics-stream"
    }
  ]
}

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "streams.metrics.cloudwatch.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

Verify metrics delivery

After a few minutes:

In CloudWatch → Metric Streams, verify the stream shows Running status
Check the MetricUpdate count is increasing in the stream’s monitoring
In Firehose → your stream → Monitoring, verify successful deliveries
Check the failure S3 bucket is empty

Metrics typically appear within 2-3 minutes of being emitted by AWS services.

Available metricsCloudWatch Metric Streams can send metrics from 200+ AWS services, including:

Service	Example Metrics
EC2	CPUUtilization, NetworkIn/Out, DiskReadOps
RDS	CPUUtilization, DatabaseConnections, FreeableMemory
Lambda	Invocations, Duration, Errors, Throttles
ECS	CPUUtilization, MemoryUtilization
ALB	RequestCount, TargetResponseTime, HTTPCode_*
WAF	AllowedRequests, BlockedRequests
ElastiCache	CacheHits, CacheMisses, CPUUtilization
SQS	NumberOfMessagesReceived, ApproximateAgeOfOldestMessage

API access

Create IAM Role/User with CloudWatchReadOnlyAccess policy
Generate access key and secret
Enter credentials in the Integrations UI

Grafana

Supported data types: Logs • Metrics • Traces

Context Lake ingestion

Promtail (Logs)
Prometheus (Metrics)

ConfigurationAdd the Phoebe endpoint to the clients section of your Promtail configuration file (typically config.yaml or promtail.yaml).Single destinationIf you’re only sending logs to Phoebe:

clients:
  - url: https://ingest.phoebe.ai/loki/api/v1/push
    headers:
      X-API-Key: ${PHOEBE_INGEST_KEY}

Multiple destinations (recommended)If you need to send logs to both your own Loki instance and Phoebe, list Phoebe as the last client. Promtail sends logs to clients sequentially in the order they’re defined, so this ensures your primary destination receives logs first:

clients:
  # Your primary Loki instance - processed first
  - url: http://your-loki-instance:3100/loki/api/v1/push

  # Phoebe - processed last to avoid blocking your primary destination
  - url: https://ingest.phoebe.ai/loki/api/v1/push
    headers:
      X-API-Key: ${PHOEBE_INGEST_KEY}

Client ordering matters: Promtail sends logs to clients sequentially on a single thread. List your primary logging destination first to ensure it always receives logs with the lowest latency.

Headers set by Promtail itself (such as X-Scope-OrgID) cannot be overwritten using the headers configuration.

Verify

Restart Promtail to apply the changes.
Check Promtail logs for any connection errors.
Contact Phoebe to confirm logs are being received in your Context Lake.

Three options for sending Prometheus metrics to Phoebe. Choose based on your deployment and scale.

Option 1: Remote Write

The remote write protocol reliably propagates samples in real-time from a sender to a receiver without loss.With the remote write option, you can send data directly to ingest.phoebe.ai](http://ingest.phoebe.ai) or deploy our Phoebe Agent in your environment. The receiver then forwards the data to us.Enable metadata WAL recordsPhoebe requires metric metadata (type, unit, help) to translate remote write to OTLP. Enable the feature flag:

./prometheus --config.file config.yml --enable-feature=metadata-wal-records

Configure remote write

remote_write:
  - url: "https://ingest.phoebe.ai"
    protobuf_message: io.prometheus.write.v2.Request
    headers:
      X-API-Key: ${PHOEBE_INGEST_KEY}

Option 2: Phoebe Agent as Prometheus Server

Phoebe Agent scrapes your instrumented services directly. Deploy with a scrape_config:

docker run \
  -e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
  -e SCRAPE_CONFIG_FILE=/etc/phoebe/scrape.yaml \
  -v /path/to/your/scrape.yaml:/etc/phoebe/scrape.yaml \
  phoebe-agent --config /otel/configs/prometheus/server-config.yaml

We support the full set of Prometheus scrape_config options.

Running multiple replicas with the same config will scrape targets multiple times. Configure each replica with different settings.

For Kubernetes with Target Allocator, contact support for setup guidance.

Option 3: Federation

Phoebe Agent scrapes time series from your existing Prometheus server. Minimal setup—just provide the URL to scrape from.

docker run -d \
  -e FEDERATE_TARGET=${PROMETHEUS_TARGET} \
  -e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
  phoebe-agent \
  --config /otel/configs/prometheus/federate-otel-config.yaml

By default, the matcher {job=~".+"} scrapes all metrics excluding Prometheus internals (prometheus_*, go_*, scrape_*, promhttp_*).To target specific metrics (e.g., recording rules):

docker run -d \
  -e FEDERATE_TARGET=${PROMETHEUS_TARGET} \
  -e FEDERATE_MATCH='{__name__=~"aggregated:.*"}' \
  -e PHOEBE_API_KEY=${PHOEBE_API_KEY} \
  phoebe-agent \
  --config /otel/configs/prometheus/federate-otel-config.yaml

Federation can be resource-intensive on the source Prometheus. Use recording rules to pre-aggregate metrics or filter to specific jobs. For high-volume environments, consider Remote Write instead.

API access

Create service account - In Grafana, go to Administration > Service accounts, click Add service account, name it (e.g., phoebe-integration), and assign Viewer role
Generate token - Click on the service account, go to Tokens tab, click Add service account token, and copy immediately (it won’t be shown again)
- Ensure the token has the required permissions for read access to Loki (logs), Prometheus/Mimir (metrics) and Tempo (traces)
Find service identifiers - In Grafana Explore, select Loki, expand “Label filters” and look for labels that identify your services Service identifier examples:
Environment Identifiers
Standard ["service_name"]
Kubernetes ["app_kubernetes_io_name", "app"]
Docker Compose ["container_name", "compose_service"]
Custom ["application_name", "service_id"]

Without correct service identifiers, Phoebe cannot group logs by service. Test in Grafana Explore first with a query like {your_service_identifier="your-service-name"}.
Enter credentials in the Integrations UI

Environment	Identifiers
Standard	`["service_name"]`
Kubernetes	`["app_kubernetes_io_name", "app"]`
Docker Compose	`["container_name", "compose_service"]`
Custom	`["application_name", "service_id"]`

Troubleshooting

Issue	Fix
No services discovered	Check identifiers match exactly (case-sensitive) with Loki labels
Missing data	Verify token permissions by testing queries in Grafana
Connection errors	Ensure `BASE_URL` is accessible and includes protocol (`https://`)

For more details, see Grafana permissions documentation.

Datadog

Supported data types: Logs • Metrics • Traces

API access

Create an API key in Datadog API settings
Create an Application key in Datadog Application Keys
Ensure the associated user/service account has these required permissions and scopes:
- Read logs (Scopes required: logs_read_data, logs_read_index_data)
- Access metrics (Scopes required: metrics_read, timeseries_query)
- View traces (Scopes required: apm_read)
- Read dashboard data (Scopes required: dashboards_read)
- Read monitor data (Scopes required: monitors_read)
Enter credentials in the Integrations UI

Google Cloud Platform

Supported data types: Logs

API access

In GCP Console, create a new service account in IAM & Admin > Service Accounts and assign it with Logs Viewer role
Create a new JSON key
Enter credentials in the Integrations UI

New Relic

Supported data types: Logs • Metrics • Traces

API access

Generate a User API key
Find the Account ID from the API keys table
Enter credentials in the Integrations UI

Troubleshooting

Verifying ingestion

Generate test data in the relevant system (log line, synthetic metric, or trace)
Check sender logs (Collector/Firehose) for HTTP errors or backoffs
Confirm with Phoebe—share the approximate timestamp and source so we can validate arrival in your Context Lake

Common issues & fixes

Authentication (401/403)

Double-check the exact header name/value:
- OpenTelemetry: X-API-Key: <key>
- Firehose HTTP destination: configure the Access key to your Phoebe key
- Promtail: X-API-Key: <key> in the headers section
Ensure no extra whitespace or quotes; prefer environment variables or secrets manager.

Connectivity / timeouts

Allow outbound HTTPS (443) to ingest.phoebe.ai from the sending network (EKS/ECS nodes, VMs, etc.)
Enable compression to reduce payload size (gzip)
Tune buffering to keep chunks ≤ 5 MB
For Firehose, reduce buffer size/interval to lower latency

OpenTelemetry data not appearing

Verify the Phoebe exporter is listed in every relevant pipeline (logs, metrics, traces)
Ensure you added an additional exporter rather than replacing existing ones
Confirm endpoint is exactly https://ingest.phoebe.ai/otel (no /v1 prefix)
Check Collector logs for otlphttp/phoebe export errors and backoff messages

Amazon Data Firehose delivery failures

Check the S3 failure bucket for recent objects; inspect payloads and error messages
Verify:
- Firehose stream is Active
- IAM roles & policies (both Firehose and CloudWatch Logs) are correct
- CloudWatch subscription filters show Active
Keep buffer size under 5 MB to avoid large payload failures

Promtail logs not appearing

Verify the Phoebe client is listed in the clients section with the correct URL (https://ingest.phoebe.ai/loki/api/v1/push).
Ensure X-API-Key header is set correctly with your Phoebe ingestion key.
If using multiple destinations, confirm Phoebe is listed last to avoid blocking your primary destination.
Check Promtail logs for connection errors or 4xx/5xx responses.
Verify outbound HTTPS (443) to ingest.phoebe.ai is allowed from the Promtail host.

Helpful HTTP responses (at sender)

2xx — accepted by Phoebe
4xx — fix configuration (keys/headers/endpoint/payload format)
5xx — transient; sender should retry (ensure retries/backoff are enabled)

Getting help

When contacting support, include:

Integration method (OpenTelemetry, Firehose, Promtail, or Prometheus)
Sender logs with error snippets
Approximate timestamp(s) and source (service name, Log Group, etc.)
Confirmation that your ingestion key is configured and how it’s stored (env/secrets)

Appendix: Security & operations best practices

Don’t hard-code secrets. Use environment variables or a secrets manager and restrict read access
TLS in transit. All endpoints use HTTPS. Ensure outbound proxies (if any) don’t downgrade TLS
Right-sized batching. Smaller batches reduce latency and make retries cheaper; larger batches are more efficient for high-throughput logs. Start with 1–5 MB buffers
Resource attributes matter. Ensure service.name, deployment.environment, and (optionally) service.version are present so investigations group data correctly
Backpressure & retries. Keep retries enabled with exponential backoff to ride out transient network hiccups

Quick start

Integrations

Investigations

Incident Review

Guides

Before you begin

OpenTelemetry (Recommended)

Amazon Web Services

Context Lake ingestion

API access

Grafana

Context Lake ingestion

Option 1: Remote Write

Option 2: Phoebe Agent as Prometheus Server

Option 3: Federation

API access

Datadog

API access

Google Cloud Platform

API access

New Relic

API access

Troubleshooting

Verifying ingestion

Common issues & fixes

Getting help

Appendix: Security & operations best practices

Quick start

Integrations

Investigations

Incident Review

Guides

​Before you begin

​OpenTelemetry (Recommended)

​Amazon Web Services

​Context Lake ingestion

​API access

​Grafana

​Context Lake ingestion

​Option 1: Remote Write

​Option 2: Phoebe Agent as Prometheus Server

​Option 3: Federation

​API access

​Datadog

​API access

​Google Cloud Platform

​API access

​New Relic

​API access

​Troubleshooting

​Verifying ingestion

​Common issues & fixes

​Getting help

​Appendix: Security & operations best practices

Before you begin

OpenTelemetry (Recommended)

Amazon Web Services

Context Lake ingestion

API access

Grafana

Context Lake ingestion

Option 1: Remote Write

Option 2: Phoebe Agent as Prometheus Server

Option 3: Federation

API access

Datadog

API access

Google Cloud Platform

API access

New Relic

API access

Troubleshooting

Verifying ingestion

Common issues & fixes

Getting help

Appendix: Security & operations best practices