Skip to main content
Supported data types: Logs • Metrics

​
Ingestion with AWS Data Firehose

Stream CloudWatch data to Phoebe using Amazon Data Firehose with an HTTP endpoint destination.
Prerequisites
  • Permissions to create IAM roles/policies, Firehose delivery streams, subscription filters, and (optionally) S3 buckets
  • Phoebe ingestion key
  • CloudWatch Log Groups to stream
Setup
1

Create (optional) S3 bucket for failures

Use this bucket for failed deliveries to aid troubleshooting.Example name: phoebe-log-stream-failures
2

Create a minimal-scope IAM role for Firehose

Firehose can auto-create a role, but a custom role limits permissions and improves security.
Policy (replace the bucket name if different):
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
              ],
              "Resource": [
                "arn:aws:s3:::phoebe-log-stream-failures",
                "arn:aws:s3:::phoebe-log-stream-failures/*"
              ]
            }
          ]
        }
Trust policy for Firehose:
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": { "Service": "firehose.amazonaws.com" },
              "Action": "sts:AssumeRole"
            }
          ]
        }
3

Create a Firehose delivery stream (HTTP endpoint)

  • Source: Direct PUT
  • Destination: HTTP Endpoint
  • HTTP endpoint URL: https://ingest.phoebe.ai/aws/firehose/logs
  • Access key: your Phoebe ingestion key
Prefer storing the key in AWS Secrets Manager; grant Firehose read access.
  • Content encoding: GZIP
  • Retry duration: ~300 seconds (tune as needed)
S3 backup:
  • Mode: Failed data only
  • Bucket: the one you created for failures
Buffering:
  • Size: 1–5 MB (smaller = lower latency)
  • Interval: 60–300 s
Service access:
  • Choose the IAM role you created for Firehose.
4

Allow CloudWatch Logs to put records into Firehose

Create an IAM role for logs.amazonaws.com to call firehose:PutRecord* on your stream.Policy (replace REGION, ACCOUNT_ID, and stream name):
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": ["firehose:PutRecord","firehose:PutRecordBatch"],
              "Resource": "arn:aws:firehose:REGION:ACCOUNT_ID:deliverystream/phoebe-firehose-stream"
            }
          ]
        }
Trust policy (CloudWatch Logs):
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": { "Service": "logs.amazonaws.com" },
              "Action": "sts:AssumeRole"
            }
          ]
        }
5

Create CloudWatch Logs subscription filter(s)

For each Log Group you want to stream:
  • Destination: Amazon Data Firehose
  • Delivery stream: your Firehose stream
  • Role: the CloudWatch-to-Firehose role
  • Filter pattern: empty for all logs (or set a pattern to restrict)
6

Verify delivery

Generate a test event:
        aws logs put-log-events \
          --log-group-name /your/log/group \
          --log-stream-name test-stream \
          --log-events timestamp=$(date +%s)000,message='{"test":"phoebe-firehose"}'
  • In Firehose → your stream → Monitoring, verify successful deliveries increase and failures remain zero.
  • Check the failure S3 bucket is empty (or inspect objects for errors).

​
API access

  1. Create IAM Role/User with CloudWatchReadOnlyAccess policy
  2. Generate access key and secret
  3. Enter credentials in the Integrations UI