AWS IAM Permissions

BoilStream requires specific AWS IAM permissions depending on which features you use. This guide covers the minimum required permissions and optional advanced permissions.

Quick Start: Minimal S3 Policy

For basic data ingestion to S3, you only need S3 permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BoilStreamS3Access",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
        },
        {
            "Sid": "BoilStreamS3Objects",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
        }
    ]
}

IAM Permissions Required for AWS S3

If you see the error Failed to initialize DuckLake infrastructure: Failed to create IAM role, your AWS credentials don't have the required IAM management permissions.

Solutions:

1. Grant IAM permissions (recommended) - see Full DuckLake Permissions below
2. Use S3-compatible storage - MinIO, R2, etc. don't require IAM management

Permission Levels

Level 1: Basic S3 Ingestion

Required permissions for data ingestion only:

Permission	Purpose
s3:ListBucket	Connectivity check on startup
s3:PutObject	Write Parquet files
s3:AbortMultipartUpload	Clean up failed uploads

Level 2: DuckLake with Static Credentials

If using DuckLake without per-user credential vending:

Permission	Purpose
s3:GetObject	Read Parquet files for queries
s3:DeleteObject	DuckLake VACUUM operations
All Level 1 permissions	-

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BoilStreamDuckLakeAccess",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR-BUCKET-NAME",
                "arn:aws:s3:::YOUR-BUCKET-NAME/*"
            ]
        }
    ]
}

Level 2b: Cluster Mode (S3 Cluster State)

Cluster mode uses S3 for cluster discovery, leader/broker coordination, and per-catalog backups. The same primary storage backend stores:

s3://YOUR-BUCKET/YOUR-PREFIX/cluster_state/
  leader.json
  cluster_secret.json
  brokers/
  catalogs/backups/

Quack Multi-Raft (cp-metadata) consensus is not on S3 — each node keeps its own NuRaft log/snapshots locally and reseeds from peers, so no S3 snapshot pointer or DR layout is required. (Bulk analytical data still lands in object storage via DuckLake hot→cold tiering; that is covered by the data-prefix permissions below.)

Required permissions:

Permission	Purpose
s3:ListBucket	Discover cluster-state objects
s3:GetObject	Read leader state, broker state, shared secret, and catalog backups
s3:PutObject	Write leader/broker state and catalog backups
s3:DeleteObject	Remove stale cluster-state objects

Recommended prefix-scoped policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BoilStreamClusterStateList",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "YOUR-PREFIX/cluster_state/*"
                    ]
                }
            }
        },
        {
            "Sid": "BoilStreamClusterStateObjects",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/YOUR-PREFIX/cluster_state/*"
        }
    ]
}

For S3-compatible storage such as RustFS, set use_path_style: true and use the provider's equivalent bucket credentials. Cluster-state objects are ordinary objects under the cluster_state/ prefix and do not require IAM role creation.

Level 3: Full DuckLake with IAM Auto-Provisioning

For enterprise deployments with per-user temporary credentials:

Permission	Purpose
sts:GetCallerIdentity	Detect AWS account ID
iam:GetRole	Check if DuckLake role exists
iam:CreateRole	Create DuckLake access role
iam:PutRolePolicy	Attach S3 policy to role
sts:AssumeRole	Vend temporary credentials to users
All Level 2 permissions	-

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BoilStreamS3Access",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR-BUCKET-NAME",
                "arn:aws:s3:::YOUR-BUCKET-NAME/*"
            ]
        },
        {
            "Sid": "BoilStreamIAMManagement",
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity",
                "iam:GetRole",
                "iam:CreateRole",
                "iam:PutRolePolicy"
            ],
            "Resource": "*"
        },
        {
            "Sid": "BoilStreamAssumeRole",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::*:role/ducklake-*"
        }
    ]
}

Configuration

Using S3-Compatible Storage (No IAM Required)

For MinIO, Cloudflare R2, or other S3-compatible storage, no IAM permissions are needed:

storage:
  backends:
    - name: "minio"
      backend_type: "s3"
      enabled: true
      primary: true
      endpoint: "http://localhost:9000"  # Non-AWS endpoint
      bucket: "boilstream"
      access_key: "minioadmin"
      secret_key: "minioadmin"
      use_path_style: true  # Required for MinIO

BoilStream automatically detects non-AWS endpoints and skips IAM provisioning.

Troubleshooting

Error: Failed to initialize DuckLake infrastructure

Full error:

Error: Failed to initialize DuckLake infrastructure: Failed to create IAM role:
ducklake-BUCKET-access: service error

Causes:

1. Missing IAM permissions (iam:CreateRole, iam:PutRolePolicy)
2. AWS credentials don't have IAM access
3. Running outside AWS without proper IAM setup

Solutions:

1. Grant the Level 3 IAM permissions to your AWS credentials
2. Use a non-AWS S3-compatible endpoint (MinIO, R2, etc.)

Error: Failed to get AWS account ID

Full error:

Failed to get AWS account ID: service error

Cause: Missing sts:GetCallerIdentity permission or invalid credentials.

Solutions:

1. Verify credentials work: aws sts get-caller-identity
2. Add sts:GetCallerIdentity to your IAM policy

Error: AssumeRole test failed

Full error:

AssumeRole test failed: AccessDenied

Cause: The IAM role was created but the credentials can't assume it.

Solutions:

1. Wait 10-30 seconds for IAM propagation
2. Verify sts:AssumeRole permission for arn:aws:iam::*:role/ducklake-*
3. Check the role's trust policy allows your account

Security Best Practices

1. Use IAM roles on EC2/ECS instead of static credentials
2. Restrict S3 access to specific bucket and prefix
3. Use separate credentials for BoilStream vs application workloads
4. Enable CloudTrail to audit S3 and IAM operations
5. Rotate credentials regularly if using static access keys

See Also

• Configuration Reference
• DuckLake Integration
• Getting Started