Authentication Troubleshooting

This guide helps diagnose and resolve common authentication and authorization issues in BoilStream.

Quick Diagnosis

Is Authentication Working?

Run this quick check to verify authentication status:

bash

# Check if authentication is enabled
echo "AUTH_PROVIDERS: $AUTH_PROVIDERS"

# Test BoilStream connectivity
curl -v http://localhost:50051 2>&1 | grep -i "connection"

# Check authentication logs
cat boilstream.logs | grep -i "auth\|token\|jwt"

Expected Log Messages

Authentication Enabled:

INFO Authentication enabled with 1 provider(s)
INFO Added Azure AD authentication provider
INFO Pre-loading authentication keys for 1 provider(s)

Authentication Disabled:

INFO No AUTH_PROVIDERS configured, authentication disabled
INFO Authentication disabled

Common Error Messages

Authentication Errors

"Missing authorization header"

Error:

Missing authorization header. Authentication is required.

Cause: DuckDB Airport extension not sending JWT token in requests.

Solutions:

sql

-- Ensure you created the secret properly
CREATE SECRET boilstream_auth (
    type airport,
    auth_token 'your-jwt-token-here',
    scope 'grpc+tls://localhost:50051/'
);

-- Verify the secret exists
SHOW ALL SECRETS;

-- Reconnect with correct location
ATTACH 'boilstream' (TYPE AIRPORT, location 'grpc+tls://localhost:50051/');

"JWT token expired"

Error:

JWT token validation failed: Token expired

Solutions:

bash

# Get a fresh token
aws cognito-idp admin-initiate-auth \
    --user-pool-id "eu-west-1_gti5vAfvC" \
    --client-id "7ml47mngu8lrcb198epbkkfi5s" \
    --auth-flow "ADMIN_NO_SRP_AUTH" \
    --auth-parameters USERNAME=testuser,PASSWORD=TempPass123!

# Update DuckDB secret with new token
CREATE OR REPLACE SECRET boilstream_auth (
    type airport,
    auth_token 'new-jwt-token-here',
    scope 'grpc+tls://localhost:50051/'
);

"TLS connection failed"

Error:

IO Error: Flight returned internal error, with message: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))

Solutions:

bash

# Check if BoilStream is running with TLS
./boilstream

# Verify TLS certificates exist
ls -la certs/server.crt certs/server.key

# Set certificate path for DuckDB
export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=certs/server.crt

# Test TLS connection
openssl s_client -connect localhost:50051 -servername localhost

DuckDB Airport Extension Issues

Secret Scope Mismatch

Error: Authentication works for some operations but not others.

Solution:

sql

-- Ensure secret scope matches connection URL exactly
CREATE SECRET boilstream_auth (
    type airport,
    auth_token 'your-jwt-token',
    scope 'grpc+tls://localhost:50051/'  -- Must match ATTACH location
);

-- Connection must use same URL
ATTACH 'boilstream' (TYPE AIRPORT, location 'grpc+tls://localhost:50051/');

Airport Extension Not Found

Error:

Extension "airport" not found

Solution:

sql

-- Install from community repository
INSTALL airport FROM community;
LOAD airport;

-- Or build your self: https://github.com/dforsber/airport

Debug Authentication Flow

Enable detailed logging to see authentication flow:

bash

# Start BoilStream with debug logs
RUST_LOG="boilstream::flight=info,boilstream::auth=debug" ./boilstream

# Check what headers DuckDB is sending
RUST_LOG="boilstream::flight=debug" ./boilstream

Expected debug output:

INFO FlightRPC Headers for do_action:
INFO   ASCII authorization: "Bearer eyJ0eXAiOiJKV1Q..."
INFO   ASCII content-type: "application/grpc"

Quick Fix Checklist

✅ Authentication Setup

[ ] AUTH_PROVIDERS environment variable set
[ ] Provider-specific environment variables configured
[ ] BoilStream started with --features pro-tier
[ ] TLS certificates available in certs/ directory

✅ DuckDB Setup

[ ] Airport extension installed and loaded
[ ] CREATE SECRET with correct JWT token
[ ] Secret scope matches ATTACH location exactly
[ ] GRPC_DEFAULT_SSL_ROOTS_FILE_PATH set for TLS

✅ Debugging

[ ] Check BoilStream logs for authentication messages
[ ] Verify JWT token is not expired
[ ] Test TLS connection with openssl s_client
[ ] Confirm user has required groups/permissions

bash

# Ensure you're sending the Authorization header with the DuckDB Airport extension
duckdb -s "CREATE PERMANENT SECRET airport_autogluon (
    type airport,
    auth_token '$TOKEN',
    scope 'grpc+tls://$HOST:50051/'
  );"

"Token validation failed"

Error:

Token validation failed: Invalid signature

Cause: JWT signature verification failed.

Solutions:

Check JWKS ConnectivityVerify TokenCheck Configuration

bash

# AWS Cognito
curl -v "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ABC123DEF/.well-known/jwks.json"

# Azure AD
curl -v "https://login.microsoftonline.com/tenant-id/discovery/v2.0/keys"

# Google Cloud
curl -v "https://www.googleapis.com/oauth2/v3/certs"

bash

# Decode JWT token to check claims
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq .

# Check token expiration
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.exp, .iat' | xargs -I {} date -d @{}

bash

# Verify environment variables
echo "Cognito Pool: $COGNITO_USER_POOL_ID"
echo "Azure Tenant: $AZURE_TENANT_ID"
echo "GCP Client: $GCP_CLIENT_ID"

Authorization Errors

"Access denied. Insufficient permissions"

Error:

Access denied. Insufficient permissions for this operation.

Cause: User doesn't have required groups, scopes, or permissions.

Diagnosis:

bash

# Check user's groups and scopes in token
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '{
  groups: .groups // ."cognito:groups" // .roles,
  scopes: .scope // .scp,
  user: .sub // .username // .email
}'

# Check BoilStream authorization configuration
env | grep -E "(ADMIN_GROUPS|WRITE_GROUPS|READ_ONLY_GROUPS|REQUIRED_.*_SCOPES)"

Solutions:

bash

# Option 1: Add user to required group (provider-specific)
# AWS Cognito
aws cognito-idp admin-add-user-to-group \
    --user-pool-id "$COGNITO_USER_POOL_ID" \
    --username "user@company.com" \
    --group-name "data-producers"

# Azure AD
az ad group member add --group "data-producers" --member-id "user-object-id"

# Google Workspace (via admin console)
# Add user to boilstream-admins@company.com group

# Option 2: Update BoilStream authorization configuration
export WRITE_GROUPS="$WRITE_GROUPS,new-group-name"

"Admin access denied"

Error:

Admin access denied. Administrative privileges required.

Cause: User trying to access admin operations without admin privileges.

Solution:

bash

# Check if user has admin groups or scopes
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.groups // ."cognito:groups" // .roles'

# Add user to admin group
export ADMIN_GROUPS="your-admin-group,$ADMIN_GROUPS"

# Or use scope-based admin access
export REQUIRED_ADMIN_SCOPES="boilstream:admin"

Provider-Specific Errors

AWS Cognito Issues

"Invalid issuer" with Cognito:

bash

# Check user pool ID and region
aws cognito-idp describe-user-pool --user-pool-id "$COGNITO_USER_POOL_ID"

# Verify issuer in token matches expected format
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.iss'
# Should be: https://cognito-idp.{region}.amazonaws.com/{user-pool-id}

Groups not appearing in token:

bash

# Check if user is in groups
aws cognito-idp admin-list-groups-for-user \
    --user-pool-id "$COGNITO_USER_POOL_ID" \
    --username "user@company.com"

# Verify token type (should be 'access' for groups)
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.token_use'

Azure AD Issues

"Invalid tenant" errors:

bash

# Check tenant ID configuration
echo "Configured: $AZURE_TENANT_ID"
echo "Token tenant: $(echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq -r '.tid')"

# For multi-tenant setup
export AZURE_TENANT_ID="common"
export AZURE_ALLOW_MULTI_TENANT="true"

Groups missing from token:

bash

# Check optional claims configuration
az ad app show --id "$AZURE_CLIENT_ID" --query "optionalClaims"

# Verify group membership
az ad user get-member-groups --id "user@company.com"

# Check if group count exceeds 200 (Azure limit)
az rest --method GET --url "https://graph.microsoft.com/v1.0/me/memberOf" \
    --headers "Authorization=Bearer $TOKEN" | jq '.value | length'

Google Cloud Issues

Domain validation failures:

bash

# Check hosted domain in token
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.hd'

# Verify domain configuration
echo "Required domain: $GCP_REQUIRE_WORKSPACE_DOMAIN"

# For personal Google accounts (not recommended for production)
unset GCP_REQUIRE_WORKSPACE_DOMAIN

STS token issues:

bash

# Check if STS tokens are allowed
echo "STS allowed: $GCP_ALLOW_STS_TOKENS"

# Verify STS token issuer
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '.iss'
# Should be: https://sts.googleapis.com

Debugging Tools

Enable Debug Logging

Enable detailed authentication logging:

bash

# Full authentication debugging
export RUST_LOG="boilstream::auth=debug"

# Provider-specific debugging
export RUST_LOG="boilstream::auth::cognito=debug"
export RUST_LOG="boilstream::auth::azure=debug"
export RUST_LOG="boilstream::auth::gcp=debug"

# Authorization debugging
export RUST_LOG="boilstream::auth::authorization=debug"

# All auth components
export RUST_LOG="boilstream::auth=debug,boilstream::flight=info"

Token Analysis Script

Create a token analysis script:

bash

#!/bin/bash
# save as analyze_token.sh

TOKEN="$1"

if [ -z "$TOKEN" ]; then
    echo "Usage: $0 <jwt_token>"
    exit 1
fi

echo "=== JWT Token Analysis ==="

# Decode header
echo "Header:"
echo "$TOKEN" | cut -d'.' -f1 | base64 -d 2>/dev/null | jq . || echo "Invalid header"

# Decode payload
echo -e "\nPayload:"
echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq . || echo "Invalid payload"

# Check expiration
echo -e "\nExpiration Check:"
EXP=$(echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq -r '.exp // empty')
if [ -n "$EXP" ]; then
    CURRENT=$(date +%s)
    if [ "$EXP" -gt "$CURRENT" ]; then
        echo "✅ Token valid until $(date -d @$EXP)"
    else
        echo "❌ Token expired at $(date -d @$EXP)"
    fi
else
    echo "⚠️  No expiration claim found"
fi

# Extract key claims
echo -e "\nKey Claims:"
echo "$TOKEN" | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '{
    issuer: .iss,
    subject: .sub,
    audience: .aud,
    groups: (.groups // ."cognito:groups" // .roles // "none"),
    scopes: (.scope // .scp // "none"),
    email: (.email // .username // .upn // "none")
}'

Network Connectivity Tests

Test JWKS endpoint connectivity:

bash

#!/bin/bash
# save as test_jwks.sh

test_jwks_endpoint() {
    local url="$1"
    local name="$2"

    echo "Testing $name JWKS endpoint..."

    if curl -s --connect-timeout 5 "$url" | jq '.keys[0].kty' >/dev/null 2>&1; then
        echo "✅ $name JWKS accessible"
    else
        echo "❌ $name JWKS failed"
        curl -v "$url"
    fi
    echo
}

# Test based on configuration
if [ -n "$COGNITO_USER_POOL_ID" ]; then
    test_jwks_endpoint "https://cognito-idp.${COGNITO_REGION:-us-east-1}.amazonaws.com/$COGNITO_USER_POOL_ID/.well-known/jwks.json" "Cognito"
fi

if [ -n "$AZURE_TENANT_ID" ]; then
    test_jwks_endpoint "https://login.microsoftonline.com/$AZURE_TENANT_ID/discovery/v2.0/keys" "Azure AD"
fi

if [ -n "$GCP_CLIENT_ID" ] || [ -n "$GCP_PROJECT_ID" ]; then
    test_jwks_endpoint "https://www.googleapis.com/oauth2/v3/certs" "Google Cloud"
fi

Performance Issues

Slow Authentication

Symptoms: First request takes several seconds to authenticate.

Cause: JWKS key loading on first request.

Solution:

bash

# BoilStream pre-loads keys on startup automatically
# Check startup logs for key preloading
cat boilstream.log | grep -i "preload\|jwks"

# If still slow, check network latency to JWKS endpoints
time curl -s "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_ABC123DEF/.well-known/jwks.json" >/dev/null

Memory Usage

Symptoms: High memory usage with authentication enabled.

Cause: JWKS caching or large numbers of cached tokens.

Diagnosis:

bash

# Check BoilStream memory usage
docker stats boilstream --no-stream

# Enable memory metrics (if available)
export RUST_LOG="boilstream::auth=debug"

Configuration Validation

Environment Variable Checker

bash

#!/bin/bash
# save as check_auth_config.sh

echo "=== BoilStream Authentication Configuration Check ==="

# Check if authentication is enabled
if [ -z "$AUTH_PROVIDERS" ]; then
    echo "❌ AUTH_PROVIDERS not set - authentication disabled"
    exit 0
else
    echo "✅ AUTH_PROVIDERS: $AUTH_PROVIDERS"
fi

# Check each provider
IFS=',' read -ra PROVIDERS <<< "$AUTH_PROVIDERS"
for provider in "${PROVIDERS[@]}"; do
    provider=$(echo "$provider" | xargs) # trim whitespace
    echo -e "\n--- Checking $provider provider ---"

    case "$provider" in
        "cognito")
            [ -n "$COGNITO_USER_POOL_ID" ] && echo "✅ COGNITO_USER_POOL_ID set" || echo "❌ COGNITO_USER_POOL_ID missing"
            [ -n "$COGNITO_REGION" ] && echo "✅ COGNITO_REGION: $COGNITO_REGION" || echo "⚠️ COGNITO_REGION not set (using default)"
            ;;
        "azure-ad"|"azure"|"entra-id")
            [ -n "$AZURE_TENANT_ID" ] && echo "✅ AZURE_TENANT_ID: $AZURE_TENANT_ID" || echo "❌ AZURE_TENANT_ID missing"
            [ -n "$AZURE_CLIENT_ID" ] && echo "✅ AZURE_CLIENT_ID set" || echo "❌ AZURE_CLIENT_ID missing"
            ;;
        "gcp"|"google"|"google-cloud")
            if [ -n "$GCP_CLIENT_ID" ] || [ -n "$GCP_PROJECT_ID" ]; then
                [ -n "$GCP_CLIENT_ID" ] && echo "✅ GCP_CLIENT_ID set"
                [ -n "$GCP_PROJECT_ID" ] && echo "✅ GCP_PROJECT_ID set"
            else
                echo "❌ Either GCP_CLIENT_ID or GCP_PROJECT_ID required"
            fi
            ;;
        *)
            echo "❌ Unknown provider: $provider"
            ;;
    esac
done

# Check authorization configuration
echo -e "\n--- Authorization Configuration ---"
auth_vars=("ADMIN_GROUPS" "WRITE_GROUPS" "READ_ONLY_GROUPS" "REQUIRED_READ_SCOPES" "REQUIRED_WRITE_SCOPES" "REQUIRED_ADMIN_SCOPES")

for var in "${auth_vars[@]}"; do
    if [ -n "${!var}" ]; then
        echo "✅ $var: ${!var}"
    else
        echo "⚠️ $var not set"
    fi
done

Test Authentication Flow

bash

#!/bin/bash
# save as test_auth_flow.sh

# Requires TOKEN environment variable
if [ -z "$TOKEN" ]; then
    echo "Please set TOKEN environment variable with a valid JWT token"
    exit 1
fi

echo "=== Testing BoilStream Authentication Flow ==="

# Test authentication
echo "Testing authentication..."
response=$(curl -s -w "%{http_code}" -H "Authorization: Bearer $TOKEN" \
    "http://localhost:50051/api/health" 2>/dev/null | tail -n1)

if [ "$response" = "200" ]; then
    echo "✅ Authentication successful"
elif [ "$response" = "401" ]; then
    echo "❌ Authentication failed (401 Unauthorized)"
elif [ "$response" = "403" ]; then
    echo "❌ Authorization failed (403 Forbidden)"
else
    echo "❌ Unexpected response: $response"
fi

# Test admin access (if applicable)
echo "Testing admin access..."
admin_response=$(curl -s -w "%{http_code}" -H "Authorization: Bearer $TOKEN" \
    "http://localhost:50160/admin/status" 2>/dev/null | tail -n1)

case "$admin_response" in
    "200") echo "✅ Admin access granted" ;;
    "401") echo "❌ Admin authentication failed" ;;
    "403") echo "❌ Admin authorization denied" ;;
    *) echo "⚠️ Admin endpoint response: $admin_response" ;;
esac

Getting Help

Log Collection

When reporting issues, collect these logs:

bash

# BoilStream logs with auth debugging
export RUST_LOG="boilstream::auth=debug"
./boilstream > boilstream-auth.log 2>&1

# Token information (remove sensitive data before sharing)
echo "$TOKEN" | cut -d'.' -f2 | base64 -d | jq '{
    iss: .iss,
    aud: .aud,
    exp: .exp,
    token_use: .token_use,
    has_groups: (.groups // ."cognito:groups" // .roles | length // 0),
    has_scopes: (.scope // .scp | if type == "string" then split(" ") | length else length // 0 end)
}' > token-info.json

# Environment configuration (remove secrets)
env | grep -E "AUTH_|COGNITO_|AZURE_|GCP_" | sed 's/=.*/=***/' > auth-config.txt

Support Information

When seeking help, provide:

Error Message: Exact error message from BoilStream logs
Provider Type: Which identity provider you're using
Token Claims: Sanitized token claims (use script above)
Configuration: Environment variables (without secrets)
Network: Results of JWKS connectivity tests

Community Resources

GitHub Issues: BoilStream Issues
Documentation: BoilStream Docs
Provider Documentation:

Environment Variables - Complete configuration reference
AWS Cognito Setup - Provider-specific troubleshooting
Azure AD Setup - Azure-specific issues
Google Cloud Setup - GCP-specific problems

Authentication Troubleshooting ​

Quick Diagnosis ​

Is Authentication Working? ​

Expected Log Messages ​

Common Error Messages ​

Authentication Errors ​

"Missing authorization header" ​

"JWT token expired" ​

"TLS connection failed" ​

DuckDB Airport Extension Issues ​

Secret Scope Mismatch ​

Airport Extension Not Found ​

Debug Authentication Flow ​

Quick Fix Checklist ​

✅ Authentication Setup ​

✅ DuckDB Setup ​

✅ Debugging ​

"Token validation failed" ​

Authorization Errors ​

"Access denied. Insufficient permissions" ​

"Admin access denied" ​

Provider-Specific Errors ​

AWS Cognito Issues ​

Azure AD Issues ​

Google Cloud Issues ​

Debugging Tools ​

Enable Debug Logging ​

Token Analysis Script ​

Network Connectivity Tests ​

Performance Issues ​

Slow Authentication ​

Memory Usage ​

Configuration Validation ​

Environment Variable Checker ​

Test Authentication Flow ​

Getting Help ​

Log Collection ​

Support Information ​

Community Resources ​

Related Topics ​

Authentication Troubleshooting

Quick Diagnosis

Is Authentication Working?

Expected Log Messages

Common Error Messages

Authentication Errors

"Missing authorization header"

"JWT token expired"

"TLS connection failed"

DuckDB Airport Extension Issues

Secret Scope Mismatch

Airport Extension Not Found

Debug Authentication Flow

Quick Fix Checklist

✅ Authentication Setup

✅ DuckDB Setup

✅ Debugging

"Token validation failed"

Authorization Errors

"Access denied. Insufficient permissions"

"Admin access denied"

Provider-Specific Errors

AWS Cognito Issues

Azure AD Issues

Google Cloud Issues

Debugging Tools

Enable Debug Logging

Token Analysis Script

Network Connectivity Tests

Performance Issues

Slow Authentication

Memory Usage

Configuration Validation

Environment Variable Checker

Test Authentication Flow

Getting Help

Log Collection

Support Information

Community Resources

Related Topics