metadata-agregator/docs/research/lidarr-metadata-api/analysis/DEPLOYMENT.md

# Lidarr Metadata API - Deployment

## Deployment Overview

The Lidarr Metadata API is designed for containerized deployment using Docker and Docker Compose. The architecture consists of 8+ containers working together to provide a complete metadata aggregation service.

## Container Architecture

### Container Inventory

| Container | Image | Purpose | CPU | Memory | Storage |
|-----------|-------|---------|-----|--------|---------|
| **musicbrainz** | ghcr.io/lidarr/mb-postgres:1.0.10 | MusicBrainz database | 2+ cores | 4GB+ | 100GB+ |
| **solr** | ghcr.io/lidarr/mb-solr:3.3.1.9 | Search index | 2 cores | 2GB | 8GB |
| **redis** | redis:6-alpine | Cache + rate limiting | 1 core | 512MB | None |
| **rabbitmq** | rabbitmq:3-management | Message queue | 1 core | 1GB | 1GB |
| **indexer** | ghcr.io/lidarr/mb-sir:latest | Search index rebuilder | 1 core | 512MB | None |
| **api-v0.3** | ghcr.io/lidarr/lidarrapi.metadata:v0.3 | Stable API version | 2 cores | 1GB | None |
| **api-testing** | ghcr.io/lidarr/lidarrapi.metadata:testing | Development API version | 2 cores | 1GB | None |
| **crawler** | ghcr.io/lidarr/lidarrapi.metadata:v0.3 | Background cache warmer | 1 core | 512MB | None |

**Total minimum resources**: 12 CPU cores, 11.5GB RAM, 110GB storage

### Docker Compose Files

The project uses multiple compose files for different environments:

#### 1. docker-compose.yml (Base)

**Purpose**: Core infrastructure services

**Services**:
- `musicbrainz`: PostgreSQL with MusicBrainz schema
- `solr`: Search index
- `redis`: Cache
- `rabbitmq`: Message queue

**File location**: `docker-compose.yml`

**Key configuration**:
```yaml
version: '3.8'

services:
  musicbrainz:
    image: ghcr.io/lidarr/mb-postgres:1.0.10
    container_name: musicbrainz
    environment:
      POSTGRES_USER: abc
      POSTGRES_PASSWORD: abc
      POSTGRES_DB: musicbrainz_db
    volumes:
      - musicbrainz-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    restart: unless-stopped
    shm_size: 2gb
  
  solr:
    image: ghcr.io/lidarr/mb-solr:3.3.1.9
    container_name: solr
    environment:
      SOLR_HEAP: 2g
    volumes:
      - solr-data:/var/solr
    ports:
      - "8983:8983"
    restart: unless-stopped
  
  redis:
    image: redis:6-alpine
    container_name: redis
    command: redis-server --maxmemory 512mb --maxmemory-policy allkeys-lfu
    ports:
      - "6379:6379"
    restart: unless-stopped
  
  rabbitmq:
    image: rabbitmq:3-management
    container_name: rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: abc
      RABBITMQ_DEFAULT_PASS: abc
    ports:
      - "5672:5672"
      - "15672:15672"
    restart: unless-stopped

volumes:
  musicbrainz-data:
  solr-data:
```

#### 2. docker-compose.dev.yml (Development)

**Purpose**: Development environment with exposed ports

**Extends**: `docker-compose.yml`

**Additional configuration**:
```yaml
version: '3.8'

services:
  api:
    image: ghcr.io/lidarr/lidarrapi.metadata:testing
    container_name: lidarr-metadata-api
    environment:
      LIDARR_METADATA_CONFIG: lidarrmetadata.config.DevelopmentConfig
      DATABASE__HOST: musicbrainz
      CACHE__REDIS_URL: redis://redis:6379/0
      SOLR__URL: http://solr:8983/solr
      RABBITMQ__HOST: rabbitmq
    ports:
      - "5001:5001"
    depends_on:
      - musicbrainz
      - solr
      - redis
      - rabbitmq
    restart: unless-stopped
  
  indexer:
    image: ghcr.io/lidarr/mb-sir:latest
    container_name: search-indexer
    environment:
      DATABASE_HOST: musicbrainz
      SOLR_URL: http://solr:8983/solr
      RABBITMQ_HOST: rabbitmq
    depends_on:
      - musicbrainz
      - solr
      - rabbitmq
    restart: unless-stopped
```

**Usage**:
```bash
docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d
```

#### 3. docker-compose.prod.yml (Production)

**Purpose**: Production deployment with dual API versions

**Extends**: `docker-compose.yml`

**Additional configuration**:
```yaml
version: '3.8'

services:
  api-v0.3:
    image: ghcr.io/lidarr/lidarrapi.metadata:v0.3
    container_name: lidarr-metadata-api-v0.3
    environment:
      LIDARR_METADATA_CONFIG: lidarrmetadata.config.ProductionConfig
      DATABASE__HOST: musicbrainz
      CACHE__REDIS_URL: redis://redis:6379/0
      SOLR__URL: http://solr:8983/solr
      RABBITMQ__HOST: rabbitmq
      SENTRY_DSN: ${SENTRY_DSN}
      CLOUDFLARE_ZONE_ID: ${CLOUDFLARE_ZONE_ID}
      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
      FANART_API_KEY: ${FANART_API_KEY}
      SPOTIFY_CLIENT_ID: ${SPOTIFY_CLIENT_ID}
      SPOTIFY_CLIENT_SECRET: ${SPOTIFY_CLIENT_SECRET}
      LASTFM_API_KEY: ${LASTFM_API_KEY}
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '2'
          memory: 1G
        reservations:
          cpus: '1'
          memory: 512M
    depends_on:
      - musicbrainz
      - solr
      - redis
      - rabbitmq
    restart: unless-stopped
  
  api-testing:
    image: ghcr.io/lidarr/lidarrapi.metadata:testing
    container_name: lidarr-metadata-api-testing
    environment:
      LIDARR_METADATA_CONFIG: lidarrmetadata.config.ProductionConfig
      DATABASE__HOST: musicbrainz
      CACHE__REDIS_URL: redis://redis:6379/0
      SOLR__URL: http://solr:8983/solr
      RABBITMQ__HOST: rabbitmq
      SENTRY_DSN: ${SENTRY_DSN}
      CLOUDFLARE_ZONE_ID: ${CLOUDFLARE_ZONE_ID}
      CLOUDFLARE_API_TOKEN: ${CLOUDFLARE_API_TOKEN}
      FANART_API_KEY: ${FANART_API_KEY}
      SPOTIFY_CLIENT_ID: ${SPOTIFY_CLIENT_ID}
      SPOTIFY_CLIENT_SECRET: ${SPOTIFY_CLIENT_SECRET}
      LASTFM_API_KEY: ${LASTFM_API_KEY}
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: '2'
          memory: 1G
    depends_on:
      - musicbrainz
      - solr
      - redis
      - rabbitmq
    restart: unless-stopped
  
  indexer:
    image: ghcr.io/lidarr/mb-sir:latest
    container_name: search-indexer
    environment:
      DATABASE_HOST: musicbrainz
      SOLR_URL: http://solr:8983/solr
      RABBITMQ_HOST: rabbitmq
    depends_on:
      - musicbrainz
      - solr
      - rabbitmq
    restart: unless-stopped
```

**Usage**:
```bash
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```

#### 4. docker-compose.crawler.yml (Crawler)

**Purpose**: Background cache warming

**Extends**: `docker-compose.yml`

**Additional configuration**:
```yaml
version: '3.8'

services:
  crawler:
    image: ghcr.io/lidarr/lidarrapi.metadata:v0.3
    container_name: lidarr-metadata-crawler
    command: lidarr-metadata-crawler
    environment:
      LIDARR_METADATA_CONFIG: lidarrmetadata.config.ProductionConfig
      DATABASE__HOST: musicbrainz
      CACHE__REDIS_URL: redis://redis:6379/0
      SOLR__URL: http://solr:8983/solr
      FANART_API_KEY: ${FANART_API_KEY}
      SPOTIFY_CLIENT_ID: ${SPOTIFY_CLIENT_ID}
      SPOTIFY_CLIENT_SECRET: ${SPOTIFY_CLIENT_SECRET}
      CRAWLER_INTERVAL: 3600
      CRAWLER_BATCH_SIZE: 100
    depends_on:
      - musicbrainz
      - redis
    restart: unless-stopped
```

**Usage**:
```bash
docker-compose -f docker-compose.yml -f docker-compose.crawler.yml up -d
```

## Dockerfile

### Multi-Stage Build

**File location**: `Dockerfile`

**Build stages**:

#### Stage 1: Dependencies

```dockerfile
FROM python:3.9-alpine AS builder

# Install build dependencies
RUN apk add --no-cache \
    gcc \
    musl-dev \
    postgresql-dev \
    libffi-dev \
    openssl-dev

# Install Poetry
RUN pip install poetry==1.1.13

# Copy dependency files
WORKDIR /app
COPY pyproject.toml poetry.lock ./

# Install dependencies
RUN poetry config virtualenvs.create false \
    && poetry install --no-dev --no-interaction --no-ansi
```

#### Stage 2: Runtime

```dockerfile
FROM python:3.9-alpine

# Install runtime dependencies
RUN apk add --no-cache \
    postgresql-libs \
    libffi \
    openssl

# Create non-root user
RUN addgroup -g 1000 lidarr \
    && adduser -D -u 1000 -G lidarr lidarr

# Copy dependencies from builder
COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy application code
WORKDIR /app
COPY --chown=lidarr:lidarr . .

# Switch to non-root user
USER lidarr

# Expose port
EXPOSE 5001

# Default command
CMD ["gunicorn", "-w", "1", "-k", "uvicorn.workers.UvicornWorker", "--bind", "0.0.0.0:5001", "--access-logfile", "-", "lidarrmetadata.server:app"]
```

### Build Arguments

```dockerfile
ARG VERSION=unknown
ARG GIT_SHA=unknown
ARG BUILD_DATE=unknown

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.revision="${GIT_SHA}"
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.source="https://github.com/Lidarr/LidarrAPI.Metadata"
```

### Build Command

```bash
docker build \
  --build-arg VERSION=10.0.0.0 \
  --build-arg GIT_SHA=$(git rev-parse HEAD) \
  --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
  -t ghcr.io/lidarr/lidarrapi.metadata:v0.3 \
  .
```

## Initialization Process

### 10-Step Initialization

The deployment requires a specific initialization sequence:

#### Step 1: Start PostgreSQL

```bash
docker-compose up -d musicbrainz
```

**Wait for**: PostgreSQL ready to accept connections

**Verification**:
```bash
docker-compose exec musicbrainz pg_isready -U abc
```

#### Step 2: Fetch MusicBrainz Dump

**Options**:

**Option A: Full dump** (recommended for production):
```bash
docker-compose exec musicbrainz /scripts/fetch-dump.sh
```

**Duration**: 4-8 hours (depends on network speed)

**Size**: ~30GB compressed, ~100GB uncompressed

**Option B: Sample dump** (for development):
```bash
docker-compose exec musicbrainz /scripts/fetch-sample-dump.sh
```

**Duration**: 30-60 minutes

**Size**: ~1GB compressed, ~5GB uncompressed

#### Step 3: Setup AMQP Triggers

```bash
docker-compose exec musicbrainz /scripts/setup-amqp.sh
```

**Purpose**: Create database triggers that publish changes to RabbitMQ

**Verification**:
```sql
SELECT * FROM pg_trigger WHERE tgname LIKE 'amqp_%';
```

#### Step 4: Create Custom Indices

```bash
docker-compose exec musicbrainz psql -U abc -d musicbrainz_db -f /scripts/create-indices.sql
```

**Indices created**:
- `idx_artist_last_updated`
- `idx_release_group_last_updated`
- `idx_release_last_updated`
- `idx_l_artist_url_last_updated`
- `idx_cover_art_date_updated`

**Duration**: 30-60 minutes

#### Step 5: Start RabbitMQ

```bash
docker-compose up -d rabbitmq
```

**Wait for**: RabbitMQ management interface available

**Verification**:
```bash
curl http://localhost:15672/api/overview -u abc:abc
```

#### Step 6: Start Solr

```bash
docker-compose up -d solr
```

**Wait for**: Solr cores created

**Verification**:
```bash
curl http://localhost:8983/solr/admin/cores?action=STATUS
```

#### Step 7: Load Search Indices

```bash
docker-compose run indexer rebuild-artist
docker-compose run indexer rebuild-album
```

**Duration**: 4-8 hours for full dataset

**Verification**:
```bash
curl "http://localhost:8983/solr/artist/select?q=*:*&rows=0" | jq '.response.numFound'
```

#### Step 8: Enable Replication

```bash
docker-compose exec musicbrainz /scripts/enable-replication.sh
```

**Purpose**: Start hourly replication from MusicBrainz master

**Verification**:
```sql
SELECT * FROM replication_control;
```

#### Step 9: Start Search Indexer

```bash
docker-compose up -d indexer
```

**Purpose**: Real-time search index updates via RabbitMQ

**Verification**:
```bash
docker-compose logs -f indexer
```

#### Step 10: Start API and Crawler

```bash
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d api-v0.3 api-testing
docker-compose -f docker-compose.yml -f docker-compose.crawler.yml up -d crawler
```

**Verification**:
```bash
curl http://localhost:5001/
```

### Initialization Script

**Complete initialization script**:

```bash
#!/bin/bash
set -e

echo "Step 1: Starting PostgreSQL..."
docker-compose up -d musicbrainz
sleep 30

echo "Step 2: Fetching MusicBrainz dump..."
docker-compose exec musicbrainz /scripts/fetch-dump.sh

echo "Step 3: Setting up AMQP triggers..."
docker-compose exec musicbrainz /scripts/setup-amqp.sh

echo "Step 4: Creating custom indices..."
docker-compose exec musicbrainz psql -U abc -d musicbrainz_db -f /scripts/create-indices.sql

echo "Step 5: Starting RabbitMQ..."
docker-compose up -d rabbitmq
sleep 10

echo "Step 6: Starting Solr..."
docker-compose up -d solr
sleep 10

echo "Step 7: Loading search indices..."
docker-compose run indexer rebuild-artist
docker-compose run indexer rebuild-album

echo "Step 8: Enabling replication..."
docker-compose exec musicbrainz /scripts/enable-replication.sh

echo "Step 9: Starting search indexer..."
docker-compose up -d indexer

echo "Step 10: Starting API and crawler..."
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d api-v0.3 api-testing
docker-compose -f docker-compose.yml -f docker-compose.crawler.yml up -d crawler

echo "Initialization complete!"
```

## CI/CD Pipeline

### Azure Pipelines

**File location**: `azure-pipelines.yml`

**Trigger**: Commits to `master` or `develop` branches

**Stages**:

#### 1. Build and Test

```yaml
stages:
  - stage: Build
    jobs:
      - job: BuildAndTest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.9'
          
          - script: |
              pip install poetry
              poetry install
            displayName: 'Install dependencies'
          
          # Tests commented out
          # - script: |
          #     poetry run pytest tests/
          #   displayName: 'Run tests'
          
          - task: SonarCloudPrepare@1
            inputs:
              SonarCloud: 'SonarCloud'
              organization: 'lidarr'
              scannerMode: 'CLI'
              configMode: 'manual'
              cliProjectKey: 'Lidarr_LidarrAPI.Metadata'
              cliProjectName: 'LidarrAPI.Metadata'
          
          - task: SonarCloudAnalyze@1
          
          - task: SonarCloudPublish@1
            inputs:
              pollingTimeoutSec: '300'
```

#### 2. Docker Build and Push

```yaml
  - stage: Docker
    dependsOn: Build
    jobs:
      - job: BuildAndPush
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: Docker@2
            inputs:
              containerRegistry: 'GitHub Container Registry'
              repository: 'lidarr/lidarrapi.metadata'
              command: 'buildAndPush'
              Dockerfile: 'Dockerfile'
              tags: |
                $(Build.SourceBranchName)
                $(Build.SourceVersion)
                latest
              buildArgs: |
                VERSION=10.0.0.0
                GIT_SHA=$(Build.SourceVersion)
                BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
```

#### 3. Sentry Release

```yaml
  - stage: Sentry
    dependsOn: Docker
    jobs:
      - job: CreateRelease
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: |
              curl -sL https://sentry.io/get-cli/ | bash
              sentry-cli releases new "lidarr-metadata@$(Build.SourceVersion)"
              sentry-cli releases set-commits "lidarr-metadata@$(Build.SourceVersion)" --auto
              sentry-cli releases finalize "lidarr-metadata@$(Build.SourceVersion)"
            env:
              SENTRY_AUTH_TOKEN: $(SENTRY_AUTH_TOKEN)
              SENTRY_ORG: lidarr
              SENTRY_PROJECT: lidarr-metadata
            displayName: 'Create Sentry release'
```

### GitHub Actions (Alternative)

**File location**: `.github/workflows/build.yml`

```yaml
name: Build and Deploy

on:
  push:
    branches: [master, develop]
  pull_request:
    branches: [master]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'
      
      - name: Install dependencies
        run: |
          pip install poetry
          poetry install
      
      - name: Run tests
        run: poetry run pytest tests/
      
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
      
      - name: Build Docker image
        run: |
          docker build \
            --build-arg VERSION=10.0.0.0 \
            --build-arg GIT_SHA=${{ github.sha }} \
            --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
            -t ghcr.io/lidarr/lidarrapi.metadata:${{ github.ref_name }} \
            .
      
      - name: Push to GitHub Container Registry
        run: |
          echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker push ghcr.io/lidarr/lidarrapi.metadata:${{ github.ref_name }}
      
      - name: Create Sentry release
        run: |
          curl -sL https://sentry.io/get-cli/ | bash
          sentry-cli releases new "lidarr-metadata@${{ github.sha }}"
          sentry-cli releases set-commits "lidarr-metadata@${{ github.sha }}" --auto
          sentry-cli releases finalize "lidarr-metadata@${{ github.sha }}"
        env:
          SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
          SENTRY_ORG: lidarr
          SENTRY_PROJECT: lidarr-metadata
```

## Infrastructure Requirements

### Minimum Requirements

| Component | CPU | Memory | Storage | Network |
|-----------|-----|--------|---------|---------|
| **MusicBrainz DB** | 2 cores | 4GB | 100GB SSD | 1Gbps |
| **Solr** | 2 cores | 2GB | 8GB SSD | 1Gbps |
| **Redis** | 1 core | 512MB | None | 1Gbps |
| **RabbitMQ** | 1 core | 1GB | 1GB SSD | 1Gbps |
| **API (x2)** | 4 cores | 2GB | None | 1Gbps |
| **Indexer** | 1 core | 512MB | None | 1Gbps |
| **Crawler** | 1 core | 512MB | None | 1Gbps |
| **Total** | **12 cores** | **11.5GB** | **110GB** | **1Gbps** |

### Recommended Production Requirements

| Component | CPU | Memory | Storage | Network |
|-----------|-----|--------|---------|---------|
| **MusicBrainz DB** | 4 cores | 8GB | 200GB NVMe | 10Gbps |
| **Solr** | 4 cores | 4GB | 16GB NVMe | 10Gbps |
| **Redis** | 2 cores | 1GB | None | 10Gbps |
| **RabbitMQ** | 2 cores | 2GB | 2GB SSD | 10Gbps |
| **API (x4)** | 8 cores | 4GB | None | 10Gbps |
| **Indexer** | 2 cores | 1GB | None | 10Gbps |
| **Crawler** | 2 cores | 1GB | None | 10Gbps |
| **Total** | **24 cores** | **21GB** | **220GB** | **10Gbps** |

### Cloud Provider Recommendations

#### AWS

**Instance types**:
- **Database**: r6g.xlarge (4 vCPU, 32GB RAM)
- **Solr**: c6g.xlarge (4 vCPU, 8GB RAM)
- **API**: c6g.large (2 vCPU, 4GB RAM) x 2
- **Other**: t4g.small (2 vCPU, 2GB RAM) x 3

**Storage**:
- **Database**: gp3 200GB (3000 IOPS, 125 MB/s)
- **Solr**: gp3 20GB (3000 IOPS)

**Estimated cost**: $400-600/month

#### Google Cloud

**Instance types**:
- **Database**: n2-highmem-4 (4 vCPU, 32GB RAM)
- **Solr**: n2-standard-4 (4 vCPU, 16GB RAM)
- **API**: n2-standard-2 (2 vCPU, 8GB RAM) x 2
- **Other**: e2-small (2 vCPU, 2GB RAM) x 3

**Storage**:
- **Database**: pd-ssd 200GB
- **Solr**: pd-ssd 20GB

**Estimated cost**: $450-650/month

#### DigitalOcean

**Droplets**:
- **Database**: Memory-Optimized 8GB ($80/month)
- **Solr**: General Purpose 4GB ($48/month)
- **API**: General Purpose 2GB ($24/month) x 2
- **Other**: Basic 2GB ($18/month) x 3

**Volumes**:
- **Database**: 200GB SSD ($20/month)
- **Solr**: 20GB SSD ($2/month)

**Estimated cost**: $250-350/month

## Monitoring and Logging

### Health Checks

**Docker Compose health checks**:

```yaml
services:
  api-v0.3:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5001/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
  
  musicbrainz:
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "abc"]
      interval: 30s
      timeout: 5s
      retries: 3
  
  solr:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8983/solr/admin/ping"]
      interval: 30s
      timeout: 5s
      retries: 3
  
  redis:
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 3s
      retries: 3
```

### Logging

**Centralized logging with Docker**:

```yaml
services:
  api-v0.3:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
```

**Log aggregation with Loki** (optional):

```yaml
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki
  
  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
```

### Metrics

**Prometheus scraping**:

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
  
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
```

## Backup and Recovery

### Database Backup

**PostgreSQL backup script**:

```bash
#!/bin/bash
BACKUP_DIR=/backups
DATE=$(date +%Y%m%d_%H%M%S)

# Backup MusicBrainz database
docker-compose exec -T musicbrainz pg_dump -U abc musicbrainz_db | gzip > $BACKUP_DIR/musicbrainz_$DATE.sql.gz

# Backup cache database
docker-compose exec -T musicbrainz pg_dump -U abc lm_cache_db | gzip > $BACKUP_DIR/cache_$DATE.sql.gz

# Cleanup old backups (keep 7 days)
find $BACKUP_DIR -name "*.sql.gz" -mtime +7 -delete
```

**Cron schedule**:
```cron
0 2 * * * /scripts/backup.sh
```

### Solr Backup

**Solr snapshot**:

```bash
curl "http://localhost:8983/solr/artist/replication?command=backup&location=/var/solr/backups&name=artist_$(date +%Y%m%d)"
curl "http://localhost:8983/solr/release-group/replication?command=backup&location=/var/solr/backups&name=album_$(date +%Y%m%d)"
```

### Recovery

**Database restore**:

```bash
# Stop services
docker-compose down

# Restore database
gunzip -c /backups/musicbrainz_20250428.sql.gz | docker-compose exec -T musicbrainz psql -U abc musicbrainz_db

# Restart services
docker-compose up -d
```

## Security Hardening

### Production Security Checklist

- [ ] Change default credentials (abc/abc)
- [ ] Use strong passwords for all services
- [ ] Enable TLS for PostgreSQL connections
- [ ] Enable TLS for Redis connections
- [ ] Use secrets management (Docker Secrets, Vault)
- [ ] Restrict network access (firewall rules)
- [ ] Enable API authentication
- [ ] Use read-only database user for API
- [ ] Enable Sentry error tracking
- [ ] Configure rate limiting
- [ ] Use HTTPS reverse proxy (Nginx, Traefik)
- [ ] Enable CORS restrictions
- [ ] Implement API key rotation
- [ ] Regular security updates
- [ ] Monitor for vulnerabilities (Snyk, Dependabot)

### Secrets Management

**Docker Secrets** (Swarm mode):

```yaml
services:
  api-v0.3:
    secrets:
      - db_password
      - redis_password
      - fanart_api_key
      - spotify_client_secret
    environment:
      DATABASE__PASSWORD_FILE: /run/secrets/db_password
      CACHE__REDIS_PASSWORD_FILE: /run/secrets/redis_password
      FANART_API_KEY_FILE: /run/secrets/fanart_api_key
      SPOTIFY_CLIENT_SECRET_FILE: /run/secrets/spotify_client_secret

secrets:
  db_password:
    external: true
  redis_password:
    external: true
  fanart_api_key:
    external: true
  spotify_client_secret:
    external: true
```

## Scaling Strategies

### Horizontal Scaling

**API instances**:

```yaml
services:
  api-v0.3:
    deploy:
      replicas: 4
      update_config:
        parallelism: 2
        delay: 10s
      restart_policy:
        condition: on-failure
```

**Load balancing** (Nginx):

```nginx
upstream lidarr_metadata_api {
    least_conn;
    server api-v0.3-1:5001;
    server api-v0.3-2:5001;
    server api-v0.3-3:5001;
    server api-v0.3-4:5001;
}

server {
    listen 80;
    server_name api.lidarr.audio;
    
    location / {
        proxy_pass http://lidarr_metadata_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}
```

### Vertical Scaling

**Increase worker count**:

```yaml
services:
  api-v0.3:
    command: gunicorn -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:5001 lidarrmetadata.server:app
```

**Increase database connections**:

```python
DATABASE = {
    'min_pool_size': 20,
    'max_pool_size': 100
}
```

## Conclusion

The deployment architecture demonstrates production-ready containerization with:

- **10-step initialization** for complex setup
- **Multi-environment support** (dev, prod, crawler)
- **Dual-version deployment** for safe rollouts
- **CI/CD integration** with Azure Pipelines
- **Comprehensive monitoring** with health checks and metrics
- **Backup and recovery** procedures
- **Security hardening** recommendations
- **Horizontal and vertical scaling** strategies

The Docker Compose approach makes deployment straightforward while maintaining flexibility for different environments.