feat: initial implementation of metadata aggregator
- gRPC service with MusicBrainz provider - PostgreSQL schema with migrations - Service layer with database-first caching - Repository pattern for data access - YAML configuration support - Research documentation for 17 music metadata projects
This commit is contained in:
@@ -0,0 +1,922 @@
|
||||
# Melodee: Deployment Analysis
|
||||
|
||||
## Deployment Strategy Overview
|
||||
|
||||
Melodee provides Docker-based deployment with multi-stage builds, Docker Compose orchestration, and automatic database migrations. The deployment architecture prioritizes ease of setup for self-hosted environments while supporting advanced configurations for production deployments.
|
||||
|
||||
Key deployment features:
|
||||
- **Docker multi-stage build**: Optimized image size and security
|
||||
- **Docker Compose**: Single-command deployment with PostgreSQL
|
||||
- **Automatic migrations**: Database schema updates on container startup
|
||||
- **12 persistent volumes**: Data persistence across container restarts
|
||||
- **Raspberry Pi support**: ARM64 compatibility for low-power hardware
|
||||
- **Podman compatibility**: Rootless container runtime support
|
||||
|
||||
## Docker Architecture
|
||||
|
||||
### Multi-Stage Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Build stage
|
||||
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
|
||||
WORKDIR /src
|
||||
|
||||
# Copy project files
|
||||
COPY ["Melodee.Web/Melodee.Web.csproj", "Melodee.Web/"]
|
||||
COPY ["Melodee.Data/Melodee.Data.csproj", "Melodee.Data/"]
|
||||
COPY ["Melodee.Core/Melodee.Core.csproj", "Melodee.Core/"]
|
||||
|
||||
# Restore dependencies
|
||||
RUN dotnet restore "Melodee.Web/Melodee.Web.csproj"
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build application
|
||||
WORKDIR "/src/Melodee.Web"
|
||||
RUN dotnet build "Melodee.Web.csproj" -c Release -o /app/build
|
||||
|
||||
# Publish application
|
||||
RUN dotnet publish "Melodee.Web.csproj" -c Release -o /app/publish /p:UseAppHost=false
|
||||
|
||||
# Runtime stage
|
||||
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
|
||||
WORKDIR /app
|
||||
|
||||
# Install FFmpeg for transcoding
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy published application
|
||||
COPY --from=build /app/publish .
|
||||
|
||||
# Copy entrypoint script
|
||||
COPY entrypoint.sh .
|
||||
RUN chmod +x entrypoint.sh
|
||||
|
||||
# Expose port
|
||||
EXPOSE 5000
|
||||
|
||||
# Set entrypoint
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
```
|
||||
|
||||
**Multi-Stage Benefits**:
|
||||
1. **Smaller image size**: Runtime image excludes SDK (saves ~500 MB)
|
||||
2. **Faster deployments**: Smaller images transfer and start faster
|
||||
3. **Security**: No build tools in production image
|
||||
4. **Layer caching**: Dependencies cached separately from source code
|
||||
|
||||
**Image Size Comparison**:
|
||||
- Single-stage (with SDK): ~1.2 GB
|
||||
- Multi-stage (runtime only): ~700 MB
|
||||
- Savings: ~500 MB (42% reduction)
|
||||
|
||||
### Entrypoint Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "Melodee v1.8.0 starting..."
|
||||
|
||||
# Wait for PostgreSQL to be ready
|
||||
echo "Waiting for PostgreSQL..."
|
||||
until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$POSTGRES_HOST" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c '\q' 2>/dev/null; do
|
||||
echo "PostgreSQL is unavailable - sleeping"
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "PostgreSQL is ready"
|
||||
|
||||
# Run database migrations
|
||||
echo "Applying database migrations..."
|
||||
dotnet ef database update --project /app/Melodee.Data.dll --no-build
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Migration failed, exiting..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Migrations applied successfully"
|
||||
|
||||
# Start application
|
||||
echo "Starting Melodee..."
|
||||
exec dotnet Melodee.Web.dll
|
||||
```
|
||||
|
||||
**Entrypoint Responsibilities**:
|
||||
1. **Database readiness check**: Waits for PostgreSQL before starting
|
||||
2. **Automatic migrations**: Applies schema changes on startup
|
||||
3. **Error handling**: Exits if migrations fail
|
||||
4. **Process replacement**: `exec` replaces shell with .NET process for proper signal handling
|
||||
|
||||
**Signal Handling**:
|
||||
The `exec` command is critical for graceful shutdown. Without it:
|
||||
- Docker sends SIGTERM to shell process
|
||||
- Shell doesn't forward signal to .NET process
|
||||
- .NET process killed with SIGKILL after timeout
|
||||
- No graceful shutdown (connections dropped, jobs interrupted)
|
||||
|
||||
With `exec`:
|
||||
- Docker sends SIGTERM directly to .NET process
|
||||
- .NET process handles shutdown gracefully
|
||||
- Connections closed cleanly
|
||||
- Background jobs complete or checkpoint
|
||||
|
||||
### Docker Compose Configuration
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
melodee:
|
||||
image: melodee:1.8.0
|
||||
container_name: melodee
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5000:5000"
|
||||
environment:
|
||||
- ASPNETCORE_ENVIRONMENT=Production
|
||||
- ASPNETCORE_URLS=http://+:5000
|
||||
- ConnectionStrings__DefaultConnection=Host=postgres;Database=melodee;Username=melodee;Password=${POSTGRES_PASSWORD}
|
||||
- MusicBrainz__CachePath=/data/mb-cache.db
|
||||
- Library__Path=/music
|
||||
- Spotify__ClientId=${SPOTIFY_CLIENT_ID}
|
||||
- Spotify__ClientSecret=${SPOTIFY_CLIENT_SECRET}
|
||||
- LastFm__ApiKey=${LASTFM_API_KEY}
|
||||
- LastFm__SharedSecret=${LASTFM_SHARED_SECRET}
|
||||
- Google__ClientId=${GOOGLE_CLIENT_ID}
|
||||
- Google__ClientSecret=${GOOGLE_CLIENT_SECRET}
|
||||
- Brave__ApiKey=${BRAVE_API_KEY}
|
||||
volumes:
|
||||
- music:/music
|
||||
- data:/data
|
||||
- logs:/var/log/melodee
|
||||
- config:/app/config
|
||||
- cache:/app/cache
|
||||
- album-art:/app/album-art
|
||||
- transcoding:/app/transcoding
|
||||
depends_on:
|
||||
- postgres
|
||||
networks:
|
||||
- melodee-network
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
postgres:
|
||||
image: postgres:17
|
||||
container_name: melodee-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- POSTGRES_DB=melodee
|
||||
- POSTGRES_USER=melodee
|
||||
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- postgres-data:/var/lib/postgresql/data
|
||||
- postgres-backups:/backups
|
||||
networks:
|
||||
- melodee-network
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U melodee"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
volumes:
|
||||
music:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /path/to/music/library
|
||||
data:
|
||||
driver: local
|
||||
logs:
|
||||
driver: local
|
||||
config:
|
||||
driver: local
|
||||
cache:
|
||||
driver: local
|
||||
album-art:
|
||||
driver: local
|
||||
transcoding:
|
||||
driver: local
|
||||
postgres-data:
|
||||
driver: local
|
||||
postgres-backups:
|
||||
driver: local
|
||||
|
||||
networks:
|
||||
melodee-network:
|
||||
driver: bridge
|
||||
```
|
||||
|
||||
**Volume Breakdown**:
|
||||
|
||||
| Volume | Purpose | Size | Backup Priority |
|
||||
|--------|---------|------|-----------------|
|
||||
| `music` | User's music library | Varies (100GB-10TB) | Critical (user data) |
|
||||
| `data` | MusicBrainz cache, app data | 2-5 GB | Medium (rebuildable) |
|
||||
| `logs` | Application logs | 1-10 GB | Low (rotated) |
|
||||
| `config` | User settings, API keys | <1 MB | Critical (secrets) |
|
||||
| `cache` | Metadata cache | 100 MB-1 GB | Low (rebuildable) |
|
||||
| `album-art` | Album cover images | 1-10 GB | Medium (re-downloadable) |
|
||||
| `transcoding` | Temporary transcoded files | 1-5 GB | None (temporary) |
|
||||
| `postgres-data` | PostgreSQL database | 1-10 GB | Critical (user data) |
|
||||
| `postgres-backups` | Database backups | 5-50 GB | Critical (disaster recovery) |
|
||||
|
||||
**Environment Variables**:
|
||||
|
||||
| Variable | Purpose | Required | Default |
|
||||
|----------|---------|----------|---------|
|
||||
| `ASPNETCORE_ENVIRONMENT` | Runtime environment | No | Production |
|
||||
| `ASPNETCORE_URLS` | Listening URLs | No | http://+:5000 |
|
||||
| `ConnectionStrings__DefaultConnection` | PostgreSQL connection | Yes | - |
|
||||
| `MusicBrainz__CachePath` | SQLite cache location | No | /data/mb-cache.db |
|
||||
| `Library__Path` | Music library path | Yes | - |
|
||||
| `Spotify__ClientId` | Spotify API credentials | No | - |
|
||||
| `Spotify__ClientSecret` | Spotify API credentials | No | - |
|
||||
| `LastFm__ApiKey` | Last.fm API credentials | No | - |
|
||||
| `LastFm__SharedSecret` | Last.fm API credentials | No | - |
|
||||
| `Google__ClientId` | Google OAuth credentials | No | - |
|
||||
| `Google__ClientSecret` | Google OAuth credentials | No | - |
|
||||
| `Brave__ApiKey` | Brave Search API key | No | - |
|
||||
|
||||
**Health Checks**:
|
||||
- **Melodee**: HTTP GET to `/health` endpoint every 30 seconds
|
||||
- **PostgreSQL**: `pg_isready` command every 10 seconds
|
||||
|
||||
Health checks enable:
|
||||
- **Automatic restarts**: Container restarts if unhealthy
|
||||
- **Load balancer integration**: Remove unhealthy instances from rotation
|
||||
- **Monitoring alerts**: Trigger notifications on health check failures
|
||||
|
||||
### Environment File (.env)
|
||||
|
||||
```bash
|
||||
# PostgreSQL
|
||||
POSTGRES_PASSWORD=your-secure-password
|
||||
|
||||
# Spotify (optional)
|
||||
SPOTIFY_CLIENT_ID=your-spotify-client-id
|
||||
SPOTIFY_CLIENT_SECRET=your-spotify-client-secret
|
||||
|
||||
# Last.fm (optional)
|
||||
LASTFM_API_KEY=your-lastfm-api-key
|
||||
LASTFM_SHARED_SECRET=your-lastfm-shared-secret
|
||||
|
||||
# Google OAuth (optional)
|
||||
GOOGLE_CLIENT_ID=your-google-client-id
|
||||
GOOGLE_CLIENT_SECRET=your-google-client-secret
|
||||
|
||||
# Brave Search (optional)
|
||||
BRAVE_API_KEY=your-brave-api-key
|
||||
```
|
||||
|
||||
**Security Considerations**:
|
||||
- `.env` file should be in `.gitignore`
|
||||
- Use strong passwords (20+ characters, mixed case, numbers, symbols)
|
||||
- Rotate API keys periodically
|
||||
- Restrict file permissions: `chmod 600 .env`
|
||||
|
||||
## Deployment Scenarios
|
||||
|
||||
### Single-Server Deployment
|
||||
|
||||
**Hardware Requirements**:
|
||||
- **CPU**: 2+ cores (4+ recommended)
|
||||
- **RAM**: 4 GB minimum (8 GB recommended)
|
||||
- **Storage**: 50 GB minimum (varies with library size)
|
||||
- **Network**: 100 Mbps+ for streaming
|
||||
|
||||
**Deployment Steps**:
|
||||
|
||||
1. **Install Docker and Docker Compose**:
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker.io docker-compose
|
||||
|
||||
# Enable Docker service
|
||||
sudo systemctl enable docker
|
||||
sudo systemctl start docker
|
||||
```
|
||||
|
||||
2. **Clone repository or create docker-compose.yml**:
|
||||
```bash
|
||||
mkdir melodee
|
||||
cd melodee
|
||||
# Create docker-compose.yml and .env files
|
||||
```
|
||||
|
||||
3. **Configure environment variables**:
|
||||
```bash
|
||||
nano .env
|
||||
# Set POSTGRES_PASSWORD and optional API keys
|
||||
```
|
||||
|
||||
4. **Update music library path**:
|
||||
```bash
|
||||
# Edit docker-compose.yml
|
||||
# Change device: /path/to/music/library to actual path
|
||||
```
|
||||
|
||||
5. **Start services**:
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
6. **Verify deployment**:
|
||||
```bash
|
||||
docker-compose ps
|
||||
docker-compose logs -f melodee
|
||||
curl http://localhost:5000/health
|
||||
```
|
||||
|
||||
7. **Access web interface**:
|
||||
```
|
||||
http://localhost:5000
|
||||
```
|
||||
|
||||
### Raspberry Pi Deployment
|
||||
|
||||
**Hardware Requirements**:
|
||||
- **Model**: Raspberry Pi 4 (4GB+ RAM recommended)
|
||||
- **Storage**: 64 GB+ microSD or USB SSD
|
||||
- **OS**: Raspberry Pi OS 64-bit or Ubuntu Server ARM64
|
||||
|
||||
**ARM64 Image Build**:
|
||||
```dockerfile
|
||||
# Use ARM64 base images
|
||||
FROM mcr.microsoft.com/dotnet/sdk:10.0-arm64v8 AS build
|
||||
# ... build stage ...
|
||||
|
||||
FROM mcr.microsoft.com/dotnet/aspnet:10.0-arm64v8 AS runtime
|
||||
# ... runtime stage ...
|
||||
```
|
||||
|
||||
**Performance Optimizations**:
|
||||
1. **Use SSD instead of microSD**: 10x faster I/O
|
||||
2. **Disable transcoding**: Use direct streaming when possible
|
||||
3. **Limit concurrent jobs**: Reduce background job parallelism
|
||||
4. **Increase swap**: Add 2-4 GB swap for memory-intensive operations
|
||||
|
||||
**Deployment Steps**:
|
||||
```bash
|
||||
# Install Docker
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sudo sh get-docker.sh
|
||||
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
|
||||
# Install Docker Compose
|
||||
sudo apt-get install -y docker-compose
|
||||
|
||||
# Deploy Melodee
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
**Resource Limits**:
|
||||
```yaml
|
||||
services:
|
||||
melodee:
|
||||
# ... other config ...
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '3'
|
||||
memory: 3G
|
||||
reservations:
|
||||
cpus: '1'
|
||||
memory: 1G
|
||||
```
|
||||
|
||||
### Reverse Proxy Deployment
|
||||
|
||||
**Nginx Configuration**:
|
||||
```nginx
|
||||
upstream melodee {
|
||||
server localhost:5000;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name music.example.com;
|
||||
|
||||
# Redirect HTTP to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name music.example.com;
|
||||
|
||||
# SSL certificates
|
||||
ssl_certificate /etc/letsencrypt/live/music.example.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/music.example.com/privkey.pem;
|
||||
|
||||
# SSL configuration
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers HIGH:!aNULL:!MD5;
|
||||
ssl_prefer_server_ciphers on;
|
||||
|
||||
# Proxy settings
|
||||
location / {
|
||||
proxy_pass http://melodee;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Timeouts for streaming
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
|
||||
# Increase max upload size for album art
|
||||
client_max_body_size 50M;
|
||||
}
|
||||
```
|
||||
|
||||
**Traefik Configuration** (Docker labels):
|
||||
```yaml
|
||||
services:
|
||||
melodee:
|
||||
# ... other config ...
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.melodee.rule=Host(`music.example.com`)"
|
||||
- "traefik.http.routers.melodee.entrypoints=websecure"
|
||||
- "traefik.http.routers.melodee.tls.certresolver=letsencrypt"
|
||||
- "traefik.http.services.melodee.loadbalancer.server.port=5000"
|
||||
```
|
||||
|
||||
### High Availability Deployment
|
||||
|
||||
**Architecture**:
|
||||
```
|
||||
┌─────────────┐
|
||||
│ Load Balancer│
|
||||
│ (HAProxy) │
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌──────────────────┼──────────────────┐
|
||||
│ │ │
|
||||
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
|
||||
│Melodee 1│ │Melodee 2│ │Melodee 3│
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
└──────────────────┼──────────────────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ PostgreSQL │
|
||||
│ Primary │
|
||||
└──────┬───────┘
|
||||
│
|
||||
┌──────▼───────┐
|
||||
│ PostgreSQL │
|
||||
│ Replica │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
**Challenges**:
|
||||
1. **Blazor Server state**: SignalR connections tied to specific server
|
||||
2. **Session affinity**: Load balancer must route user to same server
|
||||
3. **Shared storage**: Music library and album art must be accessible to all instances
|
||||
|
||||
**Solutions**:
|
||||
|
||||
**1. Redis Backplane for SignalR**:
|
||||
```csharp
|
||||
services.AddSignalR()
|
||||
.AddStackExchangeRedis(options =>
|
||||
{
|
||||
options.Configuration.EndPoints.Add("redis:6379");
|
||||
});
|
||||
```
|
||||
|
||||
**2. HAProxy Sticky Sessions**:
|
||||
```
|
||||
backend melodee
|
||||
balance roundrobin
|
||||
cookie SERVERID insert indirect nocache
|
||||
server melodee1 melodee1:5000 check cookie melodee1
|
||||
server melodee2 melodee2:5000 check cookie melodee2
|
||||
server melodee3 melodee3:5000 check cookie melodee3
|
||||
```
|
||||
|
||||
**3. NFS for Shared Storage**:
|
||||
```yaml
|
||||
volumes:
|
||||
music:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: nfs
|
||||
o: addr=nfs-server,rw
|
||||
device: ":/music"
|
||||
album-art:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: nfs
|
||||
o: addr=nfs-server,rw
|
||||
device: ":/album-art"
|
||||
```
|
||||
|
||||
**4. PostgreSQL Replication**:
|
||||
```yaml
|
||||
services:
|
||||
postgres-primary:
|
||||
image: postgres:17
|
||||
environment:
|
||||
- POSTGRES_REPLICATION_MODE=master
|
||||
- POSTGRES_REPLICATION_USER=replicator
|
||||
- POSTGRES_REPLICATION_PASSWORD=replicator-password
|
||||
volumes:
|
||||
- postgres-primary-data:/var/lib/postgresql/data
|
||||
|
||||
postgres-replica:
|
||||
image: postgres:17
|
||||
environment:
|
||||
- POSTGRES_REPLICATION_MODE=slave
|
||||
- POSTGRES_MASTER_HOST=postgres-primary
|
||||
- POSTGRES_REPLICATION_USER=replicator
|
||||
- POSTGRES_REPLICATION_PASSWORD=replicator-password
|
||||
volumes:
|
||||
- postgres-replica-data:/var/lib/postgresql/data
|
||||
```
|
||||
|
||||
## Podman Deployment
|
||||
|
||||
Podman is a daemonless, rootless container runtime compatible with Docker.
|
||||
|
||||
**Advantages**:
|
||||
- **Rootless**: Runs without root privileges
|
||||
- **Daemonless**: No background daemon process
|
||||
- **Systemd integration**: Native systemd service generation
|
||||
|
||||
**Deployment Steps**:
|
||||
|
||||
1. **Install Podman**:
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install -y podman podman-compose
|
||||
|
||||
# Fedora
|
||||
sudo dnf install -y podman podman-compose
|
||||
```
|
||||
|
||||
2. **Convert Docker Compose to Podman**:
|
||||
```bash
|
||||
# Podman Compose uses same syntax
|
||||
podman-compose up -d
|
||||
```
|
||||
|
||||
3. **Generate systemd service**:
|
||||
```bash
|
||||
# Generate service file for melodee container
|
||||
podman generate systemd --new --name melodee > ~/.config/systemd/user/melodee.service
|
||||
|
||||
# Enable service
|
||||
systemctl --user enable melodee.service
|
||||
systemctl --user start melodee.service
|
||||
```
|
||||
|
||||
**Rootless Considerations**:
|
||||
- **Port binding**: Ports <1024 require root or `sysctl net.ipv4.ip_unprivileged_port_start=80`
|
||||
- **Volume permissions**: Ensure user has read/write access to volume paths
|
||||
- **Resource limits**: Rootless containers have lower default limits
|
||||
|
||||
## Backup and Recovery
|
||||
|
||||
### Database Backup
|
||||
|
||||
**Automated Daily Backups**:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/backups/postgres"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
BACKUP_FILE="$BACKUP_DIR/melodee_$TIMESTAMP.sql.gz"
|
||||
|
||||
# Create backup
|
||||
docker exec melodee-postgres pg_dump -U melodee melodee | gzip > $BACKUP_FILE
|
||||
|
||||
# Verify backup
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "Backup successful: $BACKUP_FILE"
|
||||
else
|
||||
echo "Backup failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Retain last 30 days
|
||||
find $BACKUP_DIR -name "melodee_*.sql.gz" -mtime +30 -delete
|
||||
|
||||
# Upload to S3 (optional)
|
||||
aws s3 cp $BACKUP_FILE s3://melodee-backups/postgres/
|
||||
```
|
||||
|
||||
**Cron Schedule**:
|
||||
```cron
|
||||
0 2 * * * /usr/local/bin/backup-melodee.sh
|
||||
```
|
||||
|
||||
**Restore from Backup**:
|
||||
```bash
|
||||
# Stop Melodee
|
||||
docker-compose stop melodee
|
||||
|
||||
# Restore database
|
||||
gunzip -c /backups/postgres/melodee_20250428_020000.sql.gz | \
|
||||
docker exec -i melodee-postgres psql -U melodee melodee
|
||||
|
||||
# Start Melodee
|
||||
docker-compose start melodee
|
||||
```
|
||||
|
||||
### Volume Backup
|
||||
|
||||
**Backup Script**:
|
||||
```bash
|
||||
#!/bin/bash
|
||||
BACKUP_DIR="/backups/volumes"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
||||
# Backup config volume (contains API keys)
|
||||
docker run --rm \
|
||||
-v melodee_config:/data \
|
||||
-v $BACKUP_DIR:/backup \
|
||||
alpine tar czf /backup/config_$TIMESTAMP.tar.gz -C /data .
|
||||
|
||||
# Backup data volume (MusicBrainz cache)
|
||||
docker run --rm \
|
||||
-v melodee_data:/data \
|
||||
-v $BACKUP_DIR:/backup \
|
||||
alpine tar czf /backup/data_$TIMESTAMP.tar.gz -C /data .
|
||||
|
||||
# Backup album-art volume
|
||||
docker run --rm \
|
||||
-v melodee_album-art:/data \
|
||||
-v $BACKUP_DIR:/backup \
|
||||
alpine tar czf /backup/album-art_$TIMESTAMP.tar.gz -C /data .
|
||||
```
|
||||
|
||||
**Restore Volumes**:
|
||||
```bash
|
||||
# Restore config volume
|
||||
docker run --rm \
|
||||
-v melodee_config:/data \
|
||||
-v $BACKUP_DIR:/backup \
|
||||
alpine tar xzf /backup/config_20250428_020000.tar.gz -C /data
|
||||
|
||||
# Restore data volume
|
||||
docker run --rm \
|
||||
-v melodee_data:/data \
|
||||
-v $BACKUP_DIR:/backup \
|
||||
alpine tar xzf /backup/data_20250428_020000.tar.gz -C /data
|
||||
```
|
||||
|
||||
### Disaster Recovery
|
||||
|
||||
**Full System Recovery**:
|
||||
|
||||
1. **Install Docker and Docker Compose** on new server
|
||||
2. **Restore docker-compose.yml and .env** files
|
||||
3. **Create volumes**:
|
||||
```bash
|
||||
docker volume create melodee_config
|
||||
docker volume create melodee_data
|
||||
docker volume create melodee_album-art
|
||||
docker volume create melodee_postgres-data
|
||||
```
|
||||
|
||||
4. **Restore volume data** from backups
|
||||
5. **Restore PostgreSQL database** from backup
|
||||
6. **Start services**:
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
7. **Verify health**:
|
||||
```bash
|
||||
docker-compose ps
|
||||
curl http://localhost:5000/health
|
||||
```
|
||||
|
||||
**Recovery Time Objective (RTO)**: 1-2 hours
|
||||
**Recovery Point Objective (RPO)**: 24 hours (daily backups)
|
||||
|
||||
## Monitoring and Logging
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
**Metrics Endpoint**:
|
||||
```csharp
|
||||
app.UseEndpoints(endpoints =>
|
||||
{
|
||||
endpoints.MapMetrics("/metrics");
|
||||
});
|
||||
```
|
||||
|
||||
**Prometheus Configuration**:
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: 'melodee'
|
||||
static_configs:
|
||||
- targets: ['melodee:5000']
|
||||
metrics_path: '/metrics'
|
||||
scrape_interval: 15s
|
||||
```
|
||||
|
||||
**Key Metrics**:
|
||||
- `http_requests_total`: Total HTTP requests
|
||||
- `http_request_duration_seconds`: Request latency
|
||||
- `dotnet_gc_collections_total`: Garbage collection count
|
||||
- `process_cpu_seconds_total`: CPU usage
|
||||
- `process_resident_memory_bytes`: Memory usage
|
||||
- `melodee_scrobbles_total`: Total scrobbles submitted
|
||||
- `melodee_library_tracks_total`: Total tracks in library
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
**Dashboard Panels**:
|
||||
1. **Request Rate**: Requests per second
|
||||
2. **Response Time**: P50, P95, P99 latencies
|
||||
3. **Error Rate**: 4xx and 5xx responses
|
||||
4. **CPU Usage**: Process CPU percentage
|
||||
5. **Memory Usage**: Resident memory
|
||||
6. **Database Connections**: Active connections
|
||||
7. **Scrobble Rate**: Scrobbles per hour
|
||||
8. **Library Size**: Total tracks, albums, artists
|
||||
|
||||
### Log Aggregation
|
||||
|
||||
**Serilog to Elasticsearch**:
|
||||
```csharp
|
||||
Log.Logger = new LoggerConfiguration()
|
||||
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://elasticsearch:9200"))
|
||||
{
|
||||
AutoRegisterTemplate = true,
|
||||
IndexFormat = "melodee-logs-{0:yyyy.MM.dd}"
|
||||
})
|
||||
.CreateLogger();
|
||||
```
|
||||
|
||||
**Kibana Queries**:
|
||||
```
|
||||
# Errors in last hour
|
||||
level:Error AND @timestamp:[now-1h TO now]
|
||||
|
||||
# Slow requests (>1s)
|
||||
http.request.duration:>1000
|
||||
|
||||
# Failed scrobbles
|
||||
message:"scrobble failed"
|
||||
```
|
||||
|
||||
## Security Hardening
|
||||
|
||||
### HTTPS Configuration
|
||||
|
||||
**Let's Encrypt with Certbot**:
|
||||
```bash
|
||||
# Install Certbot
|
||||
sudo apt-get install -y certbot
|
||||
|
||||
# Obtain certificate
|
||||
sudo certbot certonly --standalone -d music.example.com
|
||||
|
||||
# Configure Nginx with certificate (see Reverse Proxy section)
|
||||
```
|
||||
|
||||
**Certificate Renewal**:
|
||||
```cron
|
||||
0 0 1 * * certbot renew --quiet && systemctl reload nginx
|
||||
```
|
||||
|
||||
### Firewall Configuration
|
||||
|
||||
**UFW (Ubuntu)**:
|
||||
```bash
|
||||
# Allow SSH
|
||||
sudo ufw allow 22/tcp
|
||||
|
||||
# Allow HTTP/HTTPS (if using reverse proxy)
|
||||
sudo ufw allow 80/tcp
|
||||
sudo ufw allow 443/tcp
|
||||
|
||||
# Allow Melodee (if direct access)
|
||||
sudo ufw allow 5000/tcp
|
||||
|
||||
# Enable firewall
|
||||
sudo ufw enable
|
||||
```
|
||||
|
||||
### Secret Management
|
||||
|
||||
**Docker Secrets** (Swarm mode):
|
||||
```yaml
|
||||
services:
|
||||
melodee:
|
||||
secrets:
|
||||
- postgres_password
|
||||
- spotify_client_secret
|
||||
environment:
|
||||
- ConnectionStrings__DefaultConnection=Host=postgres;Database=melodee;Username=melodee;Password_FILE=/run/secrets/postgres_password
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password.txt
|
||||
spotify_client_secret:
|
||||
file: ./secrets/spotify_client_secret.txt
|
||||
```
|
||||
|
||||
**Vault Integration**:
|
||||
```csharp
|
||||
var vaultClient = new VaultClient(new VaultClientSettings("http://vault:8200", "vault-token"));
|
||||
var secret = await vaultClient.V1.Secrets.KeyValue.V2.ReadSecretAsync("melodee/postgres");
|
||||
var password = secret.Data.Data["password"].ToString();
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### PostgreSQL Optimization
|
||||
|
||||
```sql
|
||||
-- Increase shared buffers (25% of RAM)
|
||||
ALTER SYSTEM SET shared_buffers = '2GB';
|
||||
|
||||
-- Increase work memory for complex queries
|
||||
ALTER SYSTEM SET work_mem = '64MB';
|
||||
|
||||
-- Increase maintenance work memory for VACUUM
|
||||
ALTER SYSTEM SET maintenance_work_mem = '512MB';
|
||||
|
||||
-- Optimize for SSD
|
||||
ALTER SYSTEM SET random_page_cost = 1.1;
|
||||
|
||||
-- Enable query planning statistics
|
||||
ALTER SYSTEM SET track_activity_query_size = 2048;
|
||||
|
||||
-- Reload configuration
|
||||
SELECT pg_reload_conf();
|
||||
```
|
||||
|
||||
### .NET Runtime Optimization
|
||||
|
||||
**Environment Variables**:
|
||||
```yaml
|
||||
environment:
|
||||
- DOTNET_GCServer=1 # Server GC mode
|
||||
- DOTNET_GCConcurrent=1 # Concurrent GC
|
||||
- DOTNET_GCRetainVM=1 # Retain virtual memory
|
||||
- DOTNET_ThreadPool_MinThreads=50 # Minimum thread pool size
|
||||
- DOTNET_ThreadPool_MaxThreads=500 # Maximum thread pool size
|
||||
```
|
||||
|
||||
### Caching Configuration
|
||||
|
||||
**Redis Cache**:
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
image: redis:7
|
||||
command: redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
```
|
||||
|
||||
**Application Configuration**:
|
||||
```csharp
|
||||
services.AddStackExchangeRedisCache(options =>
|
||||
{
|
||||
options.Configuration = "redis:6379";
|
||||
options.InstanceName = "melodee:";
|
||||
});
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Melodee's deployment architecture demonstrates production-ready containerization with Docker multi-stage builds, automatic migrations, and comprehensive volume management. The 12 persistent volumes ensure data persistence, while health checks and logging enable robust monitoring.
|
||||
|
||||
Key strengths:
|
||||
- **Easy deployment**: Single-command Docker Compose setup
|
||||
- **Automatic migrations**: Database schema updates on startup
|
||||
- **Raspberry Pi support**: ARM64 compatibility for low-power deployments
|
||||
- **Podman compatibility**: Rootless container runtime support
|
||||
|
||||
Key challenges:
|
||||
- **Horizontal scaling**: Blazor Server requires sticky sessions and Redis backplane
|
||||
- **Backup complexity**: 12 volumes require coordinated backup strategy
|
||||
- **Secret management**: API keys in environment variables (consider Vault)
|
||||
|
||||
The architecture positions Melodee for both simple self-hosted deployments and advanced production configurations with high availability and monitoring.
|
||||
Reference in New Issue
Block a user