fujin/metadata-agregator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial implementation of metadata aggregator

Browse Source 
- gRPC service with MusicBrainz provider
- PostgreSQL schema with migrations
- Service layer with database-first caching
- Repository pattern for data access
- YAML configuration support
- Research documentation for 17 music metadata projects
This commit is contained in:
Alexander
2026-04-28 16:27:14 +02:00
commit a1f6701bac
163 changed files with 95884 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/melodee/analysis/API.md
+1262
View File
File diff suppressed because it is too large Load Diff
docs/research/melodee/analysis/ARCHITECTURE.md
+1294
View File
File diff suppressed because it is too large Load Diff
docs/research/melodee/analysis/CODEBASE.md
+1271
View File
File diff suppressed because it is too large Load Diff
docs/research/melodee/analysis/DATA.md
+1271
View File
File diff suppressed because it is too large Load Diff
docs/research/melodee/analysis/DEPLOYMENT.md
+922
View File
@@ -0,0 +1,922 @@
# Melodee: Deployment Analysis
## Deployment Strategy Overview
Melodee provides Docker-based deployment with multi-stage builds, Docker Compose orchestration, and automatic database migrations. The deployment architecture prioritizes ease of setup for self-hosted environments while supporting advanced configurations for production deployments.
Key deployment features:
- **Docker multi-stage build**: Optimized image size and security
- **Docker Compose**: Single-command deployment with PostgreSQL
- **Automatic migrations**: Database schema updates on container startup
- **12 persistent volumes**: Data persistence across container restarts
- **Raspberry Pi support**: ARM64 compatibility for low-power hardware
- **Podman compatibility**: Rootless container runtime support
## Docker Architecture
### Multi-Stage Dockerfile
```dockerfile
# Build stage
FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
# Copy project files
COPY ["Melodee.Web/Melodee.Web.csproj", "Melodee.Web/"]
COPY ["Melodee.Data/Melodee.Data.csproj", "Melodee.Data/"]
COPY ["Melodee.Core/Melodee.Core.csproj", "Melodee.Core/"]
# Restore dependencies
RUN dotnet restore "Melodee.Web/Melodee.Web.csproj"
# Copy source code
COPY . .
# Build application
WORKDIR "/src/Melodee.Web"
RUN dotnet build "Melodee.Web.csproj" -c Release -o /app/build
# Publish application
RUN dotnet publish "Melodee.Web.csproj" -c Release -o /app/publish /p:UseAppHost=false
# Runtime stage
FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
WORKDIR /app
# Install FFmpeg for transcoding
RUN apt-get update && \
apt-get install -y --no-install-recommends ffmpeg && \
rm -rf /var/lib/apt/lists/*
# Copy published application
COPY --from=build /app/publish .
# Copy entrypoint script
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh
# Expose port
EXPOSE 5000
# Set entrypoint
ENTRYPOINT ["./entrypoint.sh"]
```
**Multi-Stage Benefits**:
1. **Smaller image size**: Runtime image excludes SDK (saves ~500 MB)
2. **Faster deployments**: Smaller images transfer and start faster
3. **Security**: No build tools in production image
4. **Layer caching**: Dependencies cached separately from source code
**Image Size Comparison**:
- Single-stage (with SDK): ~1.2 GB
- Multi-stage (runtime only): ~700 MB
- Savings: ~500 MB (42% reduction)
### Entrypoint Script
```bash
#!/bin/bash
set -e
echo "Melodee v1.8.0 starting..."
# Wait for PostgreSQL to be ready
echo "Waiting for PostgreSQL..."
until PGPASSWORD=$POSTGRES_PASSWORD psql -h "$POSTGRES_HOST" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c '\q' 2>/dev/null; do
echo "PostgreSQL is unavailable - sleeping"
sleep 2
done
echo "PostgreSQL is ready"
# Run database migrations
echo "Applying database migrations..."
dotnet ef database update --project /app/Melodee.Data.dll --no-build
if [ $? -ne 0 ]; then
echo "Migration failed, exiting..."
exit 1
fi
echo "Migrations applied successfully"
# Start application
echo "Starting Melodee..."
exec dotnet Melodee.Web.dll
```
**Entrypoint Responsibilities**:
1. **Database readiness check**: Waits for PostgreSQL before starting
2. **Automatic migrations**: Applies schema changes on startup
3. **Error handling**: Exits if migrations fail
4. **Process replacement**: `exec` replaces shell with .NET process for proper signal handling
**Signal Handling**:
The `exec` command is critical for graceful shutdown. Without it:
- Docker sends SIGTERM to shell process
- Shell doesn't forward signal to .NET process
- .NET process killed with SIGKILL after timeout
- No graceful shutdown (connections dropped, jobs interrupted)
With `exec`:
- Docker sends SIGTERM directly to .NET process
- .NET process handles shutdown gracefully
- Connections closed cleanly
- Background jobs complete or checkpoint
### Docker Compose Configuration
```yaml
version: '3.8'
services:
melodee:
image: melodee:1.8.0
container_name: melodee
restart: unless-stopped
ports:
- "5000:5000"
environment:
- ASPNETCORE_ENVIRONMENT=Production
- ASPNETCORE_URLS=http://+:5000
- ConnectionStrings__DefaultConnection=Host=postgres;Database=melodee;Username=melodee;Password=${POSTGRES_PASSWORD}
- MusicBrainz__CachePath=/data/mb-cache.db
- Library__Path=/music
- Spotify__ClientId=${SPOTIFY_CLIENT_ID}
- Spotify__ClientSecret=${SPOTIFY_CLIENT_SECRET}
- LastFm__ApiKey=${LASTFM_API_KEY}
- LastFm__SharedSecret=${LASTFM_SHARED_SECRET}
- Google__ClientId=${GOOGLE_CLIENT_ID}
- Google__ClientSecret=${GOOGLE_CLIENT_SECRET}
- Brave__ApiKey=${BRAVE_API_KEY}
volumes:
- music:/music
- data:/data
- logs:/var/log/melodee
- config:/app/config
- cache:/app/cache
- album-art:/app/album-art
- transcoding:/app/transcoding
depends_on:
- postgres
networks:
- melodee-network
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
postgres:
image: postgres:17
container_name: melodee-postgres
restart: unless-stopped
environment:
- POSTGRES_DB=melodee
- POSTGRES_USER=melodee
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
volumes:
- postgres-data:/var/lib/postgresql/data
- postgres-backups:/backups
networks:
- melodee-network
healthcheck:
test: ["CMD-SHELL", "pg_isready -U melodee"]
interval: 10s
timeout: 5s
retries: 5
volumes:
music:
driver: local
driver_opts:
type: none
o: bind
device: /path/to/music/library
data:
driver: local
logs:
driver: local
config:
driver: local
cache:
driver: local
album-art:
driver: local
transcoding:
driver: local
postgres-data:
driver: local
postgres-backups:
driver: local
networks:
melodee-network:
driver: bridge
```
**Volume Breakdown**:
| Volume | Purpose | Size | Backup Priority |
|--------|---------|------|-----------------|
| `music` | User's music library | Varies (100GB-10TB) | Critical (user data) |
| `data` | MusicBrainz cache, app data | 2-5 GB | Medium (rebuildable) |
| `logs` | Application logs | 1-10 GB | Low (rotated) |
| `config` | User settings, API keys | <1 MB | Critical (secrets) |
| `cache` | Metadata cache | 100 MB-1 GB | Low (rebuildable) |
| `album-art` | Album cover images | 1-10 GB | Medium (re-downloadable) |
| `transcoding` | Temporary transcoded files | 1-5 GB | None (temporary) |
| `postgres-data` | PostgreSQL database | 1-10 GB | Critical (user data) |
| `postgres-backups` | Database backups | 5-50 GB | Critical (disaster recovery) |
**Environment Variables**:
| Variable | Purpose | Required | Default |
|----------|---------|----------|---------|
| `ASPNETCORE_ENVIRONMENT` | Runtime environment | No | Production |
| `ASPNETCORE_URLS` | Listening URLs | No | http://+:5000 |
| `ConnectionStrings__DefaultConnection` | PostgreSQL connection | Yes | - |
| `MusicBrainz__CachePath` | SQLite cache location | No | /data/mb-cache.db |
| `Library__Path` | Music library path | Yes | - |
| `Spotify__ClientId` | Spotify API credentials | No | - |
| `Spotify__ClientSecret` | Spotify API credentials | No | - |
| `LastFm__ApiKey` | Last.fm API credentials | No | - |
| `LastFm__SharedSecret` | Last.fm API credentials | No | - |
| `Google__ClientId` | Google OAuth credentials | No | - |
| `Google__ClientSecret` | Google OAuth credentials | No | - |
| `Brave__ApiKey` | Brave Search API key | No | - |
**Health Checks**:
- **Melodee**: HTTP GET to `/health` endpoint every 30 seconds
- **PostgreSQL**: `pg_isready` command every 10 seconds
Health checks enable:
- **Automatic restarts**: Container restarts if unhealthy
- **Load balancer integration**: Remove unhealthy instances from rotation
- **Monitoring alerts**: Trigger notifications on health check failures
### Environment File (.env)
```bash
# PostgreSQL
POSTGRES_PASSWORD=your-secure-password
# Spotify (optional)
SPOTIFY_CLIENT_ID=your-spotify-client-id
SPOTIFY_CLIENT_SECRET=your-spotify-client-secret
# Last.fm (optional)
LASTFM_API_KEY=your-lastfm-api-key
LASTFM_SHARED_SECRET=your-lastfm-shared-secret
# Google OAuth (optional)
GOOGLE_CLIENT_ID=your-google-client-id
GOOGLE_CLIENT_SECRET=your-google-client-secret
# Brave Search (optional)
BRAVE_API_KEY=your-brave-api-key
```
**Security Considerations**:
- `.env` file should be in `.gitignore`
- Use strong passwords (20+ characters, mixed case, numbers, symbols)
- Rotate API keys periodically
- Restrict file permissions: `chmod 600 .env`
## Deployment Scenarios
### Single-Server Deployment
**Hardware Requirements**:
- **CPU**: 2+ cores (4+ recommended)
- **RAM**: 4 GB minimum (8 GB recommended)
- **Storage**: 50 GB minimum (varies with library size)
- **Network**: 100 Mbps+ for streaming
**Deployment Steps**:
1. **Install Docker and Docker Compose**:
```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y docker.io docker-compose
# Enable Docker service
sudo systemctl enable docker
sudo systemctl start docker
```
2. **Clone repository or create docker-compose.yml**:
```bash
mkdir melodee
cd melodee
# Create docker-compose.yml and .env files
```
3. **Configure environment variables**:
```bash
nano .env
# Set POSTGRES_PASSWORD and optional API keys
```
4. **Update music library path**:
```bash
# Edit docker-compose.yml
# Change device: /path/to/music/library to actual path
```
5. **Start services**:
```bash
docker-compose up -d
```
6. **Verify deployment**:
```bash
docker-compose ps
docker-compose logs -f melodee
curl http://localhost:5000/health
```
7. **Access web interface**:
```
http://localhost:5000
```
### Raspberry Pi Deployment
**Hardware Requirements**:
- **Model**: Raspberry Pi 4 (4GB+ RAM recommended)
- **Storage**: 64 GB+ microSD or USB SSD
- **OS**: Raspberry Pi OS 64-bit or Ubuntu Server ARM64
**ARM64 Image Build**:
```dockerfile
# Use ARM64 base images
FROM mcr.microsoft.com/dotnet/sdk:10.0-arm64v8 AS build
# ... build stage ...
FROM mcr.microsoft.com/dotnet/aspnet:10.0-arm64v8 AS runtime
# ... runtime stage ...
```
**Performance Optimizations**:
1. **Use SSD instead of microSD**: 10x faster I/O
2. **Disable transcoding**: Use direct streaming when possible
3. **Limit concurrent jobs**: Reduce background job parallelism
4. **Increase swap**: Add 2-4 GB swap for memory-intensive operations
**Deployment Steps**:
```bash
# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
# Add user to docker group
sudo usermod -aG docker $USER
# Install Docker Compose
sudo apt-get install -y docker-compose
# Deploy Melodee
docker-compose up -d
```
**Resource Limits**:
```yaml
services:
melodee:
# ... other config ...
deploy:
resources:
limits:
cpus: '3'
memory: 3G
reservations:
cpus: '1'
memory: 1G
```
### Reverse Proxy Deployment
**Nginx Configuration**:
```nginx
upstream melodee {
server localhost:5000;
}
server {
listen 80;
server_name music.example.com;
# Redirect HTTP to HTTPS
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name music.example.com;
# SSL certificates
ssl_certificate /etc/letsencrypt/live/music.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/music.example.com/privkey.pem;
# SSL configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
# Proxy settings
location / {
proxy_pass http://melodee;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts for streaming
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# Increase max upload size for album art
client_max_body_size 50M;
}
```
**Traefik Configuration** (Docker labels):
```yaml
services:
melodee:
# ... other config ...
labels:
- "traefik.enable=true"
- "traefik.http.routers.melodee.rule=Host(`music.example.com`)"
- "traefik.http.routers.melodee.entrypoints=websecure"
- "traefik.http.routers.melodee.tls.certresolver=letsencrypt"
- "traefik.http.services.melodee.loadbalancer.server.port=5000"
```
### High Availability Deployment
**Architecture**:
```
┌─────────────┐
│ Load Balancer│
│ (HAProxy) │
└──────┬───────┘
│
┌──────────────────┼──────────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│Melodee 1│ │Melodee 2│ │Melodee 3│
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
└──────────────────┼──────────────────┘
│
┌──────▼───────┐
│ PostgreSQL │
│ Primary │
└──────┬───────┘
│
┌──────▼───────┐
│ PostgreSQL │
│ Replica │
└──────────────┘
```
**Challenges**:
1. **Blazor Server state**: SignalR connections tied to specific server
2. **Session affinity**: Load balancer must route user to same server
3. **Shared storage**: Music library and album art must be accessible to all instances
**Solutions**:
**1. Redis Backplane for SignalR**:
```csharp
services.AddSignalR()
.AddStackExchangeRedis(options =>
{
options.Configuration.EndPoints.Add("redis:6379");
});
```
**2. HAProxy Sticky Sessions**:
```
backend melodee
balance roundrobin
cookie SERVERID insert indirect nocache
server melodee1 melodee1:5000 check cookie melodee1
server melodee2 melodee2:5000 check cookie melodee2
server melodee3 melodee3:5000 check cookie melodee3
```
**3. NFS for Shared Storage**:
```yaml
volumes:
music:
driver: local
driver_opts:
type: nfs
o: addr=nfs-server,rw
device: ":/music"
album-art:
driver: local
driver_opts:
type: nfs
o: addr=nfs-server,rw
device: ":/album-art"
```
**4. PostgreSQL Replication**:
```yaml
services:
postgres-primary:
image: postgres:17
environment:
- POSTGRES_REPLICATION_MODE=master
- POSTGRES_REPLICATION_USER=replicator
- POSTGRES_REPLICATION_PASSWORD=replicator-password
volumes:
- postgres-primary-data:/var/lib/postgresql/data
postgres-replica:
image: postgres:17
environment:
- POSTGRES_REPLICATION_MODE=slave
- POSTGRES_MASTER_HOST=postgres-primary
- POSTGRES_REPLICATION_USER=replicator
- POSTGRES_REPLICATION_PASSWORD=replicator-password
volumes:
- postgres-replica-data:/var/lib/postgresql/data
```
## Podman Deployment
Podman is a daemonless, rootless container runtime compatible with Docker.
**Advantages**:
- **Rootless**: Runs without root privileges
- **Daemonless**: No background daemon process
- **Systemd integration**: Native systemd service generation
**Deployment Steps**:
1. **Install Podman**:
```bash
# Ubuntu/Debian
sudo apt-get install -y podman podman-compose
# Fedora
sudo dnf install -y podman podman-compose
```
2. **Convert Docker Compose to Podman**:
```bash
# Podman Compose uses same syntax
podman-compose up -d
```
3. **Generate systemd service**:
```bash
# Generate service file for melodee container
podman generate systemd --new --name melodee > ~/.config/systemd/user/melodee.service
# Enable service
systemctl --user enable melodee.service
systemctl --user start melodee.service
```
**Rootless Considerations**:
- **Port binding**: Ports <1024 require root or `sysctl net.ipv4.ip_unprivileged_port_start=80`
- **Volume permissions**: Ensure user has read/write access to volume paths
- **Resource limits**: Rootless containers have lower default limits
## Backup and Recovery
### Database Backup
**Automated Daily Backups**:
```bash
#!/bin/bash
BACKUP_DIR="/backups/postgres"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/melodee_$TIMESTAMP.sql.gz"
# Create backup
docker exec melodee-postgres pg_dump -U melodee melodee | gzip > $BACKUP_FILE
# Verify backup
if [ $? -eq 0 ]; then
echo "Backup successful: $BACKUP_FILE"
else
echo "Backup failed"
exit 1
fi
# Retain last 30 days
find $BACKUP_DIR -name "melodee_*.sql.gz" -mtime +30 -delete
# Upload to S3 (optional)
aws s3 cp $BACKUP_FILE s3://melodee-backups/postgres/
```
**Cron Schedule**:
```cron
0 2 * * * /usr/local/bin/backup-melodee.sh
```
**Restore from Backup**:
```bash
# Stop Melodee
docker-compose stop melodee
# Restore database
gunzip -c /backups/postgres/melodee_20250428_020000.sql.gz | \
docker exec -i melodee-postgres psql -U melodee melodee
# Start Melodee
docker-compose start melodee
```
### Volume Backup
**Backup Script**:
```bash
#!/bin/bash
BACKUP_DIR="/backups/volumes"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Backup config volume (contains API keys)
docker run --rm \
-v melodee_config:/data \
-v $BACKUP_DIR:/backup \
alpine tar czf /backup/config_$TIMESTAMP.tar.gz -C /data .
# Backup data volume (MusicBrainz cache)
docker run --rm \
-v melodee_data:/data \
-v $BACKUP_DIR:/backup \
alpine tar czf /backup/data_$TIMESTAMP.tar.gz -C /data .
# Backup album-art volume
docker run --rm \
-v melodee_album-art:/data \
-v $BACKUP_DIR:/backup \
alpine tar czf /backup/album-art_$TIMESTAMP.tar.gz -C /data .
```
**Restore Volumes**:
```bash
# Restore config volume
docker run --rm \
-v melodee_config:/data \
-v $BACKUP_DIR:/backup \
alpine tar xzf /backup/config_20250428_020000.tar.gz -C /data
# Restore data volume
docker run --rm \
-v melodee_data:/data \
-v $BACKUP_DIR:/backup \
alpine tar xzf /backup/data_20250428_020000.tar.gz -C /data
```
### Disaster Recovery
**Full System Recovery**:
1. **Install Docker and Docker Compose** on new server
2. **Restore docker-compose.yml and .env** files
3. **Create volumes**:
```bash
docker volume create melodee_config
docker volume create melodee_data
docker volume create melodee_album-art
docker volume create melodee_postgres-data
```
4. **Restore volume data** from backups
5. **Restore PostgreSQL database** from backup
6. **Start services**:
```bash
docker-compose up -d
```
7. **Verify health**:
```bash
docker-compose ps
curl http://localhost:5000/health
```
**Recovery Time Objective (RTO)**: 1-2 hours
**Recovery Point Objective (RPO)**: 24 hours (daily backups)
## Monitoring and Logging
### Prometheus Metrics
**Metrics Endpoint**:
```csharp
app.UseEndpoints(endpoints =>
{
endpoints.MapMetrics("/metrics");
});
```
**Prometheus Configuration**:
```yaml
scrape_configs:
- job_name: 'melodee'
static_configs:
- targets: ['melodee:5000']
metrics_path: '/metrics'
scrape_interval: 15s
```
**Key Metrics**:
- `http_requests_total`: Total HTTP requests
- `http_request_duration_seconds`: Request latency
- `dotnet_gc_collections_total`: Garbage collection count
- `process_cpu_seconds_total`: CPU usage
- `process_resident_memory_bytes`: Memory usage
- `melodee_scrobbles_total`: Total scrobbles submitted
- `melodee_library_tracks_total`: Total tracks in library
### Grafana Dashboard
**Dashboard Panels**:
1. **Request Rate**: Requests per second
2. **Response Time**: P50, P95, P99 latencies
3. **Error Rate**: 4xx and 5xx responses
4. **CPU Usage**: Process CPU percentage
5. **Memory Usage**: Resident memory
6. **Database Connections**: Active connections
7. **Scrobble Rate**: Scrobbles per hour
8. **Library Size**: Total tracks, albums, artists
### Log Aggregation
**Serilog to Elasticsearch**:
```csharp
Log.Logger = new LoggerConfiguration()
.WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://elasticsearch:9200"))
{
AutoRegisterTemplate = true,
IndexFormat = "melodee-logs-{0:yyyy.MM.dd}"
})
.CreateLogger();
```
**Kibana Queries**:
```
# Errors in last hour
level:Error AND @timestamp:[now-1h TO now]
# Slow requests (>1s)
http.request.duration:>1000
# Failed scrobbles
message:"scrobble failed"
```
## Security Hardening
### HTTPS Configuration
**Let's Encrypt with Certbot**:
```bash
# Install Certbot
sudo apt-get install -y certbot
# Obtain certificate
sudo certbot certonly --standalone -d music.example.com
# Configure Nginx with certificate (see Reverse Proxy section)
```
**Certificate Renewal**:
```cron
0 0 1 * * certbot renew --quiet && systemctl reload nginx
```
### Firewall Configuration
**UFW (Ubuntu)**:
```bash
# Allow SSH
sudo ufw allow 22/tcp
# Allow HTTP/HTTPS (if using reverse proxy)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# Allow Melodee (if direct access)
sudo ufw allow 5000/tcp
# Enable firewall
sudo ufw enable
```
### Secret Management
**Docker Secrets** (Swarm mode):
```yaml
services:
melodee:
secrets:
- postgres_password
- spotify_client_secret
environment:
- ConnectionStrings__DefaultConnection=Host=postgres;Database=melodee;Username=melodee;Password_FILE=/run/secrets/postgres_password
secrets:
postgres_password:
file: ./secrets/postgres_password.txt
spotify_client_secret:
file: ./secrets/spotify_client_secret.txt
```
**Vault Integration**:
```csharp
var vaultClient = new VaultClient(new VaultClientSettings("http://vault:8200", "vault-token"));
var secret = await vaultClient.V1.Secrets.KeyValue.V2.ReadSecretAsync("melodee/postgres");
var password = secret.Data.Data["password"].ToString();
```
## Performance Tuning
### PostgreSQL Optimization
```sql
-- Increase shared buffers (25% of RAM)
ALTER SYSTEM SET shared_buffers = '2GB';
-- Increase work memory for complex queries
ALTER SYSTEM SET work_mem = '64MB';
-- Increase maintenance work memory for VACUUM
ALTER SYSTEM SET maintenance_work_mem = '512MB';
-- Optimize for SSD
ALTER SYSTEM SET random_page_cost = 1.1;
-- Enable query planning statistics
ALTER SYSTEM SET track_activity_query_size = 2048;
-- Reload configuration
SELECT pg_reload_conf();
```
### .NET Runtime Optimization
**Environment Variables**:
```yaml
environment:
- DOTNET_GCServer=1 # Server GC mode
- DOTNET_GCConcurrent=1 # Concurrent GC
- DOTNET_GCRetainVM=1 # Retain virtual memory
- DOTNET_ThreadPool_MinThreads=50 # Minimum thread pool size
- DOTNET_ThreadPool_MaxThreads=500 # Maximum thread pool size
```
### Caching Configuration
**Redis Cache**:
```yaml
services:
redis:
image: redis:7
command: redis-server --maxmemory 1gb --maxmemory-policy allkeys-lru
volumes:
- redis-data:/data
```
**Application Configuration**:
```csharp
services.AddStackExchangeRedisCache(options =>
{
options.Configuration = "redis:6379";
options.InstanceName = "melodee:";
});
```
## Conclusion
Melodee's deployment architecture demonstrates production-ready containerization with Docker multi-stage builds, automatic migrations, and comprehensive volume management. The 12 persistent volumes ensure data persistence, while health checks and logging enable robust monitoring.
Key strengths:
- **Easy deployment**: Single-command Docker Compose setup
- **Automatic migrations**: Database schema updates on startup
- **Raspberry Pi support**: ARM64 compatibility for low-power deployments
- **Podman compatibility**: Rootless container runtime support
Key challenges:
- **Horizontal scaling**: Blazor Server requires sticky sessions and Redis backplane
- **Backup complexity**: 12 volumes require coordinated backup strategy
- **Secret management**: API keys in environment variables (consider Vault)
The architecture positions Melodee for both simple self-hosted deployments and advanced production configurations with high availability and monitoring.
docs/research/melodee/analysis/INTEGRATIONS.md
+1143
View File
File diff suppressed because it is too large Load Diff
docs/research/melodee/analysis/OVERVIEW.md
+377
View File
@@ -0,0 +1,377 @@
# Melodee: Project Overview
## Executive Summary
Melodee is a self-hosted music server and metadata aggregator built on .NET 10 and Blazor Server. The project positions itself as a modern alternative to traditional music servers, emphasizing metadata quality, multi-protocol API support, and extensibility. With 62 GitHub stars and active development, Melodee represents a niche but technically sophisticated approach to personal music library management.
The system's core value proposition centers on intelligent metadata aggregation from six different providers, a multi-stage processing pipeline that transforms raw audio files into organized library entries, and compatibility with existing music client ecosystems through three distinct API protocols.
## Project Identity
**Repository**: https://github.com/melodee-project/melodee
**Version**: 1.8.0
**License**: MIT
**Primary Language**: C# (.NET 10)
**UI Framework**: Blazor Server with Radzen components
**Database**: PostgreSQL 17 (primary), SQLite (MusicBrainz cache)
**Stars**: 62
**Status**: Active development
The MIT license makes Melodee suitable for both personal and commercial use without significant legal constraints. The choice of .NET 10 indicates commitment to modern framework features and performance characteristics, though it also creates a dependency on Microsoft's release cycle.
## Core Capabilities
### Multi-Protocol API Support
Melodee implements three distinct API protocols, each serving different client ecosystems:
1. **Native REST API** (`/api/v1/`): JWT-based authentication, modern RESTful design, full feature access
2. **OpenSubsonic** (`/rest/`): Token and salt authentication, compatibility with Subsonic clients (DSub, Ultrasonic, Sublime Music)
3. **Jellyfin API** (`/api/jf/`): Custom token authentication, compatibility with Jellyfin clients
This multi-protocol approach maximizes client compatibility without forcing users into a single ecosystem. The rate limiting differs per protocol: Native API allows 30 requests per 30 seconds, authentication endpoints limit to 10 per 60 seconds, and Jellyfin endpoints permit 200 per 60 seconds.
### Metadata Aggregation Pipeline
The system processes music files through four distinct stages:
1. **Inbound**: Raw file ingestion and validation
2. **Staging**: Metadata extraction and provider queries
3. **Storage**: File organization and normalization
4. **Database**: Entity persistence and indexing
Six metadata providers contribute to the aggregation process:
- **MusicBrainz**: Primary source with local SQLite cache, monthly updates
- **Last.fm**: Social metadata, play counts, similar artists
- **Spotify**: Album art, popularity metrics (client credentials flow)
- **iTunes**: Commercial metadata, preview URLs
- **Deezer**: European market metadata
- **Brave Search**: Fallback web search for obscure releases
The MusicBrainz cache strategy deserves attention. Rather than querying the remote API for every lookup, Melodee maintains a local SQLite database updated monthly. This reduces latency and respects MusicBrainz rate limits while ensuring metadata freshness.
### Background Job Architecture
Melodee uses Quartz.NET to orchestrate 17 background jobs with dependency chaining. Jobs handle:
- Metadata provider synchronization
- Library scanning and updates
- Scrobble submission (Last.fm and internal)
- Database maintenance and optimization
- Cache invalidation
- Statistics calculation
- Podcast feed updates
Job chaining allows complex workflows. For example, a library scan job triggers metadata enrichment jobs, which then trigger cache invalidation, which finally triggers statistics recalculation. This declarative approach keeps the system responsive while handling computationally expensive operations asynchronously.
## Technical Foundation
### .NET 10 and Blazor Server
The choice of Blazor Server over Blazor WebAssembly or traditional SPA frameworks has specific implications:
**Advantages**:
- Full .NET runtime access without WASM limitations
- Smaller initial payload (no framework download)
- Direct database access without API layer overhead
- Real-time updates via SignalR (used for Party Mode)
**Tradeoffs**:
- Server-side rendering requires persistent connection
- Higher server resource usage per user
- Network latency affects UI responsiveness
- Scaling requires sticky sessions or Redis backplane
For a self-hosted music server with typically 1-10 concurrent users, these tradeoffs favor Blazor Server. The SignalR connection enables Party Mode, where multiple users see synchronized playback state.
### Database Architecture
PostgreSQL 17 serves as the primary data store with over 100 migrations and 40+ entities. The migration count suggests iterative development and schema evolution. Entity Framework Core 10 provides the ORM layer.
SQLite handles the MusicBrainz cache separately. This dual-database approach isolates read-heavy cache queries from transactional music library operations. The cache can be rebuilt without affecting user data.
Key entity categories:
- **Library entities**: Albums, Artists, Tracks, Genres
- **User entities**: Users, Playlists, Favorites, Scrobbles
- **Metadata entities**: Provider mappings, external IDs, cached responses
- **System entities**: Jobs, Logs, Settings, Health checks
The 100+ migrations indicate active schema development. This can complicate upgrades if migrations aren't carefully managed, but the Docker entrypoint.sh script handles automatic migration application on container startup.
### Audio Processing
FFmpeg handles transcoding for format conversion and bitrate adjustment. ImageSharp processes album art (resizing, format conversion, optimization). Audio tagging uses two libraries:
- **ATL (Audio Tools Library)**: Primary tagging engine, supports 20+ formats
- **IdSharp**: Fallback for ID3v2 edge cases
This dual-library approach suggests the developers encountered limitations in a single tagging library and opted for redundancy rather than forking or extensive patching.
## User-Facing Features
### Party Mode
SignalR-powered synchronized playback across multiple clients. One user controls playback, others see real-time updates. This feature differentiates Melodee from traditional music servers that treat each session independently.
Implementation likely uses SignalR groups to broadcast playback state changes. The Blazor Server architecture makes this natural since the SignalR connection already exists for UI updates.
### Podcast Support
Melodee handles podcast feeds alongside music libraries. This positions it as a unified media server rather than music-only. Podcast-specific features likely include:
- RSS feed parsing and updates
- Episode download management
- Playback position tracking
- Subscription management
The background job system handles periodic feed checks and episode downloads.
### MQL Query Language
Melodee implements a custom query language (MQL) for advanced library searches. This suggests power users can construct complex queries beyond simple text search. Examples might include:
- `artist:Radiohead AND year:>2000`
- `genre:Jazz OR genre:Blues`
- `playcount:>10 AND rating:>=4`
The implementation likely uses a parser (possibly ANTLR or hand-written recursive descent) to convert MQL strings into LINQ expressions or SQL queries.
### Charts and Analytics
The system generates charts based on listening history:
- Most played tracks/albums/artists
- Listening trends over time
- Genre distribution
- Discovery metrics (new vs. familiar content)
These features require the scrobbling system to capture play events and the background jobs to aggregate statistics.
### User Requests
Users can request missing albums or corrections. This creates a feedback loop where library gaps become visible to administrators. The feature likely stores requests as database entities with status tracking (pending, fulfilled, rejected).
## Internationalization
Support for 10 languages indicates a global user base or internationalization-first design. Blazor's localization system uses resource files (.resx) for string management. The 10 languages suggest community contributions for translations.
Language support affects:
- UI strings
- Error messages
- Email templates
- API documentation
The Scalar API documentation tool likely generates localized API docs automatically.
## Authentication and Security
### Google OAuth Integration
OAuth support allows users to authenticate with Google accounts rather than managing separate credentials. This reduces friction for new users and delegates security concerns to Google's infrastructure.
Implementation uses standard OAuth 2.0 authorization code flow:
1. User clicks "Sign in with Google"
2. Redirect to Google consent screen
3. Google redirects back with authorization code
4. Melodee exchanges code for access token
5. Melodee retrieves user profile
6. Melodee creates or updates local user account
### JWT for Native API
The native REST API uses JWT tokens for stateless authentication. Clients receive a token after login and include it in the `Authorization: Bearer <token>` header for subsequent requests.
JWT advantages:
- Stateless (no server-side session storage)
- Self-contained (claims embedded in token)
- Scalable (no session affinity required)
JWT tradeoffs:
- Token revocation requires additional infrastructure (blacklist or short expiry)
- Token size larger than session IDs
- Clock skew can cause validation issues
### Rate Limiting
Per-protocol rate limits prevent abuse:
- **API endpoints**: 30 requests per 30 seconds
- **Authentication**: 10 requests per 60 seconds
- **Jellyfin endpoints**: 200 requests per 60 seconds
The higher Jellyfin limit suggests those clients make more frequent requests, possibly for real-time playback state updates.
Rate limiting implementation likely uses in-memory sliding window counters keyed by IP address or user ID. For distributed deployments, this would require Redis or similar shared state.
## Observability
### Logging with Serilog
Serilog provides structured logging with two sinks:
- **Console**: Human-readable output for development and container logs
- **File (CLEF)**: Compact Log Event Format for machine parsing
CLEF (Compact Log Event Format) is JSON-based, making logs easily ingestible by log aggregation tools (Seq, Elasticsearch, Splunk). This suggests the developers anticipate production deployments where centralized logging matters.
### Health Checks
The `/health` endpoint exposes system status for monitoring tools. Health checks likely verify:
- Database connectivity
- Metadata provider availability
- Background job status
- Disk space
- Cache validity
Kubernetes and Docker Swarm can use this endpoint for liveness and readiness probes.
### Admin UI
Blazor-based admin interface provides visibility into:
- Job execution history and status
- User management
- Library statistics
- System settings
- Log viewing
This eliminates the need for database access or log file inspection for routine administration.
## Platform Compatibility
### Raspberry Pi Support
Explicit Raspberry Pi compatibility indicates ARM architecture support and resource-conscious design. Running on Raspberry Pi 4 (4GB RAM) requires:
- Efficient memory usage
- ARM64 .NET runtime
- Minimal CPU overhead for background jobs
- Optimized database queries
This positions Melodee as suitable for home server deployments on low-power hardware.
### Podman Support
Podman compatibility alongside Docker shows awareness of rootless container runtimes. Podman's daemonless architecture and rootless mode appeal to security-conscious users.
The Docker Compose file likely works with Podman Compose with minimal or no modifications. Volume mounts and networking must avoid Docker-specific assumptions.
## Development Practices
### Testing Strategy
Three testing frameworks indicate comprehensive test coverage:
1. **xUnit**: Unit and integration tests for business logic
2. **bUnit**: Blazor component testing
3. **NBomber**: Load and performance testing
The inclusion of NBomber suggests performance is a first-class concern. Load tests likely verify:
- API throughput under concurrent requests
- Database query performance with large libraries
- Memory usage during metadata aggregation
- Background job execution time
### Code Quality
Biome linting enforces code style and catches common errors. Biome is a fast, Rust-based linter and formatter that supports JavaScript, TypeScript, JSON, and CSS. Its presence suggests frontend code (likely for admin UI customization or build scripts) follows consistent style rules.
The combination of .NET analyzers (built into SDK) and Biome creates a multi-layered quality gate.
## Competitive Positioning
Melodee competes with established music servers:
- **Subsonic/Airsonic**: Older Java-based servers with large client ecosystems
- **Navidrome**: Go-based, lightweight, OpenSubsonic-compatible
- **Jellyfin**: Full media server (music, video, TV) with broad client support
- **Plex**: Commercial media server with free tier
- **Emby**: Commercial media server, Jellyfin's predecessor
Melodee's differentiators:
- **Metadata quality**: Six providers vs. typical 1-2
- **Multi-protocol**: Native + OpenSubsonic + Jellyfin vs. single protocol
- **Modern stack**: .NET 10 + Blazor vs. older frameworks
- **Party Mode**: Synchronized playback vs. independent sessions
- **MQL**: Advanced queries vs. basic search
The 62 stars suggest Melodee hasn't achieved mainstream adoption. This could reflect:
- Newer project (less time to accumulate stars)
- Niche appeal (power users who value metadata quality)
- Competition from established alternatives
- .NET ecosystem smaller than Go/Rust for self-hosted tools
## Use Cases
### Personal Music Library
Primary use case: individual managing a local music collection with high metadata standards. The six-provider aggregation ensures accurate artist names, release dates, genres, and album art even for obscure releases.
### Family Media Server
Multiple user accounts, playlists, and Party Mode support family sharing. Google OAuth simplifies account creation for non-technical family members.
### Podcast Aggregator
Podcast support makes Melodee a unified audio server. Users avoid separate podcast apps and music apps.
### Music Discovery Platform
Charts, analytics, and Last.fm integration enable discovery workflows. Users see listening patterns and explore similar artists.
### Development Platform
MIT license and modern .NET stack make Melodee suitable as a foundation for custom music server projects. Developers can fork and extend without licensing concerns.
## Limitations and Considerations
### Blazor Server Scalability
Persistent SignalR connections limit horizontal scaling. Each user consumes server memory and CPU for UI rendering. Scaling beyond 50-100 concurrent users requires careful architecture (Redis backplane, sticky sessions, or migration to Blazor WebAssembly).
### Metadata Provider Dependencies
Six providers create six points of failure. If MusicBrainz, Last.fm, or Spotify change APIs or rate limits, metadata quality degrades. The local MusicBrainz cache mitigates this for the primary provider.
### Migration Complexity
100+ migrations complicate upgrades, especially if users skip versions. The Docker entrypoint handles automatic migration, but rollback scenarios require careful planning.
### .NET Ecosystem
.NET 10 requires users comfortable with .NET runtime installation or Docker. This narrows the audience compared to Go or Rust single-binary distributions.
### Client Compatibility
While OpenSubsonic and Jellyfin APIs provide broad client support, the native API requires custom clients or API consumers. The project's 62 stars suggest limited native client development.
## Future Potential
### Federated Libraries
Multiple Melodee instances could federate, allowing users to share libraries across households while maintaining local control.
### Machine Learning
Listening history and metadata enable recommendation engines, auto-playlist generation, and mood-based categorization.
### Blockchain Integration
NFT-based music ownership or decentralized metadata storage could differentiate Melodee in web3 contexts.
### Mobile Apps
Native iOS and Android apps using the REST API would reduce dependence on third-party clients.
### Video Support
Expanding beyond audio to music videos or concerts would position Melodee as a full media server competitor to Jellyfin and Plex.
## Conclusion
Melodee represents a technically sophisticated music server built on modern .NET foundations. The multi-protocol API support, six-provider metadata aggregation, and Blazor Server UI create a compelling package for users who prioritize metadata quality and extensibility.
The project's 62 stars indicate niche appeal rather than mainstream adoption. This likely reflects the competitive landscape (established alternatives like Navidrome and Jellyfin) and the .NET ecosystem's smaller footprint in self-hosted software compared to Go or Rust.
For developers evaluating music server options, Melodee offers:
- **Strengths**: Metadata quality, modern stack, multi-protocol support, MIT license
- **Tradeoffs**: Blazor Server scalability, .NET runtime dependency, smaller community
The project's active development (version 1.8.0, 100+ migrations) suggests ongoing improvement. Whether Melodee achieves broader adoption depends on community growth, client ecosystem development, and continued differentiation from established competitors.