metadata-agregator/docs/research/meelo/analysis/ARCHITECTURE.md

Meelo Architecture

System Overview

Meelo implements a microservices architecture with four application services and four infrastructure services, orchestrated via Docker Compose. Each service has a single responsibility and communicates through well-defined interfaces (REST APIs, message queues).

┌─────────────────────────────────────────────────────────────┐
│                          Nginx                              │
│  Reverse Proxy (Port 80)                                    │
│  Routes: / → Front, /api/ → Server, /scanner/ → Scanner    │
└─────────────────────────────────────────────────────────────┘
         │              │                │              │
    ┌────┘         ┌────┘           ┌────┘         ┌────┘
    │              │                │              │
┌───▼────┐   ┌────▼─────┐   ┌──────▼───┐   ┌──────▼────┐
│ Front  │   │  Server  │   │ Scanner  │   │  Matcher  │
│ Next.js│   │  NestJS  │   │   Go     │   │  FastAPI  │
│ :3000  │   │  :4000   │   │  :8133   │   │   :6789   │
└────────┘   └────┬─────┘   └────┬─────┘   └─────┬─────┘
                  │              │               │
         ┌────────┼──────────────┼───────────────┘
         │        │              │
    ┌────▼───┐ ┌─▼──────────┐ ┌─▼──────────┐
    │ Postgres│ │ MeiliSearch│ │  RabbitMQ  │
    │  :5432  │ │   :7700    │ │   :5672    │
    └─────────┘ └────────────┘ └────────────┘

Service Responsibilities

Server (NestJS 11, TypeScript)

Port: 4000
Database: PostgreSQL via Prisma ORM
Search: MeiliSearch client
Messaging: RabbitMQ publisher

Module Structure

NestJS organizes code into modules. Each module encapsulates related functionality:

Core Domain Modules

• ArtistModule: CRUD operations, relationships to albums/songs/videos
• AlbumModule: Album management, release associations
• SongModule: Song entities, track relationships, lyrics
• TrackModule: Individual track instances (audio/video)
• ReleaseModule: Physical/digital release variants
• GenreModule: Genre taxonomy and associations
• VideoModule: Music video management

Supporting Modules

• AuthModule: JWT authentication, user registration, login
• UserModule: User management, preferences, scrobbler connections
• LibraryModule: Library configuration, scan triggers
• FileModule: File metadata, checksums, fingerprints
• PlaylistModule: Playlist CRUD, entry management
• LyricsModule: Plain and synced lyrics storage

Integration Modules

• ExternalMetadataModule: Provider data aggregation
• SearchModule: MeiliSearch indexing and queries
• ScrobblerModule: Last.fm and ListenBrainz integration
• StreamModule: Audio/video streaming endpoints
• EventsModule: WebSocket notifications for UI updates

Infrastructure Modules

• PrismaModule: Database connection and ORM
• MeiliSearchModule: Search client configuration
• RabbitMQModule: Message queue publisher

Data Flow

1. Incoming Request: Nginx forwards to Server at /api/*
2. Controller: Route handler validates request, extracts JWT
3. Service: Business logic executes, calls Prisma for data
4. Repository: Prisma queries PostgreSQL
5. Response: JSON returned to client

For write operations:

1. Service updates database via Prisma
2. Service publishes event to RabbitMQ (if needed)
3. Service updates MeiliSearch index
4. Service emits WebSocket event for live UI updates

Authentication Flow

1. User submits credentials to /api/auth/login
2. AuthService validates against bcrypt hash in database
3. JWT signed with JWT_SIGNATURE from .env
4. Token returned to client
5. Client includes token in Authorization: Bearer <token> header
6. JwtStrategy validates token on protected routes
7. User object attached to request context

Anonymous mode (ALLOW_ANONYMOUS=1) bypasses this flow.

Scrobbling Flow

1. User authorizes Last.fm via OAuth (callback to /api/scrobblers/lastfm/callback)
2. Server exchanges code for access token
3. Token stored in UserScrobbler table
4. On track play, ScrobblerService posts to Last.fm API
5. ListenBrainz uses simpler token-based auth (user provides token directly)

Search Integration

1. On entity creation/update, service calls MeiliSearchService.index()
2. Service transforms entity to search document
3. Document pushed to MeiliSearch via HTTP API
4. Client queries /api/search?q=<term>
5. Server forwards to MeiliSearch
6. Results enriched with database data (illustrations, counts)
7. JSON returned to client

Scanner (Go 1.25, Echo v5)

Port: 8133
Framework: Echo HTTP server
Dependencies: FFmpeg, FFprobe, AcoustID

Responsibilities

1. Filesystem Watching: Monitor library directories for changes
2. Metadata Extraction: Parse audio/video files using FFprobe
3. Fingerprinting: Generate AcoustID fingerprints for matching
4. Filename Parsing: Apply regex from settings.json to extract metadata
5. File Registration: POST file metadata to Server API
6. Match Triggering: Publish events to RabbitMQ for Matcher consumption

Scan Process

1. Trigger: POST to /scanner/scan/:libraryId or filesystem event
2. Discovery: Walk directory tree, filter by extension (.mp3, .flac, .m4a, .mkv, etc.)
3. Extraction: For each file:
   • Run FFprobe to get duration, bitrate, codec, embedded tags
   • Generate AcoustID fingerprint using chromaprint
   • Parse filename using regex from settings.json
   • Calculate file checksum (SHA256)
4. Registration: POST to Server /api/files with:
   • File path
   • Checksum
   • Fingerprint
   • Extracted metadata (title, artist, album, track number)
   • Technical details (duration, bitrate, codec)
5. Event Publishing: Publish to RabbitMQ queue file.added with file ID
6. Repeat: Process next file

Filename Regex

Settings.json contains trackRegex pattern. Example:

(?P<artist>[^/]+)/(?P<album>[^/]+)/(?P<disc>\d+)-(?P<track>\d+) (?P<title>.+)\.(?P<ext>\w+)

Named capture groups extract metadata when embedded tags are missing or untrusted.

Health Monitoring

Scanner exposes GET / endpoint. Returns JSON with:

• Service status
• Active scan tasks
• Last scan timestamp
• Library statistics

Docker health check hits this endpoint every 30 seconds.

Error Handling

• File Read Errors: Log and skip file, continue scan
• FFprobe Failures: Retry once, then skip
• Server API Errors: Retry with exponential backoff (max 3 attempts)
• RabbitMQ Unavailable: Queue events in memory, flush when connection restored

Matcher (Python 3.14, FastAPI)

Port: 6789
Framework: FastAPI with async HTTP
Messaging: RabbitMQ consumer

Responsibilities

1. Event Consumption: Listen to RabbitMQ file.added queue
2. Provider Queries: Fetch metadata from 8 external sources
3. Data Aggregation: Merge results based on priority in settings.json
4. Metadata Push: POST enriched data to Server API

Provider Architecture

Each provider is a separate module implementing a common interface:

class Provider(ABC):
    @abstractmethod
    async def search_track(self, fingerprint: str, title: str, artist: str) -> Optional[TrackMetadata]:
        pass
    
    @abstractmethod
    async def fetch_artist(self, artist_id: str) -> Optional[ArtistMetadata]:
        pass
    
    @abstractmethod
    async def fetch_album(self, album_id: str) -> Optional[AlbumMetadata]:
        pass

Provider Modules

• musicbrainz.py: Primary database, uses musicbrainzngs library
• genius.py: Lyrics and song descriptions, requires API token
• wikipedia.py: Artist/album context, uses Wikipedia API
• wikidata.py: Structured data (areas, relationships), SPARQL queries
• discogs.py: Release details, requires API token
• allmusic.py: Editorial reviews, web scraping (no official API)
• metacritic.py: Critic scores, web scraping
• lrclib.py: Synced lyrics, public API

Matching Flow

1. Event Received: RabbitMQ delivers file.added message with file ID
2. File Fetch: GET /api/files/:id from Server to retrieve metadata
3. Provider Selection: Read settings.json for enabled providers and priority
4. Parallel Queries: Launch async tasks for each provider:
   • MusicBrainz: Query by AcoustID fingerprint
   • Genius: Search by title + artist
   • Wikipedia: Search by artist name
   • Wikidata: Query by MusicBrainz ID (if found)
   • Discogs: Search by release title
   • AllMusic: Scrape by artist + album
   • Metacritic: Scrape by album title
   • LrcLib: Search by title + artist + duration
5. Result Aggregation: Merge results based on priority:
   • MusicBrainz IDs take precedence
   • Lyrics: prefer synced (LrcLib) over plain (Genius)
   • Descriptions: concatenate from multiple sources
   • Ratings: average across providers
6. Metadata Push: POST to Server /api/external-metadata with:
   • Track/album/artist IDs
   • Descriptions
   • Ratings
   • Source URLs
   • Provider names
7. Acknowledgment: ACK message to RabbitMQ

Rate Limiting

Providers have different rate limits:

• MusicBrainz: 1 request/second (enforced by library)
• Genius: 10 requests/second (API limit)
• Wikipedia: No official limit, use 5 requests/second
• Wikidata: No limit, SPARQL endpoint is fast
• Discogs: 60 requests/minute (API limit)
• AllMusic: No API, scraping limited to 1 request/second
• Metacritic: No API, scraping limited to 1 request/second
• LrcLib: No official limit, use 10 requests/second

Matcher implements per-provider rate limiters using aiolimiter.

Error Handling

• Provider Timeout: Skip provider, continue with others
• HTTP Errors: Retry with exponential backoff (max 3 attempts)
• Parsing Errors: Log and skip provider result
• Server API Errors: NACK message to RabbitMQ for redelivery
• No Results: Push empty metadata (Server marks as "not found")

Configuration

Settings.json controls provider behavior:

{
  "providers": {
    "musicbrainz": { "enabled": true },
    "genius": { "enabled": true, "token": "..." },
    "wikipedia": { "enabled": true },
    "wikidata": { "enabled": true },
    "discogs": { "enabled": false },
    "allmusic": { "enabled": false },
    "metacritic": { "enabled": false },
    "lrclib": { "enabled": true }
  },
  "metadata": {
    "order": ["musicbrainz", "genius", "wikipedia", "lrclib"]
  }
}

Disabled providers are skipped. Order determines priority for conflicting data.

Front (Next.js 16, React)

Port: 3000
Framework: Next.js with SSR
UI: Material-UI components
State: Jotai atoms
Data Fetching: TanStack Query
i18n: i18next

Responsibilities

1. User Interface: Render pages for browsing, playback, settings
2. API Communication: Fetch data from Server via REST
3. State Management: Manage playback queue, user preferences, auth tokens
4. Internationalization: Support multiple languages

Page Structure

• /: Home page with recent albums, top artists
• /artists: Artist grid with search
• /artists/:id: Artist detail with albums, songs, videos
• /albums: Album grid with filters
• /albums/:id: Album detail with tracks, releases
• /songs: Song list with search
• /songs/:id: Song detail with tracks, lyrics
• /playlists: User playlists
• /playlists/:id: Playlist detail with tracks
• /videos: Music video grid
• /videos/:id: Video player
• /search: Global search results
• /settings: User preferences, library management, scrobbler setup

State Management

Jotai atoms store global state:

• authAtom: JWT token, user info
• playbackAtom: Current track, queue, position, volume
• settingsAtom: Theme, language, playback preferences

TanStack Query caches API responses:

• useArtists(): Fetch artist list
• useArtist(id): Fetch artist detail
• useAlbums(): Fetch album list
• useAlbum(id): Fetch album detail
• useTracks(): Fetch track list
• useSearch(query): Fetch search results

Queries invalidate on mutations (create playlist, update settings).

Playback Flow

1. User clicks track
2. playbackAtom updated with track ID
3. Component fetches stream URL: /api/tracks/:id/stream
4. HTML5 <audio> element loads stream
5. Playback starts
6. On play event, POST to /api/scrobblers/scrobble (if enabled)
7. On track end, advance queue, repeat flow

Video playback uses <video> element with transcoder stream.

Mobile App

Expo/React Native app shares components and state logic with web. Differences:

• Navigation: React Navigation instead of Next.js router
• Storage: AsyncStorage instead of localStorage
• Media: expo-av instead of HTML5 audio/video
• Notifications: expo-notifications for background playback

Monorepo structure:

front/
  web/          # Next.js app
  mobile/       # Expo app
  shared/       # Common components, hooks, state

Internationalization

i18next with JSON translation files:

locales/
  en/
    common.json
    artist.json
    album.json
  fr/
    common.json
    artist.json
    album.json

Language switcher in settings. Detects browser locale on first visit.

Infrastructure Services

PostgreSQL

Port: 5432
Image: postgres:alpine3.14
Volume: meelo_db

Stores all persistent data. Prisma manages schema migrations. Health check via pg_isready.

MeiliSearch

Port: 7700
Image: meilisearch:v1.5
Volume: meelo_search

Indexes artists, albums, songs, videos. Configured with:

• Searchable attributes: name, title, artist names
• Filterable attributes: genre, year, type
• Sortable attributes: releaseDate, name
• Ranking rules: typo, words, proximity, attribute, sort, exactness

Health check via GET /health.

RabbitMQ

Port: 5672 (AMQP), 15672 (management UI)
Image: rabbitmq:4.2-alpine
Volume: meelo_rabbitmq_data

Message queue for event-driven architecture. Queues:

• file.added: Scanner publishes, Matcher consumes
• metadata.updated: Matcher publishes, Server consumes (future use)

Health check via rabbitmq-diagnostics ping.

Kyoo Transcoder

Port: 7666
Volume: meelo_transcoder_cache

Transcodes video files for web playback. Supports:

• Adaptive bitrate streaming (HLS)
• Multiple resolutions (480p, 720p, 1080p)
• Codec conversion (H.264, VP9)
• Subtitle burning

Server proxies requests to transcoder. Client receives HLS manifest.

Nginx

Port: 80
Image: nginx:1.29.7-alpine
Config: Mounted from nginx.conf

Routes requests to services:

location / {
    proxy_pass http://front:3000;
}

location /api/ {
    proxy_pass http://server:4000;
}

location /scanner/ {
    proxy_pass http://scanner:8133;
}

location /matcher/ {
    proxy_pass http://matcher:6789;
}

Handles WebSocket upgrades for Server events.

Inter-Service Communication

REST APIs

• Front → Server: All data fetching (artists, albums, tracks, playlists)
• Scanner → Server: File registration, library queries
• Matcher → Server: Metadata push, file queries
• Server → MeiliSearch: Index updates, search queries
• Server → Transcoder: Video stream requests

Message Queue

• Scanner → RabbitMQ: Publish file.added events
• RabbitMQ → Matcher: Deliver file.added events

Database

• Server → PostgreSQL: All CRUD operations via Prisma

Startup Orchestration

Docker Compose defines service dependencies and health checks:

1. PostgreSQL starts first, health check via pg_isready
2. MeiliSearch starts, health check via GET /health
3. RabbitMQ starts, health check via rabbitmq-diagnostics ping
4. Server starts after database/search/queue are healthy
   • Runs Prisma migrations
   • Seeds initial data (admin user if none exists)
   • Connects to MeiliSearch and RabbitMQ
5. Scanner starts after Server is healthy
   • Registers with Server API
   • Begins filesystem watching
6. Matcher starts after Server and RabbitMQ are healthy
   • Connects to RabbitMQ
   • Begins consuming events
7. Front starts after Server is healthy
   • SSR requires Server API for initial data
8. Transcoder starts independently (no dependencies)
9. Nginx starts last, after all application services are healthy

Health checks run every 30 seconds. Unhealthy services restart automatically.

Data Consistency

Transactions

Prisma transactions ensure atomicity:

await prisma.$transaction([
  prisma.song.create({ data: songData }),
  prisma.track.create({ data: trackData }),
  prisma.file.update({ where: { id: fileId }, data: { trackId } })
]);

If any operation fails, all rollback.

Event Ordering

RabbitMQ guarantees message order per queue. Matcher processes events sequentially to avoid race conditions.

Search Consistency

MeiliSearch updates are asynchronous. Brief window where database and search index diverge. Acceptable for this use case (eventual consistency).

Cache Invalidation

TanStack Query invalidates caches on mutations:

const mutation = useMutation({
  mutationFn: createPlaylist,
  onSuccess: () => {
    queryClient.invalidateQueries(['playlists']);
  }
});

Scalability Considerations

Horizontal Scaling

• Scanner: Run multiple instances for different libraries
• Matcher: Run multiple consumers for faster enrichment
• Front: Stateless, can run multiple instances behind load balancer

Vertical Scaling

• Server: CPU-bound for complex queries, benefits from more cores
• MeiliSearch: Memory-bound, benefits from more RAM
• PostgreSQL: I/O-bound, benefits from SSD and connection pooling

Bottlenecks

• Matcher: Limited by external provider rate limits
• Transcoder: CPU-intensive, limits concurrent video streams
• Database: Complex queries (artist with all albums/songs/videos) can be slow

Monitoring and Observability

Logging

• Server: NestJS Logger with configurable levels (error, warn, info, debug)
• Scanner: zerolog with structured JSON output
• Matcher: Python logging with JSON formatter
• Front: Console logs in development, silent in production

All logs written to stdout, captured by Docker.

Health Checks

Every service exposes health endpoint:

• Server: GET /api/health
• Scanner: GET /
• Matcher: GET /health
• Front: GET /api/health (Next.js API route)

Docker Compose monitors these endpoints.

Metrics

No built-in Prometheus metrics. Future enhancement.

Security Architecture

Authentication

• JWT: Signed tokens with expiration
• API Keys: x-api-key header for Scanner/Matcher
• Bcrypt: Password hashing with salt rounds = 10

Authorization

• Admin Flag: Users have isAdmin boolean
• Ownership: Users can only modify their own playlists
• Public Playlists: Readable by all, writable by owner or if allowChanges=true

Network Isolation

Docker Compose creates private network. Only Nginx exposes port 80. Internal services not accessible from host.

Input Validation

• Server: NestJS validation pipes with class-validator
• Scanner: Go struct validation
• Matcher: Pydantic models

Invalid input returns 400 Bad Request.

SQL Injection

Prisma uses parameterized queries. No raw SQL in codebase.

XSS Protection

React escapes output by default. No dangerouslySetInnerHTML except for sanitized lyrics.

Deployment Variants

Production (docker-compose.yml)

Pre-built images from Docker Hub. Environment variables from .env. Volumes for persistence. Restart policy: always.

Development (docker-compose.dev.yml)

Mounted source directories. Hot reload enabled. Exposed ports for debugging (PostgreSQL 5432, MeiliSearch 7700, RabbitMQ 15672). Restart policy: unless-stopped.

Local Build (docker-compose.local.yml)

Builds images from source using Dockerfiles. Tests changes before pushing to Docker Hub. Same volumes and network as production.

Configuration Management

Environment Variables (.env)

Deployment-specific settings:

• PORT: Server port (default 4000)
• PUBLIC_URL: External URL for OAuth callbacks
• CONFIG_DIR: Path to settings.json
• DATA_DIR: Path to music files
• JWT_SIGNATURE: Secret for signing tokens
• GENIUS_ACCESS_TOKEN: Genius API key
• DISCOGS_ACCESS_TOKEN: Discogs API key
• LASTFM_API_KEY, LASTFM_API_SECRET: Last.fm OAuth

Settings File (settings.json)

User preferences:

• trackRegex: Filename parsing pattern
• metadata.source: Prefer embedded tags or external providers
• metadata.order: Provider priority list
• providers: Enable/disable specific providers
• compilations: Rules for detecting compilation albums

Server reads settings.json on startup. Changes require restart.

Error Recovery

Service Failures

Docker restart policy handles crashes. Health checks detect hung processes.

Database Corruption

PostgreSQL volume backups recommended. Restore from backup if corruption detected.

Message Queue Failures

RabbitMQ persists messages to disk. Unacknowledged messages redelivered on restart.

Search Index Corruption

Rebuild MeiliSearch index from database:

curl -X POST http://localhost:4000/api/search/reindex

Server iterates all entities, pushes to MeiliSearch.

Performance Optimization

Database Indexes

Prisma schema defines indexes on:

• Foreign keys (artistId, albumId, songId)
• Unique constraints (slug, checksum)
• Frequently queried fields (releaseDate, type)

Query Optimization

• Eager Loading: Prisma include to avoid N+1 queries
• Pagination: Limit/offset for large result sets
• Caching: TanStack Query caches API responses client-side

Asset Optimization

• Images: Illustrations stored as blurhash + URL
• Lazy Loading: Front loads images on scroll
• Code Splitting: Next.js splits bundles per page

Testing Strategy

Unit Tests

• Server: Jest tests for services, controllers, utilities
• Matcher: pytest tests for provider modules
• Scanner: Go tests for file parsing, fingerprinting

Integration Tests

• Server: Test API endpoints with in-memory database
• Matcher: Mock external provider responses

End-to-End Tests

Not implemented. Future enhancement with Playwright.

Coverage

SonarCloud tracks coverage per service. Minimum threshold: 80%.

Summary

Meelo's architecture separates concerns across four microservices, each optimized for its task. The event-driven design decouples scanning from enrichment, enabling parallel processing and fault tolerance. Infrastructure services (PostgreSQL, MeiliSearch, RabbitMQ) provide persistence, search, and messaging. Docker Compose orchestrates startup order and health monitoring. The result is a scalable, maintainable system that handles complex metadata workflows without blocking user interactions.