feat: initial implementation of metadata aggregator
- gRPC service with MusicBrainz provider - PostgreSQL schema with migrations - Service layer with database-first caching - Repository pattern for data access - YAML configuration support - Research documentation for 17 music metadata projects
This commit is contained in:
Alexander
@@ -0,0 +1,564 @@
|
||||
# Meelo Evaluation
|
||||
|
||||
## Strengths
|
||||
|
||||
### Data Model Sophistication
|
||||
|
||||
Meelo's data model is the most mature among self-hosted music servers. The Album/Release and Song/Track distinctions accurately represent real-world music organization.
|
||||
|
||||
**Album vs Release**:
|
||||
- Albums are abstract concepts (e.g., "Abbey Road")
|
||||
- Releases are physical/digital manifestations (original, 2019 remaster, deluxe edition)
|
||||
- One album can have multiple releases with different track listings, mastering, labels
|
||||
|
||||
This mirrors how music collectors think. A remaster is not a different album, it's a different release of the same album.
|
||||
|
||||
**Song vs Track**:
|
||||
- Songs are compositions (e.g., "Come Together")
|
||||
- Tracks are recordings (studio version, live version, acoustic version)
|
||||
- One song can have multiple tracks across different releases
|
||||
|
||||
This enables tracking different performances of the same composition without creating duplicate songs.
|
||||
|
||||
**Song Groups**:
|
||||
- Group versions of the same composition (original, covers, remixes)
|
||||
- Example: "Hallelujah" by Leonard Cohen, Jeff Buckley, Pentatonix
|
||||
- Enables discovering different interpretations
|
||||
|
||||
No other self-hosted music server implements this level of versioning.
|
||||
|
||||
### Multi-Provider Metadata
|
||||
|
||||
Meelo queries 8 external providers:
|
||||
1. **MusicBrainz**: Primary database, most accurate
|
||||
2. **Genius**: Lyrics and song descriptions
|
||||
3. **Wikipedia**: Artist/album context
|
||||
4. **Wikidata**: Structured data
|
||||
5. **Discogs**: Release details
|
||||
6. **AllMusic**: Editorial reviews
|
||||
7. **Metacritic**: Critic scores
|
||||
8. **LrcLib**: Synced lyrics
|
||||
|
||||
**Aggregation Strategy**:
|
||||
- Priority-based merging (MusicBrainz > Genius > Wikipedia)
|
||||
- Concatenate descriptions from multiple sources
|
||||
- Average ratings across providers
|
||||
- Prefer synced lyrics over plain
|
||||
|
||||
**Result**: Richer metadata than single-provider systems. Descriptions combine MusicBrainz facts, Wikipedia context, and Genius annotations.
|
||||
|
||||
### Music Video Support
|
||||
|
||||
Videos are first-class citizens, not afterthoughts.
|
||||
|
||||
**Video Types**:
|
||||
- Official music videos
|
||||
- Live performances
|
||||
- Lyric videos
|
||||
- Behind the scenes
|
||||
- Interviews
|
||||
- Documentaries
|
||||
|
||||
**Integration**:
|
||||
- Videos link to songs (same as audio tracks)
|
||||
- Kyoo transcoder handles adaptive streaming
|
||||
- UI treats videos equally with audio
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: No video support
|
||||
- **Jellyfin**: Videos are separate media type, not linked to songs
|
||||
- **Plex**: Similar to Jellyfin
|
||||
|
||||
Meelo is the only self-hosted music server with proper music video integration.
|
||||
|
||||
### Event-Driven Architecture
|
||||
|
||||
RabbitMQ decouples scanning from enrichment.
|
||||
|
||||
**Flow**:
|
||||
1. Scanner registers file with Server
|
||||
2. Scanner publishes event to RabbitMQ
|
||||
3. Matcher consumes event asynchronously
|
||||
4. Matcher queries providers in parallel
|
||||
5. Matcher pushes enriched metadata to Server
|
||||
|
||||
**Benefits**:
|
||||
- Scanning doesn't block on provider queries
|
||||
- Matcher can retry failed providers without re-scanning
|
||||
- Multiple matchers can process events in parallel
|
||||
- Provider failures don't stop scanning
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Synchronous metadata fetching blocks scanning
|
||||
- **Airsonic**: No external metadata providers
|
||||
|
||||
### Scrobbling Built-In
|
||||
|
||||
Last.fm and ListenBrainz integration is native, not a plugin.
|
||||
|
||||
**Features**:
|
||||
- OAuth flow for Last.fm
|
||||
- Token-based auth for ListenBrainz
|
||||
- Automatic scrobbling on track play
|
||||
- "Now playing" updates
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Last.fm only, requires external scrobbler
|
||||
- **Airsonic**: No built-in scrobbling
|
||||
|
||||
### Mobile App
|
||||
|
||||
Expo/React Native app shares code with web frontend.
|
||||
|
||||
**Shared**:
|
||||
- Components (ArtistCard, AlbumCard, TrackList)
|
||||
- Hooks (useArtists, useAlbums, usePlayback)
|
||||
- State management (Jotai atoms)
|
||||
|
||||
**Mobile-Specific**:
|
||||
- React Navigation instead of Next.js router
|
||||
- AsyncStorage instead of localStorage
|
||||
- expo-av for media playback
|
||||
- expo-notifications for background playback
|
||||
|
||||
**Result**: Feature parity between web and mobile without duplicating code.
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Third-party mobile apps (Substreamer, Subtracks)
|
||||
- **Jellyfin**: Official mobile app, but music is secondary
|
||||
|
||||
### Search Performance
|
||||
|
||||
MeiliSearch provides sub-100ms search across large libraries.
|
||||
|
||||
**Features**:
|
||||
- Typo tolerance (handles misspellings)
|
||||
- Faceted search (filter by genre, year, type)
|
||||
- Instant results (as-you-type)
|
||||
- Relevance ranking
|
||||
|
||||
**Indexed Entities**:
|
||||
- Artists (name, sort name)
|
||||
- Albums (name, artist name, type, release date)
|
||||
- Songs (name, artist name, type)
|
||||
- Videos (name, artist name, type)
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Database full-text search (slower, no typo tolerance)
|
||||
- **Airsonic**: Basic SQL LIKE queries
|
||||
|
||||
### Active Development
|
||||
|
||||
**Indicators**:
|
||||
- 40 releases (consistent iteration)
|
||||
- 1,095 stars (healthy community)
|
||||
- GitHub Actions CI/CD per service
|
||||
- SonarCloud quality gates
|
||||
- Regular commits (weekly)
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Active (single maintainer)
|
||||
- **Airsonic**: Stagnant (last release 2020)
|
||||
- **Funkwhale**: Active but slower
|
||||
|
||||
### Geographic Context
|
||||
|
||||
Areas (countries, cities, regions) are first-class entities.
|
||||
|
||||
**Features**:
|
||||
- ISO 3166 codes
|
||||
- Parent/child hierarchy (city → state → country)
|
||||
- Artist associations (birthplace, formation location)
|
||||
|
||||
**Use Case**:
|
||||
- Browse artists by location
|
||||
- Discover local music scenes
|
||||
- Understand artist context
|
||||
|
||||
**Comparison**: No other self-hosted music server has area support.
|
||||
|
||||
### Code Quality
|
||||
|
||||
**Measures**:
|
||||
- SonarCloud enforces 80% coverage, no critical bugs
|
||||
- Biome linting for TypeScript
|
||||
- Pyright type checking for Python
|
||||
- golangci-lint for Go
|
||||
- Jest, pytest, Go testing
|
||||
|
||||
**Result**: High code quality, low bug rate.
|
||||
|
||||
## Weaknesses
|
||||
|
||||
### Complex Deployment
|
||||
|
||||
8+ containers required:
|
||||
1. Server (NestJS)
|
||||
2. Scanner (Go)
|
||||
3. Matcher (Python)
|
||||
4. Front (Next.js)
|
||||
5. PostgreSQL
|
||||
6. MeiliSearch
|
||||
7. RabbitMQ
|
||||
8. Kyoo Transcoder
|
||||
9. Nginx
|
||||
|
||||
**Challenges**:
|
||||
- Docker Compose orchestration
|
||||
- Health check dependencies
|
||||
- Volume management
|
||||
- Network configuration
|
||||
- Resource allocation
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Single binary, no dependencies
|
||||
- **Airsonic**: Single JAR, embedded database option
|
||||
|
||||
**Impact**: High barrier to entry for non-technical users.
|
||||
|
||||
### Multi-Language Stack
|
||||
|
||||
4 languages across services:
|
||||
- TypeScript (Server, Front)
|
||||
- Go (Scanner)
|
||||
- Python (Matcher)
|
||||
- TypeScript again (Front mobile)
|
||||
|
||||
**Challenges**:
|
||||
- Different toolchains (npm, go, pip)
|
||||
- Different testing frameworks (Jest, Go testing, pytest)
|
||||
- Different linting tools (Biome, golangci-lint, Ruff)
|
||||
- Harder to contribute (need expertise in multiple languages)
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Single language (Go)
|
||||
- **Airsonic**: Single language (Java)
|
||||
|
||||
**Impact**: Steeper learning curve for contributors.
|
||||
|
||||
### Heavy Infrastructure
|
||||
|
||||
Required services:
|
||||
- **PostgreSQL**: Relational database
|
||||
- **MeiliSearch**: Search engine
|
||||
- **RabbitMQ**: Message queue
|
||||
- **Kyoo Transcoder**: Video transcoding
|
||||
|
||||
**Resource Requirements**:
|
||||
- Minimum: 4GB RAM, 2 CPU cores
|
||||
- Recommended: 8GB RAM, 4 CPU cores
|
||||
- Storage: 10GB + library size
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: 512MB RAM, 1 CPU core, SQLite
|
||||
- **Airsonic**: 1GB RAM, 1 CPU core, embedded database
|
||||
|
||||
**Impact**: Not suitable for low-power devices (Raspberry Pi 3, old NAS).
|
||||
|
||||
### Requires Clean Collection
|
||||
|
||||
Meelo works best with well-organized music:
|
||||
- Embedded metadata (ID3 tags, Vorbis comments)
|
||||
- Standard folder structure (Artist/Album/Track)
|
||||
- Consistent naming
|
||||
|
||||
**Challenges**:
|
||||
- Messy collections require manual cleanup
|
||||
- Missing tags need filename regex
|
||||
- Inconsistent naming breaks matching
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: More forgiving, uses folder structure
|
||||
- **Jellyfin**: Handles messy collections better
|
||||
|
||||
**Impact**: Not suitable for users with poorly organized libraries.
|
||||
|
||||
### GPL-3.0 License
|
||||
|
||||
**Restrictions**:
|
||||
- Derivative works must be GPL-3.0
|
||||
- Source code must be disclosed
|
||||
- No proprietary forks
|
||||
|
||||
**Impact**:
|
||||
- Prevents commercial SaaS offerings
|
||||
- Limits corporate adoption
|
||||
- Acceptable for self-hosters, restrictive for businesses
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: GPL-3.0 (same restrictions)
|
||||
- **Jellyfin**: GPL-2.0 (similar restrictions)
|
||||
- **Airsonic**: GPL-3.0 (same restrictions)
|
||||
|
||||
### Kyoo Transcoder Dependency
|
||||
|
||||
Video transcoding relies on external project (Kyoo).
|
||||
|
||||
**Risks**:
|
||||
- Kyoo development stalls
|
||||
- Breaking changes in Kyoo API
|
||||
- Meelo must maintain compatibility
|
||||
|
||||
**Comparison**:
|
||||
- **Jellyfin**: Built-in transcoder (FFmpeg wrapper)
|
||||
- **Plex**: Built-in transcoder
|
||||
|
||||
**Impact**: Video support is fragile.
|
||||
|
||||
### No Prometheus Metrics
|
||||
|
||||
No built-in metrics for monitoring.
|
||||
|
||||
**Missing**:
|
||||
- Request rates
|
||||
- Error rates
|
||||
- Latency percentiles
|
||||
- Queue depths
|
||||
- Provider response times
|
||||
|
||||
**Workaround**: Parse logs or use external monitoring.
|
||||
|
||||
**Comparison**:
|
||||
- **Navidrome**: Prometheus metrics endpoint
|
||||
- **Jellyfin**: No metrics
|
||||
|
||||
**Impact**: Harder to monitor in production.
|
||||
|
||||
## Integration Potential
|
||||
|
||||
### Data Model
|
||||
|
||||
**Applicability**: Excellent reference for metadata aggregator.
|
||||
|
||||
**Lessons**:
|
||||
- Separate abstract entities (Album, Song) from concrete instances (Release, Track)
|
||||
- Use song groups for versioning
|
||||
- Store external metadata separately from core entities
|
||||
- Use local identifiers for cross-referencing
|
||||
|
||||
**Adoption**:
|
||||
- Implement Album/Release distinction
|
||||
- Implement Song/Track distinction
|
||||
- Implement song groups for covers/remixes
|
||||
- Separate ExternalMetadata table
|
||||
|
||||
### Provider Pattern
|
||||
|
||||
**Applicability**: Directly applicable to metadata aggregator.
|
||||
|
||||
**Architecture**:
|
||||
- Base provider interface (search, fetch)
|
||||
- Per-provider modules (musicbrainz.py, genius.py)
|
||||
- Factory pattern for provider instantiation
|
||||
- Parallel queries with asyncio
|
||||
- Rate limiting per provider
|
||||
- Priority-based aggregation
|
||||
|
||||
**Adoption**:
|
||||
- Copy provider interface design
|
||||
- Implement factory pattern
|
||||
- Use asyncio for parallel queries
|
||||
- Implement per-provider rate limiters
|
||||
- Use priority-based merging
|
||||
|
||||
### Event-Driven Enrichment
|
||||
|
||||
**Applicability**: Scalable approach for metadata aggregator.
|
||||
|
||||
**Architecture**:
|
||||
- Scanner publishes events to queue
|
||||
- Matcher consumes events asynchronously
|
||||
- Server receives enriched metadata via API
|
||||
- Decouples scanning from enrichment
|
||||
|
||||
**Adoption**:
|
||||
- Use message queue (RabbitMQ, Redis Streams)
|
||||
- Separate scanner and matcher services
|
||||
- Enable retries without re-scanning
|
||||
|
||||
### Search Integration
|
||||
|
||||
**Applicability**: Fast search is critical for metadata aggregator.
|
||||
|
||||
**Architecture**:
|
||||
- MeiliSearch for full-text search
|
||||
- Index on entity creation/update
|
||||
- Typo tolerance and faceted search
|
||||
- Sub-100ms response times
|
||||
|
||||
**Adoption**:
|
||||
- Integrate MeiliSearch or Typesense
|
||||
- Index artists, albums, songs
|
||||
- Implement as-you-type search
|
||||
|
||||
## Relevance to Metadata Aggregator
|
||||
|
||||
### High Relevance
|
||||
|
||||
**Data Model**:
|
||||
- Album/Release and Song/Track distinctions are essential for accurate metadata
|
||||
- Song groups enable tracking versions and covers
|
||||
- External metadata separation keeps provider data clean
|
||||
|
||||
**Provider Architecture**:
|
||||
- Factory pattern simplifies adding new providers
|
||||
- Parallel queries optimize performance
|
||||
- Rate limiting prevents API bans
|
||||
- Priority-based aggregation ensures quality
|
||||
|
||||
**Event-Driven Design**:
|
||||
- Decouples metadata fetching from file scanning
|
||||
- Enables retries without re-processing
|
||||
- Scales horizontally (multiple matchers)
|
||||
|
||||
### Medium Relevance
|
||||
|
||||
**Search Integration**:
|
||||
- Fast search improves user experience
|
||||
- Typo tolerance handles misspellings
|
||||
- Faceted search enables filtering
|
||||
|
||||
**Scrobbling**:
|
||||
- OAuth flows are reusable patterns
|
||||
- Token management is standard practice
|
||||
|
||||
**Mobile App**:
|
||||
- Code sharing between web and mobile reduces duplication
|
||||
- Monorepo structure simplifies version coordination
|
||||
|
||||
### Low Relevance
|
||||
|
||||
**Video Support**:
|
||||
- Metadata aggregator may not handle videos
|
||||
- Transcoding is out of scope
|
||||
|
||||
**Geographic Context**:
|
||||
- Areas are nice-to-have, not essential
|
||||
- ISO 3166 codes are useful for standardization
|
||||
|
||||
**Deployment Complexity**:
|
||||
- Metadata aggregator may use simpler deployment (single service)
|
||||
- Docker Compose is overkill for smaller projects
|
||||
|
||||
## Comparison with Alternatives
|
||||
|
||||
### vs Navidrome
|
||||
|
||||
**Meelo Advantages**:
|
||||
- Richer data model (Album/Release, Song/Track)
|
||||
- Multi-provider metadata (8 vs 1)
|
||||
- Music video support
|
||||
- Built-in scrobbling
|
||||
- Search performance (MeiliSearch vs SQL)
|
||||
|
||||
**Navidrome Advantages**:
|
||||
- Simpler deployment (single binary)
|
||||
- Lower resource requirements (512MB vs 4GB)
|
||||
- Faster startup (no dependencies)
|
||||
- More mature (older project)
|
||||
|
||||
**Verdict**: Meelo for metadata richness, Navidrome for simplicity.
|
||||
|
||||
### vs Jellyfin
|
||||
|
||||
**Meelo Advantages**:
|
||||
- Music-focused (not general media server)
|
||||
- Better music metadata (Album/Release, Song/Track)
|
||||
- Multi-provider enrichment
|
||||
- Faster search (MeiliSearch)
|
||||
|
||||
**Jellyfin Advantages**:
|
||||
- Handles all media types (movies, TV, music)
|
||||
- Larger community
|
||||
- More mature
|
||||
- Better transcoding (built-in)
|
||||
|
||||
**Verdict**: Meelo for music collectors, Jellyfin for general media.
|
||||
|
||||
### vs Airsonic
|
||||
|
||||
**Meelo Advantages**:
|
||||
- Modern stack (NestJS, Next.js vs Java)
|
||||
- Active development (40 releases vs stagnant)
|
||||
- Better metadata (multi-provider)
|
||||
- Search performance
|
||||
|
||||
**Airsonic Advantages**:
|
||||
- Simpler deployment (single JAR)
|
||||
- Subsonic API compatibility
|
||||
- Larger ecosystem (mobile apps)
|
||||
|
||||
**Verdict**: Meelo for modern features, Airsonic for stability.
|
||||
|
||||
### vs Funkwhale
|
||||
|
||||
**Meelo Advantages**:
|
||||
- Better metadata model
|
||||
- Multi-provider enrichment
|
||||
- Faster search
|
||||
|
||||
**Funkwhale Advantages**:
|
||||
- Federated (share music across instances)
|
||||
- Social features (follows, favorites)
|
||||
- Podcast support
|
||||
|
||||
**Verdict**: Meelo for personal use, Funkwhale for communities.
|
||||
|
||||
## Recommendations for Metadata Aggregator
|
||||
|
||||
### Adopt
|
||||
|
||||
1. **Data Model**:
|
||||
- Implement Album/Release distinction
|
||||
- Implement Song/Track distinction
|
||||
- Implement song groups for versions
|
||||
- Separate ExternalMetadata table
|
||||
|
||||
2. **Provider Pattern**:
|
||||
- Base provider interface
|
||||
- Per-provider modules
|
||||
- Factory pattern
|
||||
- Parallel queries with asyncio
|
||||
- Rate limiting per provider
|
||||
- Priority-based aggregation
|
||||
|
||||
3. **Event-Driven Architecture**:
|
||||
- Message queue for decoupling
|
||||
- Separate scanner and matcher services
|
||||
- Retry logic without re-scanning
|
||||
|
||||
### Adapt
|
||||
|
||||
1. **Search Integration**:
|
||||
- Use MeiliSearch or Typesense
|
||||
- Index on entity creation/update
|
||||
- Implement typo tolerance
|
||||
|
||||
2. **Scrobbling**:
|
||||
- OAuth flows for Last.fm
|
||||
- Token-based auth for ListenBrainz
|
||||
|
||||
3. **Code Quality**:
|
||||
- Linting (Biome, Ruff)
|
||||
- Type checking (TypeScript, Pyright)
|
||||
- Testing (Jest, pytest)
|
||||
- SonarCloud quality gates
|
||||
|
||||
### Avoid
|
||||
|
||||
1. **Complex Deployment**:
|
||||
- Prefer single service or fewer containers
|
||||
- Avoid heavy infrastructure (PostgreSQL, RabbitMQ) if possible
|
||||
- Use SQLite for smaller deployments
|
||||
|
||||
2. **Multi-Language Stack**:
|
||||
- Stick to one or two languages
|
||||
- Avoid mixing TypeScript, Go, Python unless necessary
|
||||
|
||||
3. **Kyoo Dependency**:
|
||||
- If video support needed, use built-in transcoder (FFmpeg)
|
||||
- Avoid external dependencies for core features
|
||||
|
||||
## Summary
|
||||
|
||||
Meelo excels at data modeling, multi-provider metadata enrichment, and music video support. The Album/Release and Song/Track distinctions are the most accurate representation of real-world music organization among self-hosted servers. The provider pattern with parallel queries and priority-based aggregation is directly applicable to metadata aggregators. The event-driven architecture scales well and decouples concerns. However, deployment complexity (8+ containers), multi-language stack (TypeScript, Go, Python), and heavy infrastructure (PostgreSQL, MeiliSearch, RabbitMQ) limit accessibility. The GPL-3.0 license restricts commercial use. For a metadata aggregator, adopt the data model and provider architecture, adapt the search integration and scrobbling patterns, but avoid the deployment complexity and multi-language stack. Meelo is an excellent reference for sophisticated metadata handling in a self-hosted context.
|
||||
Reference in New Issue
Block a user