MusicFS/docs/v2/features/metadata-enrichment-standalone.md

Metadata Enrichment (Standalone Mode): Design Doc

Authors: Sisyphus
Status: Draft
Last Updated: 2026-05-18
Reviewers: —
Approvers: —
Document Link: docs/v2/plans/metadata-enrichment-standalone.md
Prerequisites: architecture.md, week-12-external-metadata.md

---

1. Abstract

When musicfs operates without the music-agregator orchestrator, it should still be able to enrich file metadata (genres, label, artwork URL, album type) by querying the metadata-agregator service directly. This document describes a built-in metadata provider compiled into musicfs that queries metadata-agregator's gRPC SearchAlbums endpoint using artist + album names extracted from file tags. Enrichment is lazy and non-blocking — file access always returns immediately using embedded tags, while a background worker enriches metadata asynchronously.

This plan supersedes the week-12 plan's approach of embedding MusicBrainz/Discogs/Last.fm HTTP clients directly into musicfs. Instead, musicfs delegates all external metadata resolution to metadata-agregator, which already handles provider APIs, rate limiting, and caching.

2. Background

2.1. Current State

musicfs extracts audio metadata via symphonia (FLAC, MP3, AAC, OGG, Opus) and stores it in AudioMeta. This metadata is whatever the file tags contain — typically title, artist, album, year, track number.

The existing plugin system (musicfs-plugins) defines a MetadataPlugin trait for external metadata lookup, but:

• No plugins have been implemented yet.
• The plugin system only supports native .so and WASM plugins.
• A gRPC client to metadata-agregator would require bundling an async runtime and tonic inside a .so — an awkward fit.

Meanwhile, metadata-agregator is a Go gRPC service that:

• Searches MusicBrainz by artist + album name (SearchAlbums RPC).
• Caches results in PostgreSQL.
• Returns rich metadata: genres, cover URL, label, release date, album type, artist credits.

2.2. Pain Points

• musicfs files lack genres, artwork URLs, and label info unless the original files were meticulously tagged.
• The week-12 plan proposed embedding 4 separate HTTP API clients (MusicBrainz, Discogs, Last.fm, AcoustID) directly into musicfs, duplicating what metadata-agregator already does.
• The MetadataPlugin trait is designed for .so/WASM plugins, which is wrong for a core infrastructure gRPC client.

3. Goals & Non-Goals

3.1. Goals

• G1: Enrich file metadata with genres, label, album type, and cover URL by querying metadata-agregator via gRPC.
• G2: Never block file access — enrichment happens in background.
• G3: Make the provider entirely optional — disabled by default, musicfs works identically without it.
• G4: Respect enrichment source priority so orchestrator pushes (from the full-system mode) are not overwritten.

3.2. Non-Goals

• NG1: Embedding MusicBrainz/Discogs/Last.fm HTTP clients directly into musicfs (metadata-agregator handles this).
• NG2: Audio fingerprinting (AcoustID) — deferred to future work.
• NG3: Modifying the existing MetadataPlugin trait — the built-in provider is separate from the plugin system.
• NG4: Bidirectional communication — musicfs only queries metadata-agregator, never the reverse.

4. Proposed Design

4.1. High-Level Architecture

@startuml
!theme plain
skinparam componentStyle rectangle

package "musicfs" as mfs {
  component "FUSE Layer\n(readdir/open/read)" as fuse
  component "MetadataCache / DB" as db
  component "OverlayReader\n(synthesize headers)" as overlay
  component "EnrichmentQueue\n(bounded, async)" as queue
  component "EnrichmentWorker\n(background)" as worker
}

component "metadata-agregator\nSearchAlbums(query, artist)" as meta

fuse -right-> db : lookup metadata
db -right-> overlay : serve with overlay

fuse -down-> queue : enriched_at NULL?\npush request
queue -down-> worker : dequeue
worker -down-> meta : gRPC:\nSearchAlbums(\n  query=album,\n  artist=artist)
meta -up-> worker : Album (genres,\nlabel, cover_url)
worker -up-> db : write enriched\nmetadata to overlay

note bottom of meta
  metadata-agregator handles:
  • MusicBrainz API
  • rate limiting
  • PostgreSQL cache
end note

note right of fuse
  File access is never blocked.
  Returns embedded tags immediately.
  Enrichment happens async.
end note
@enduml

4.2. Enrichment Flow

@startuml
!theme plain
skinparam sequenceMessageAlign center

participant "Media Player" as mp
participant "FUSE Layer" as fuse
participant "MetadataCache\n(SQLite)" as db
participant "EnrichmentQueue" as queue
participant "EnrichmentWorker" as worker
participant "metadata-agregator" as meta

== File Access (non-blocking) ==

mp -> fuse : open("/Pink Floyd/The Wall/01 - In the Flesh.flac")
fuse -> db : lookup(virtual_path)
db --> fuse : AudioMeta(artist, album, title, ...)\nenriched_at = NULL

fuse -> queue : try_push(file_id, artist="Pink Floyd", album="The Wall")
note right of queue : non-blocking,\nbounded queue

fuse --> mp : return file handle\n(with embedded tags only)

== Background Enrichment (async) ==

queue -> worker : dequeue(file_id, artist, album)

worker -> worker : check enrichment_source\n(skip if 'orchestrator' or 'provider')

worker -> worker : dedup check:\nalready enriched same album?\n(reuse cached result)

worker -> meta : SearchAlbums(\n  query="The Wall",\n  artist="Pink Floyd",\n  limit=1)
meta --> worker : Album(\n  genres=["Progressive Rock", "Art Rock"],\n  label="Harvest",\n  cover_url="https://...",\n  album_type="album")

worker -> db : update_metadata(\n  file_id,\n  genres, label, cover_url,\n  enrichment_source='provider',\n  enriched_at=now())

worker -> worker : publish EventBus::FileModified

note over mp : next access sees\nenriched metadata
@enduml

4.3. Detailed Design

4.3.1. Configuration

Add [metadata_provider] section to config.toml:

[metadata_provider]
enabled = false                       # disabled by default
endpoint = "http://localhost:50051"    # metadata-agregator gRPC
timeout_ms = 5000                     # per-request timeout
retry_max = 3                         # max retries on failure
retry_backoff_ms = 1000               # initial backoff between retries
queue_size = 256                      # enrichment queue capacity

Config struct addition in musicfs-core/src/config.rs:

#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct MetadataProviderConfig {
    #[serde(default)]
    pub enabled: bool,
    #[serde(default = "default_provider_endpoint")]
    pub endpoint: String,
    #[serde(default = "default_provider_timeout_ms")]
    pub timeout_ms: u64,
    #[serde(default = "default_retry_max")]
    pub retry_max: u32,
    #[serde(default = "default_retry_backoff_ms")]
    pub retry_backoff_ms: u64,
    #[serde(default = "default_queue_size")]
    pub queue_size: usize,
}

4.3.2. Built-in Metadata Provider

New module in musicfs-metadata (not a plugin, compiled in):

// musicfs-metadata/src/provider.rs

pub struct MetadataAgregatorProvider {
    client: MetadataServiceClient<Channel>,
    config: MetadataProviderConfig,
}

impl MetadataAgregatorProvider {
    pub async fn connect(config: &MetadataProviderConfig)
        -> Result<Self>;

    /// Query metadata-agregator by artist + album names.
    /// Returns enriched metadata if a match is found.
    pub async fn lookup(
        &self,
        artist: &str,
        album: &str,
    ) -> Result<Option<EnrichedMetadata>>;
}

The lookup method calls SearchAlbums(query=album, artist=artist, limit=1) on metadata-agregator. If a result is returned, it maps the response to EnrichedMetadata:

pub struct EnrichedMetadata {
    pub genres: Vec<String>,
    pub label: Option<String>,
    pub album_type: Option<String>,
    pub cover_url: Option<String>,
    pub release_date: Option<String>,
    pub total_tracks: Option<u32>,
    pub total_discs: Option<u32>,
}

4.3.3. ExternalMetadata Extension

Extend the existing ExternalMetadata in musicfs-plugins/src/traits.rs to carry richer data:

pub struct ExternalMetadata {
    // existing fields...
    pub title: Option<String>,
    pub artist: Option<String>,
    pub album: Option<String>,
    pub album_artist: Option<String>,
    pub genre: Option<String>,      // kept for backward compat
    pub year: Option<u32>,
    pub track: Option<u32>,
    pub disc: Option<u32>,
    pub musicbrainz_id: Option<String>,
    pub artwork_url: Option<String>,

    // new fields
    pub genres: Vec<String>,
    pub label: Option<String>,
    pub album_type: Option<String>,
    pub cover_url: Option<String>,
}

4.3.4. Database Schema Changes

Add columns to file_metadata table in musicfs-cache/src/schema.sql:

ALTER TABLE file_metadata ADD COLUMN enrichment_source TEXT;
    -- 'embedded' | 'provider' | 'orchestrator'
ALTER TABLE file_metadata ADD COLUMN enriched_at INTEGER;
    -- unix timestamp, NULL = not enriched
ALTER TABLE file_metadata ADD COLUMN enrichment_attempts INTEGER DEFAULT 0;
    -- number of failed enrichment attempts
ALTER TABLE file_metadata ADD COLUMN last_enrichment_error TEXT;
    -- last error message, NULL if no error
ALTER TABLE file_metadata ADD COLUMN genres_json TEXT;
    -- JSON array: '["Progressive Rock","Art Rock"]'
    -- separate from existing `genre` (singular) for backward compat
ALTER TABLE file_metadata ADD COLUMN label TEXT;
ALTER TABLE file_metadata ADD COLUMN album_type TEXT;
ALTER TABLE file_metadata ADD COLUMN cover_url TEXT;

Note: The existing genre TEXT column (singular) is preserved for backward compatibility. genres_json stores the full list. The singular genre field is set to the first genre in the array when enriched.

4.3.5. Background Enrichment Queue + Worker

// musicfs-metadata/src/enrichment.rs

pub struct EnrichmentQueue {
    tx: mpsc::Sender<EnrichmentRequest>,
    /// Tracks in-flight (artist, album) pairs to prevent duplicate
    /// API calls when multiple tracks from the same album are
    /// accessed simultaneously.
    in_flight: Arc<DashSet<(String, String)>>,
}

struct EnrichmentRequest {
    file_id: FileId,
    artist: String,
    album: String,
}

pub struct EnrichmentWorker {
    rx: mpsc::Receiver<EnrichmentRequest>,
    provider: Arc<MetadataAgregatorProvider>,
    db: Arc<Database>,
    event_bus: Arc<EventBus>,
    in_flight: Arc<DashSet<(String, String)>>,
    config: MetadataProviderConfig,
}

Enqueue-time dedup

When EnrichmentQueue::try_push() is called, it checks the in_flight DashSet before pushing. If (artist, album) is already in the set, the request is dropped (the worker will enrich all files with the same album in one pass). This prevents 12 simultaneous track opens from making 12 identical API calls.

If try_push fails because the queue is full, log at WARN level and increment enrichment_queue_drops_total metric.

Worker loop (single-threaded, processes one at a time):

1. Dequeue EnrichmentRequest.
2. Check enrichment_attempts — skip if >= retry_max.
3. Atomic conflict check: write uses conditional SQL:

   UPDATE file_metadata SET
     genres_json = ?, label = ?, album_type = ?, cover_url = ?,
     genre = ?,  -- first genre for backward compat
     enrichment_source = 'provider',
     enriched_at = strftime('%s', 'now'),
     enrichment_attempts = 0,
     last_enrichment_error = NULL
   WHERE file_id = ?
     AND (enrichment_source IS NULL OR enrichment_source = 'embedded')
   

   This prevents the TOCTOU race — if the orchestrator wrote between dequeue and now, the WHERE clause prevents overwrite. The UPDATE returns rows_affected=0, which the worker treats as "skip, already enriched by higher-priority source".
4. Deduplicate by (artist, album) — if another file in the same album was already enriched, reuse the cached EnrichedMetadata result for all files with the same (artist, album) pair.
5. Call provider.lookup(artist, album).
6. On success: execute atomic update (step 3) for all files with this (artist, album). Publish EventBus::FileModified for each updated file. Remove (artist, album) from in_flight set.
7. On failure: increment enrichment_attempts, set last_enrichment_error. If attempts < retry_max, re-enqueue with exponential backoff (retry_backoff_ms * 2^attempts). If attempts >= retry_max, log at WARN and stop retrying. Remove from in_flight set.

Shutdown behavior

Queue contents are lost on shutdown. This is acceptable — files will be re-queued on next access since enriched_at is still NULL. Enrichment is idempotent.

4.3.6. FUSE Integration Point

In the FUSE readdir / getattr / open path (musicfs-fuse/src/ops.rs), after loading AudioMeta from DB:

if metadata_provider.is_enabled()
    && file_meta.enriched_at.is_none()
    && file_meta.enrichment_attempts < config.retry_max
    && file_meta.audio.artist.is_some()
    && file_meta.audio.album.is_some()
{
    if let Err(_) = enrichment_queue.try_push(EnrichmentRequest {
        file_id: file_meta.id,
        artist: file_meta.audio.artist.unwrap(),
        album: file_meta.audio.album.unwrap(),
    }) {
        // Queue full — file will be retried on next access
        tracing::warn!(
            file_id = ?file_meta.id,
            "enrichment queue full, dropping request"
        );
        metrics::ENRICHMENT_QUEUE_DROPS.inc();
    }
    // Non-blocking: returns immediately with embedded tags
}

The enrichment_attempts < retry_max check prevents files that have permanently failed enrichment (e.g., metadata-agregator has no match) from being re-queued on every access.

4.3.7. Conflict Resolution

Source	Priority	Writes When
orchestrator	Highest	Always overwrites (full-system mode push)
provider	Medium	Only if current source is NULL or 'embedded'
embedded	Lowest	Implicit default from file tag parsing

Conflict resolution is enforced atomically at write time using conditional SQL (WHERE enrichment_source IS NULL OR enrichment_source = 'embedded'), not at dequeue time. This prevents the TOCTOU race where the orchestrator writes between the worker's check and the worker's write.

4.3.8. Proto Changes Required

The existing UpdateMetadataRequest in musicfs.proto must be extended to carry the new enrichment fields:

// Add to UpdateMetadataRequest:
optional string label = 40;
optional string album_type = 41;
optional string cover_url = 42;

Note on genres: metadata-agregator returns repeated Genre (objects with id + name). The provider extracts genre names and stores them as a JSON array in genres_json. The singular genre field in UpdateMetadataRequest (already exists at field 9) is set to the first/primary genre for backward compat.

4.3.9. cover_url Usage

cover_url is stored in the metadata overlay but is not used by musicfs for artwork embedding or display in this plan. It is stored for consumption by external tools (e.g., media players that query musicfs's gRPC GetMetadata and fetch artwork themselves). Artwork download and caching is deferred to future work.

5. Cross-Cutting Concerns

5.1. Security & Privacy

• gRPC connection to metadata-agregator is plaintext (internal network). TLS can be added via config if needed.
• No PII involved — only music metadata.
• No API keys stored in musicfs — metadata-agregator handles provider auth.

5.2. Observability

New tracing spans and metrics:

Metric	Type	Description
enrichment_queue_depth	Gauge	Current queue size
enrichment_queue_drops_total	Counter	Requests dropped (queue full)
enrichment_inflight_albums	Gauge	In-flight (artist, album) dedup set size
enrichment_lookups_total	Counter	Total provider lookups
enrichment_hits_total	Counter	Successful matches
enrichment_misses_total	Counter	No match found
enrichment_errors_total	Counter	Provider errors
enrichment_skipped_total	Counter	Skipped (higher-priority source already wrote)
enrichment_latency_ms	Histogram	Lookup latency

5.3. Scalability & Performance

• Queue is bounded (default 256) — backpressure via try_push.
• Album-level deduplication: 12 tracks in same album = 1 lookup.
• No impact on file read latency — enrichment is fully async.
• metadata-agregator caches in PostgreSQL, so repeated lookups are cheap.

5.4. Testing Plan

Test	Type	Validates
test_provider_connect	Unit	gRPC connection setup
test_lookup_match	Unit (mock)	SearchAlbums → EnrichedMetadata mapping
test_lookup_no_match	Unit (mock)	Graceful handling of empty results, increments attempts
test_enrichment_queue_push	Unit	Queue push + in_flight dedup
test_enrichment_queue_full_drops	Unit	try_push fails gracefully, logs, increments metric
test_enrichment_worker_writes_db	Integration	DB write after lookup
test_enrichment_atomic_conflict	Integration	Orchestrator writes between dequeue and worker write → worker does NOT overwrite
test_enrichment_retry_backoff	Unit	Failed attempts increment counter, exponential backoff
test_enrichment_max_attempts_stop	Unit	After retry_max failures, file not re-queued
test_config_disabled	Unit	No queue/worker when disabled
test_album_dedup_simultaneous	Integration	12 tracks opened at once → 1 API call
test_genre_backward_compat	Unit	genres_json stored as array, genre set to first entry

6. Alternatives Considered

6.1. Native .so Plugin

Rejected. Requires bundling a separate async runtime + tonic gRPC stack inside a dynamically loaded library. ABI instability, duplicate runtimes, and deployment complexity outweigh the "purity" of using the plugin system.

6.2. Direct MusicBrainz/Discogs/Last.fm HTTP Clients (week-12 plan)

Rejected. metadata-agregator already handles these providers with rate limiting, caching, and deduplication. Embedding HTTP clients in musicfs would duplicate this work and couple musicfs to specific provider APIs.

6.3. WASM Plugin

Rejected. WASI networking is immature. gRPC over WASM adds unnecessary latency and complexity.

6.4. On-Demand Blocking Lookup

Rejected. Blocking file access while waiting for a gRPC response would cause latency spikes and kill media player UX. Background async is the only acceptable approach.

7. Implementation Plan

Phase 1: Foundation (Day 1)

• Add MetadataProviderConfig to config.rs
• Add DB schema columns: enrichment_source, enriched_at, enrichment_attempts, last_enrichment_error, genres_json, label, album_type, cover_url
• Add label, album_type, cover_url fields to UpdateMetadataRequest in musicfs.proto
• Extend ExternalMetadata struct
• Update config.example.toml

Phase 2: Provider + Worker (Day 1–2)

• Implement MetadataAgregatorProvider (gRPC client wrapper)
• Implement EnrichmentQueue with DashSet in-flight dedup
• Implement EnrichmentWorker with:
  • Atomic conditional write (WHERE enrichment_source IS NULL OR ...)
  • Retry tracking (enrichment_attempts, exponential backoff)
  • Album-level result caching
• Add queue drop logging + metrics
• Wire into startup (musicfs-cli) — conditional on config

Phase 3: Integration + Tests (Day 2)

• Wire enrichment trigger in FUSE getattr/readdir path (with enrichment_attempts < retry_max guard)
• Write unit tests: atomic conflict, queue drops, retry backoff, max attempts, genre backward compat
• Write integration test: 12-track simultaneous dedup
• Write integration test with in-memory DB + mock gRPC server
• Update architecture.md with metadata provider component

8. Glossary / References

Term	Definition
metadata-agregator	Go gRPC service that searches MusicBrainz and caches results in PostgreSQL
Enrichment	Adding genres, label, artwork URL to file metadata beyond what's in file tags
Overlay	musicfs mechanism for serving modified metadata without changing origin files
AudioMeta	Core metadata struct extracted from file tags by symphonia
ExternalMetadata	Metadata returned by external providers (plugin trait)
enrichment_source	Tracks who last wrote metadata: embedded, provider, or orchestrator

• metadata-agregator proto
• musicfs-plugins traits
• musicfs-cache overlay
• architecture.md