feat: initial implementation of metadata aggregator

- gRPC service with MusicBrainz provider - PostgreSQL schema with migrations - Service layer with database-first caching - Repository pattern for data access - YAML configuration support - Research documentation for 17 music metadata projects
2026-04-28 16:27:14 +02:00
commit a1f6701bac
163 changed files with 95884 additions and 0 deletions
@@ -0,0 +1,714 @@
+# minim: Architecture
+
+## Architectural Pattern
+
+minim follows a **library architecture**, not a client-server or microservices pattern. There is no daemon, no HTTP server, no background processes. The library runs entirely within the caller's Python process.
+
+**Invocation Model:**
+```python
+from minim import spotify, tidal, qobuz
+from minim.audio import Audio
+
+# Instantiate API client
+client = spotify.WebAPI(client_id="...", client_secret="...")
+
+# Make API calls
+results = client.search("Radiohead", types=["artist", "album"])
+
+# Process audio files
+audio = Audio("track.flac")
+audio.set_metadata_using_spotify(results["tracks"]["items"][0])
+audio.write_metadata()
+```
+
+All operations are synchronous, blocking calls. No event loop, no async/await in v1.
+
+## Module Organization
+
+The codebase is organized into eight top-level modules:
+
+```
+minim/
+├── __init__.py          # Package initialization, version info
+├── audio.py             # Audio file handling, metadata I/O
+├── discogs.py           # Discogs API client
+├── itunes.py            # iTunes Search API client
+├── qobuz.py             # Qobuz API client
+├── spotify.py           # Spotify Web API + private lyrics
+├── tidal.py             # TIDAL public + private API
+└── utility.py           # Shared utilities
+```
+
+**No Subpackages:** All modules are at the top level. No hierarchical organization despite 35K+ lines of code.
+
+**Module Independence:** Each API client module is self-contained. No cross-dependencies between `spotify.py`, `tidal.py`, etc. They share only `utility.py` and standard library imports.
+
+## Class Hierarchy
+
+### Audio Module
+
+```
+Audio (base class)
+├── FLAC
+├── MP3
+├── MP4
+├── OggVorbis
+└── WAVE
+```
+
+**Factory Pattern:** `Audio(filepath)` auto-detects format and returns appropriate subclass instance.
+
+**Detection Logic:**
+1. Check file extension (`.flac`, `.mp3`, `.m4a`, `.ogg`, `.wav`)
+2. If ambiguous, read magic bytes from file header
+3. Instantiate corresponding subclass
+4. Raise `ValueError` if format unsupported
+
+**Shared Interface:** All subclasses implement:
+- `read_metadata()`: Parse tags from file
+- `write_metadata()`: Write tags to file
+- `convert(output_path, format)`: Transcode via FFmpeg
+- `set_metadata_using_{service}(data)`: Map service JSON to tags
+
+### API Client Classes
+
+Each service module defines one or more API client classes:
+
+**discogs.py:**
+- `API`: Main Discogs API client (database, marketplace, collection, wantlist)
+
+**itunes.py:**
+- `SearchAPI`: iTunes Search API client
+
+**qobuz.py:**
+- `PrivateAPI`: Qobuz API client (uses undocumented endpoints)
+
+**spotify.py:**
+- `WebAPI`: Official Spotify Web API client
+- `PrivateLyricsService`: Undocumented Musixmatch integration for lyrics
+
+**tidal.py:**
+- `API`: Public TIDAL API (documented endpoints)
+- `PrivateAPI`: Private TIDAL API (undocumented endpoints for streaming URLs, lyrics, credits)
+
+**Naming Convention:** "Private" indicates use of undocumented endpoints. These are reverse-engineered from web/mobile apps and may break without notice.
+
+## Authentication Flow
+
+All API clients follow a consistent initialization and authentication pattern:
+
+### 1. Initialization (`__init__`)
+
+```python
+def __init__(self, client_id=None, client_secret=None, access_token=None, ...):
+    # Check environment variables
+    self.client_id = client_id or os.getenv("SERVICE_CLIENT_ID")
+    self.client_secret = client_secret or os.getenv("SERVICE_CLIENT_SECRET")
+    
+    # Load from config file
+    config = ConfigParser()
+    config.read(os.path.expanduser("~/minim.cfg"))
+    
+    if config.has_section("service"):
+        self.access_token = config.get("service", "access_token", fallback=None)
+        self.refresh_token = config.get("service", "refresh_token", fallback=None)
+    
+    # Use provided tokens if available
+    if access_token:
+        self.access_token = access_token
+```
+
+**Precedence:** Explicit parameters > environment variables > config file
+
+### 2. Flow Selection (`set_flow`)
+
+```python
+def set_flow(self, flow_type="authorization_code", redirect_uri="http://localhost:8888", ...):
+    self.flow_type = flow_type
+    self.redirect_uri = redirect_uri
+    self.scopes = scopes
+```
+
+**Supported Flows (Spotify example):**
+- `authorization_code`: Full user access, requires user login
+- `pkce`: Proof Key for Code Exchange, for mobile/desktop apps
+- `client_credentials`: App-only access, no user context
+- `web_player`: Extract token from browser cookie (undocumented)
+
+### 3. Token Acquisition (`set_access_token`)
+
+```python
+def set_access_token(self, method="http.server"):
+    if self.flow_type == "authorization_code":
+        # Generate authorization URL
+        auth_url = self._build_auth_url()
+        
+        # Open browser or print URL
+        webbrowser.open(auth_url)
+        
+        # Start callback server
+        if method == "http.server":
+            code = self._listen_http_server()
+        elif method == "flask":
+            code = self._listen_flask()
+        elif method == "playwright":
+            code = self._automate_browser()
+        
+        # Exchange code for token
+        token_response = self._exchange_code(code)
+        self.access_token = token_response["access_token"]
+        self.refresh_token = token_response.get("refresh_token")
+        
+        # Save to config
+        self._save_config()
+```
+
+**Callback Methods:**
+
+**http.server (default):**
+```python
+def _listen_http_server(self):
+    server = HTTPServer(("localhost", 8888), CallbackHandler)
+    server.handle_request()  # Block until callback received
+    return server.authorization_code
+```
+
+**Flask:**
+```python
+def _listen_flask(self):
+    app = Flask(__name__)
+    
+    @app.route("/callback")
+    def callback():
+        code = request.args.get("code")
+        # Store code and shutdown
+        return "Authorization successful"
+    
+    app.run(port=8888)
+```
+
+**Playwright:**
+```python
+def _automate_browser(self):
+    from playwright.sync_api import sync_playwright
+    
+    with sync_playwright() as p:
+        browser = p.chromium.launch(headless=True)
+        page = browser.new_page()
+        
+        # Navigate to auth URL
+        page.goto(auth_url)
+        
+        # Fill login form (service-specific selectors)
+        page.fill("#username", self.email)
+        page.fill("#password", self.password)
+        page.click("button[type=submit]")
+        
+        # Wait for redirect
+        page.wait_for_url(f"{self.redirect_uri}*")
+        
+        # Extract code from URL
+        code = parse_qs(urlparse(page.url).query)["code"][0]
+        browser.close()
+        return code
+```
+
+### 4. Token Persistence
+
+```python
+def _save_config(self):
+    config = ConfigParser()
+    config.read(os.path.expanduser("~/minim.cfg"))
+    
+    if not config.has_section("service"):
+        config.add_section("service")
+    
+    config.set("service", "access_token", self.access_token)
+    if self.refresh_token:
+        config.set("service", "refresh_token", self.refresh_token)
+    
+    with open(os.path.expanduser("~/minim.cfg"), "w") as f:
+        config.write(f)
+```
+
+**File Format (INI):**
+```ini
+[spotify]
+client_id = abc123
+client_secret = def456
+access_token = BQC...
+refresh_token = AQD...
+expires_at = 1672531200
+
+[tidal]
+client_id = xyz789
+access_token = eyJ...
+refresh_token = eyJ...
+```
+
+**Security:** Plain text storage. File permissions default to user-readable (0644 on Unix). No encryption, no OS keychain integration.
+
+### 5. Token Refresh
+
+```python
+def _request(self, method, url, **kwargs):
+    # Check if token expired
+    if self.expires_at and time.time() >= self.expires_at:
+        self._refresh_access_token()
+    
+    # Make request with current token
+    response = requests.request(
+        method, url,
+        headers=self._get_headers(),
+        **kwargs
+    )
+    
+    # Handle 401 Unauthorized (token invalid)
+    if response.status_code == 401:
+        self._refresh_access_token()
+        # Retry request
+        response = requests.request(method, url, headers=self._get_headers(), **kwargs)
+    
+    return response
+
+def _refresh_access_token(self):
+    response = requests.post(
+        self.token_url,
+        data={
+            "grant_type": "refresh_token",
+            "refresh_token": self.refresh_token,
+            "client_id": self.client_id,
+            "client_secret": self.client_secret
+        }
+    )
+    
+    token_data = response.json()
+    self.access_token = token_data["access_token"]
+    self.expires_at = time.time() + token_data["expires_in"]
+    
+    # Update refresh token if provided
+    if "refresh_token" in token_data:
+        self.refresh_token = token_data["refresh_token"]
+    
+    self._save_config()
+```
+
+**Automatic Refresh:** Transparent to caller. If a request fails with 401, the client refreshes the token and retries automatically.
+
+## Request Handling
+
+All API clients implement a common `_request()` method:
+
+```python
+def _request(self, method: str, url: str, **kwargs) -> dict:
+    """
+    Make HTTP request with authentication.
+    
+    Args:
+        method: HTTP method (GET, POST, PUT, DELETE)
+        url: Full URL or path (prepended with base_url if relative)
+        **kwargs: Passed to requests.request()
+    
+    Returns:
+        JSON response as dict
+    
+    Raises:
+        RuntimeError: If response status is not 2xx
+    """
+    # Prepend base URL if path is relative
+    if not url.startswith("http"):
+        url = self.base_url + url
+    
+    # Add authentication headers
+    headers = kwargs.pop("headers", {})
+    headers.update(self._get_headers())
+    
+    # Make request
+    response = requests.request(method, url, headers=headers, **kwargs)
+    
+    # Check status
+    if not response.ok:
+        raise RuntimeError(
+            f"{method} {url} failed: {response.status_code} {response.text}"
+        )
+    
+    # Parse JSON
+    return response.json()
+```
+
+**Header Injection:** Each service implements `_get_headers()`:
+
+**Spotify (Bearer token):**
+```python
+def _get_headers(self):
+    return {"Authorization": f"Bearer {self.access_token}"}
+```
+
+**Discogs (OAuth 1.0a signature):**
+```python
+def _get_headers(self):
+    oauth = OAuth1(
+        self.consumer_key,
+        client_secret=self.consumer_secret,
+        resource_owner_key=self.access_token,
+        resource_owner_secret=self.access_token_secret
+    )
+    return oauth  # requests-oauthlib handles header generation
+```
+
+**Qobuz (X-App-Id header + Bearer token):**
+```python
+def _get_headers(self):
+    return {
+        "X-App-Id": self.app_id,
+        "Authorization": f"Bearer {self.access_token}"
+    }
+```
+
+**Error Handling:** All HTTP errors raise `RuntimeError` with status code and response body. No typed exceptions, no retry logic, no exponential backoff.
+
+**Rate Limiting:** Not implemented. Caller responsible for respecting service rate limits.
+
+## Metadata Mapping Architecture
+
+The `Audio` class provides service-specific metadata setters that normalize API responses to a common schema:
+
+```python
+class Audio:
+    def set_metadata_using_spotify(self, track_data: dict):
+        """Map Spotify track object to audio metadata."""
+        self.title = track_data["name"]
+        self.artist = ", ".join(a["name"] for a in track_data["artists"])
+        self.album = track_data["album"]["name"]
+        self.date = track_data["album"]["release_date"]
+        self.track_number = track_data["track_number"]
+        self.disc_number = track_data["disc_number"]
+        self.isrc = track_data.get("external_ids", {}).get("isrc")
+        
+        # Fetch artwork
+        if track_data["album"]["images"]:
+            artwork_url = track_data["album"]["images"][0]["url"]
+            self.artwork = requests.get(artwork_url).content
+    
+    def set_metadata_using_tidal(self, track_data: dict):
+        """Map TIDAL track object to audio metadata."""
+        self.title = track_data["title"]
+        self.artist = ", ".join(a["name"] for a in track_data["artists"])
+        self.album = track_data["album"]["title"]
+        self.date = track_data["streamStartDate"][:10]  # ISO date to YYYY-MM-DD
+        self.track_number = track_data["trackNumber"]
+        self.disc_number = track_data["volumeNumber"]
+        self.isrc = track_data.get("isrc")
+        
+        # Fetch artwork (construct URL from cover ID)
+        if track_data["album"]["cover"]:
+            cover_id = track_data["album"]["cover"].replace("-", "/")
+            artwork_url = f"https://resources.tidal.com/images/{cover_id}/1280x1280.jpg"
+            self.artwork = requests.get(artwork_url).content
+```
+
+**Normalization Challenges:**
+
+1. **Artist Representation:**
+   - Spotify: Array of objects `[{"name": "Artist"}]`
+   - TIDAL: Array of objects `[{"name": "Artist"}]`
+   - iTunes: String `"Artist"`
+   - Qobuz: Object `{"name": "Artist"}` (single artist)
+
+2. **Date Formats:**
+   - Spotify: ISO 8601 `"2023-01-15"` or year-only `"2023"`
+   - TIDAL: ISO 8601 with time `"2023-01-15T00:00:00.000Z"`
+   - iTunes: ISO 8601 `"2023-01-15T00:00:00Z"`
+   - Qobuz: Unix timestamp or ISO 8601
+
+3. **Artwork URLs:**
+   - Spotify: Array of images with different sizes `[{"url": "...", "width": 640, "height": 640}]`
+   - TIDAL: Cover ID requiring URL construction
+   - iTunes: Direct URL `"artworkUrl100"`, `"artworkUrl600"`
+   - Qobuz: Direct URL with size parameter
+
+4. **Track/Disc Numbers:**
+   - Spotify: Separate `track_number` and `disc_number` fields
+   - TIDAL: `trackNumber` and `volumeNumber`
+   - iTunes: Combined `"trackNumber": "3/12"` (track 3 of 12)
+   - Qobuz: Separate `track_number` and `media_number`
+
+**Mapping Strategy:** Each `set_metadata_using_*()` method handles service-specific quirks and normalizes to the `Audio` class's internal representation.
+
+## Audio File I/O Architecture
+
+The `Audio` class uses `mutagen` for reading and writing metadata:
+
+```python
+class Audio:
+    def __init__(self, filepath: str):
+        self.filepath = filepath
+        self._file = mutagen.File(filepath)
+        
+        if isinstance(self._file, mutagen.flac.FLAC):
+            self.__class__ = FLAC
+        elif isinstance(self._file, mutagen.mp3.MP3):
+            self.__class__ = MP3
+        elif isinstance(self._file, mutagen.mp4.MP4):
+            self.__class__ = MP4
+        # ... etc
+    
+    def write_metadata(self):
+        """Write metadata to file. Implemented by subclasses."""
+        raise NotImplementedError
+
+class FLAC(Audio):
+    def write_metadata(self):
+        """Write Vorbis Comments to FLAC file."""
+        self._file["TITLE"] = self.title
+        self._file["ARTIST"] = self.artist
+        self._file["ALBUM"] = self.album
+        self._file["DATE"] = self.date
+        self._file["TRACKNUMBER"] = str(self.track_number)
+        self._file["DISCNUMBER"] = str(self.disc_number)
+        
+        if self.artwork:
+            picture = mutagen.flac.Picture()
+            picture.data = self.artwork
+            picture.type = 3  # Front cover
+            picture.mime = "image/jpeg"
+            self._file.add_picture(picture)
+        
+        self._file.save()
+
+class MP3(Audio):
+    def write_metadata(self):
+        """Write ID3v2 tags to MP3 file."""
+        from mutagen.id3 import TIT2, TPE1, TALB, TDRC, TRCK, TPOS, APIC
+        
+        self._file["TIT2"] = TIT2(encoding=3, text=self.title)
+        self._file["TPE1"] = TPE1(encoding=3, text=self.artist)
+        self._file["TALB"] = TALB(encoding=3, text=self.album)
+        self._file["TDRC"] = TDRC(encoding=3, text=self.date)
+        self._file["TRCK"] = TRCK(encoding=3, text=str(self.track_number))
+        self._file["TPOS"] = TPOS(encoding=3, text=str(self.disc_number))
+        
+        if self.artwork:
+            self._file["APIC"] = APIC(
+                encoding=3,
+                mime="image/jpeg",
+                type=3,  # Front cover
+                desc="Cover",
+                data=self.artwork
+            )
+        
+        self._file.save()
+```
+
+**Tag Format Mapping:**
+
+| Field | FLAC (Vorbis) | MP3 (ID3v2) | MP4 (Atoms) |
+|-------|---------------|-------------|-------------|
+| Title | `TITLE` | `TIT2` | `\xa9nam` |
+| Artist | `ARTIST` | `TPE1` | `\xa9ART` |
+| Album | `ALBUM` | `TALB` | `\xa9alb` |
+| Date | `DATE` | `TDRC` | `\xa9day` |
+| Track # | `TRACKNUMBER` | `TRCK` | `trkn` |
+| Disc # | `DISCNUMBER` | `TPOS` | `disk` |
+| Artwork | `METADATA_BLOCK_PICTURE` | `APIC` | `covr` |
+
+**Format Conversion:**
+
+```python
+def convert(self, output_path: str, format: str, **ffmpeg_options):
+    """Convert audio file to different format using FFmpeg."""
+    import subprocess
+    
+    cmd = [
+        "ffmpeg",
+        "-i", self.filepath,
+        "-c:a", self._get_codec(format),
+        **self._build_ffmpeg_args(ffmpeg_options),
+        output_path
+    ]
+    
+    subprocess.run(cmd, check=True)
+    
+    # Copy metadata to converted file
+    converted = Audio(output_path)
+    converted.title = self.title
+    converted.artist = self.artist
+    # ... copy all fields
+    converted.write_metadata()
+
+def _get_codec(self, format: str) -> str:
+    """Map format to FFmpeg codec."""
+    codecs = {
+        "flac": "flac",
+        "mp3": "libmp3lame",
+        "m4a": "aac",
+        "ogg": "libvorbis",
+        "wav": "pcm_s16le"
+    }
+    return codecs.get(format, format)
+```
+
+## Configuration Architecture
+
+**File Location:** `~/minim.cfg` (expands to user's home directory)
+
+**Format:** INI-style via Python's `ConfigParser`
+
+**Structure:**
+```ini
+[discogs]
+consumer_key = ...
+consumer_secret = ...
+access_token = ...
+access_token_secret = ...
+
+[qobuz]
+app_id = ...
+app_secret = ...
+email = user@example.com
+password = ...
+access_token = ...
+
+[spotify]
+client_id = ...
+client_secret = ...
+access_token = ...
+refresh_token = ...
+expires_at = 1672531200
+
+[tidal]
+client_id = ...
+client_secret = ...
+access_token = ...
+refresh_token = ...
+user_id = 12345
+country_code = US
+```
+
+**Reading:**
+```python
+config = ConfigParser()
+config.read(os.path.expanduser("~/minim.cfg"))
+
+if config.has_section("spotify"):
+    access_token = config.get("spotify", "access_token", fallback=None)
+    refresh_token = config.get("spotify", "refresh_token", fallback=None)
+```
+
+**Writing:**
+```python
+config = ConfigParser()
+config.read(os.path.expanduser("~/minim.cfg"))
+
+if not config.has_section("spotify"):
+    config.add_section("spotify")
+
+config.set("spotify", "access_token", new_token)
+
+with open(os.path.expanduser("~/minim.cfg"), "w") as f:
+    config.write(f)
+```
+
+**Thread Safety:** Not thread-safe. Concurrent writes from multiple processes can corrupt the file. No file locking implemented.
+
+## Error Handling Architecture
+
+**Strategy:** Fail-fast with `RuntimeError`
+
+**API Errors:**
+```python
+def _request(self, method, url, **kwargs):
+    response = requests.request(method, url, **kwargs)
+    
+    if not response.ok:
+        raise RuntimeError(
+            f"{method} {url} failed with status {response.status_code}: {response.text}"
+        )
+    
+    return response.json()
+```
+
+**File Errors:**
+```python
+def __init__(self, filepath):
+    if not os.path.exists(filepath):
+        raise FileNotFoundError(f"Audio file not found: {filepath}")
+    
+    self._file = mutagen.File(filepath)
+    
+    if self._file is None:
+        raise ValueError(f"Unsupported audio format: {filepath}")
+```
+
+**No Typed Exceptions:** All errors are generic `RuntimeError`, `ValueError`, `FileNotFoundError`. No custom exception hierarchy.
+
+**No Retry Logic:** Failed requests are not retried. Caller must implement retry logic if needed.
+
+**No Logging:** Errors are raised, not logged. No warning messages for non-critical issues.
+
+## Dependency Injection
+
+minim does not use formal dependency injection. Configuration is passed via:
+
+1. **Constructor parameters:** `WebAPI(client_id="...", client_secret="...")`
+2. **Environment variables:** `os.getenv("SPOTIFY_CLIENT_ID")`
+3. **Config file:** `ConfigParser().read("~/minim.cfg")`
+
+**No DI Framework:** No use of `injector`, `dependency-injector`, or similar libraries.
+
+**Testing Implications:** Difficult to mock API clients. Tests use real API calls with credentials from environment variables or config file.
+
+## Concurrency Model
+
+**Synchronous Only:** All operations are blocking, synchronous calls.
+
+**No Async Support:** No `async`/`await`, no `asyncio`, no `aiohttp`.
+
+**Threading:** Not thread-safe. Shared state (config file, token refresh) can cause race conditions.
+
+**Multiprocessing:** Safe for read-only operations. Token refresh in multiple processes can corrupt config file.
+
+## Extensibility
+
+**Adding New Services:**
+
+1. Create new module (e.g., `apple_music.py`)
+2. Define API client class with `__init__`, `set_flow`, `set_access_token`, `_request`, `_get_headers`
+3. Implement service-specific methods (`search`, `get_track`, etc.)
+4. Add `set_metadata_using_apple_music()` to `Audio` class
+
+**No Plugin System:** No formal extension mechanism. New services require modifying the library code.
+
+**Subclassing:** API client classes can be subclassed to override behavior:
+
+```python
+class CustomSpotifyAPI(spotify.WebAPI):
+    def _request(self, method, url, **kwargs):
+        # Add custom logging
+        print(f"Making request: {method} {url}")
+        return super()._request(method, url, **kwargs)
+```
+
+## Deployment Architecture
+
+**Not Applicable:** minim is a library, not a deployable service. No server, no containers, no orchestration.
+
+**Distribution:** Install via pip from source repository.
+
+**Runtime:** Runs in caller's Python process. No separate runtime environment.
+
+## Summary
+
+minim's architecture is straightforward and pragmatic:
+
+- **Library pattern** with no server components
+- **Synchronous, blocking** operations throughout
+- **Consistent authentication flow** across all services
+- **Automatic token management** with file-based persistence
+- **Service-specific metadata mapping** to common schema
+- **Format-agnostic audio I/O** via mutagen
+- **Fail-fast error handling** with generic exceptions
+
+The architecture prioritizes simplicity and ease of use over scalability and robustness. It's well-suited for personal projects, scripts, and research but lacks features needed for production services (async, rate limiting, typed exceptions, secure storage).
+
+The v2 rewrite on the `dev` branch addresses many architectural limitations while preserving the core design philosophy.