metadata-agregator/docs/INGESTION_MUSICBRAINZ.md

# MusicBrainz Ingestion

Architecture documentation for ingesting music metadata from MusicBrainz.

---

## Overview

**MusicBrainz** is an open music encyclopedia maintained by the MetaBrainz Foundation. It serves as the canonical source for music metadata with community-curated data covering artists, releases, recordings, and works.

| Attribute | Value |
|-----------|-------|
| Data Quality | High (community-curated) |
| Coverage | ~2M artists, ~3M releases, ~30M recordings |
| Update Frequency | Real-time edits, weekly dumps |
| API Style | REST with Lucene search |
| Cost | Free (rate-limited) |

---

## Data Model

MusicBrainz uses a hierarchical model that separates abstract concepts from concrete manifestations.

### Entity Hierarchy

```
                        ┌──────────┐
                        │   WORK   │  ← Composition (the song as written)
                        │  (ISWC)  │     "Bohemian Rhapsody" by Freddie Mercury
                        └────┬─────┘
                             │ performed as
                             ▼
                        ┌──────────┐
                        │RECORDING │  ← Unique audio (specific performance)
                        │  (ISRC)  │     Studio version, live version, demo
                        └────┬─────┘
                             │ appears on
                             ▼
┌──────────┐           ┌──────────┐
│  ARTIST  │◄─────────►│ RELEASE  │  ← Physical/digital product
│  (MBID)  │ credited  │  (UPC)   │     US CD, UK Vinyl, Spotify release
└──────────┘    on     └────┬─────┘
                             │ variant of
                             ▼
                        ┌──────────┐
                        │ RELEASE  │  ← Abstract album concept
                        │  GROUP   │     "A Night at the Opera" (all editions)
                        └──────────┘
```

### Core Entities

| Entity | Description | Identifier | Example |
|--------|-------------|------------|---------|
| **Artist** | Musician, band, orchestra, composer | MBID | Queen, Freddie Mercury |
| **Work** | Abstract composition | ISWC | "Bohemian Rhapsody" (the song) |
| **Recording** | Specific audio performance | ISRC | Studio recording of Bohemian Rhapsody |
| **Release** | Concrete product (CD, vinyl, digital) | Barcode/UPC | 1975 UK vinyl pressing |
| **Release Group** | Abstract album (all editions) | MBID | "A Night at the Opera" |
| **Label** | Record label or imprint | MBID | EMI, Hollywood Records |

### Key Distinction: Release vs Release Group

**Release Group** = The abstract album concept
- "Nevermind" by Nirvana

**Release** = A specific physical or digital product
- 1991 US CD (DGC)
- 1991 UK CD (Geffen)
- 2011 Deluxe Edition (4 CDs)
- 2021 30th Anniversary Super Deluxe

This separation allows tracking all variants while maintaining a single "album" identity.

### Key Distinction: Recording vs Work

**Work** = The composition (what was written)
- Composer: Kurt Cobain
- ISWC identifier
- No audio - just the abstract song

**Recording** = A specific audio capture
- Performer: Nirvana
- ISRC identifier
- Has duration, audio characteristics
- Multiple recordings of same work (studio, live, acoustic)

---

## Relationship System

MusicBrainz uses **Advanced Relationships (ARs)** to connect entities with typed, attributed links.

### Relationship Types

**Artist ↔ Artist:**
- `member of band` (with dates)
- `collaboration`
- `teacher of`

**Artist ↔ Recording:**
- `performer` (with instrument)
- `producer`
- `engineer`
- `mix`

**Artist ↔ Work:**
- `composer`
- `lyricist`
- `writer`

**Recording ↔ Work:**
- `performance of`

**Artist ↔ URL:**
- `official homepage`
- `social network` (Spotify, YouTube, etc.)
- `streaming`

### Relationship Attributes

Relationships carry attributes providing detail:

```
Artist: John Lennon
  └─► Recording: "Come Together"
      Relationship: performer
      Attributes: 
        - instrument: vocals
        - instrument: rhythm guitar
```

---

## API Access Patterns

### Three Methods

| Method | Purpose | Use Case |
|--------|---------|----------|
| **Lookup** | Fetch single entity by MBID | Known entity, need full details |
| **Browse** | Paginate related entities | All albums by artist, all tracks on album |
| **Search** | Find entities by criteria | Find artist by name, recording by ISRC |

### Lookup

Direct fetch by MusicBrainz ID (MBID). Returns single entity with optional related data via `inc` parameter.

Related data options: `releases`, `recordings`, `url-rels`, `artist-rels`, `genres`, `labels`, `media`, `isrcs`

**Limitation:** Related entities capped at 25 per request. Use Browse for complete lists.

### Browse

Paginated fetch of entities related to another entity. Supports up to 100 items per request. Must iterate with offset for complete data.

### Search

Lucene-syntax queries across entity fields. Useful for:
- Finding entities by name (fuzzy matching)
- Looking up by external identifier (ISRC, barcode)
- Filtering by attributes (country, type, date)

---

## Rate Limiting

| Rule | Limit |
|------|-------|
| Requests per second | **1** (hard limit) |
| Burst allowance | None |
| Violation penalty | HTTP 503 until rate drops |
| User-Agent | **Required** (blocked without) |

User-Agent format: `AppName/Version ( contact-url-or-email )`

---

## Entity Mapping to Internal Schema

### Artist

| MusicBrainz | Internal | Notes |
|-------------|----------|-------|
| `id` | `source_id` | MBID stored as external reference |
| `name` | `name` | |
| `sort-name` | `sort_name` | |
| `type` | `artist_type` | Person, Group, Orchestra, etc. |
| `country` | `country` | ISO code |
| `life-span.begin` | `formed_date` | |
| `life-span.end` | `disbanded_date` | |
| `disambiguation` | `description` | Short disambiguator |
| URL relationship (image) | `image_url` | From Wikimedia Commons link |

### Album (from Release Group)

| MusicBrainz | Internal | Notes |
|-------------|----------|-------|
| `id` | `source_id` | Release Group MBID |
| `title` | `title` | |
| `primary-type` | `album_type` | Album, EP, Single |
| `first-release-date` | `release_date` | Earliest release |
| Label from release | `label_id` | From canonical release |

### Track (from Recording)

| MusicBrainz | Internal | Notes |
|-------------|----------|-------|
| `id` | `source_id` | Recording MBID |
| `title` | `title` | |
| `length` | `duration_ms` | In milliseconds |
| `isrcs[0]` | `isrc` | First ISRC if multiple |
| Work relationship | `work_id` | Link to composition |

### Work

| MusicBrainz | Internal | Notes |
|-------------|----------|-------|
| `id` | `source_id` | Work MBID |
| `title` | `title` | |
| `type` | `work_type` | Song, Symphony, Opera, etc. |
| `language` | `language` | ISO code |

### Label

| MusicBrainz | Internal | Notes |
|-------------|----------|-------|
| `id` | `source_id` | Label MBID |
| `name` | `name` | |
| `country` | `country` | ISO code |
| `life-span.begin` | `founded_date` | |

---

## Ingestion Flow

### Artist Discovery

```
INPUT: Artist name
         │
         ▼
┌─────────────────────────────────────┐
│  SEARCH by name                     │
│  → Ranked matches with scores       │
│  → Select highest + verify          │
└─────────────────┬───────────────────┘
                  │ MBID
                  ▼
┌─────────────────────────────────────┐
│  LOOKUP with relationships          │
│  → URLs, genres, band members       │
└─────────────────┬───────────────────┘
                  │
                  ▼
         STORE: artist + external_id + genres
```

### Discography Sync

```
INPUT: Artist MBID
         │
         ▼
┌─────────────────────────────────────┐
│  BROWSE all release-groups          │
│  → Filter: album, ep, single        │
│  → Paginate until exhausted         │
└─────────────────┬───────────────────┘
                  │ for each
                  ▼
┌─────────────────────────────────────┐
│  LOOKUP release-group               │
│  → Get releases list                │
│  → Select canonical release         │
└─────────────────┬───────────────────┘
                  │ release MBID
                  ▼
┌─────────────────────────────────────┐
│  LOOKUP release with tracks         │
│  → Media structure (discs)          │
│  → Track positions                  │
│  → ISRCs, label info                │
└─────────────────┬───────────────────┘
                  │
                  ▼
         STORE: album + tracks + positions
```

### Canonical Release Selection

When a release-group has multiple releases, select one as canonical:

| Priority | Criteria |
|----------|----------|
| 1 | Status: Official > Promotional > Bootleg |
| 2 | Format: Digital > CD > Vinyl |
| 3 | Completeness: Has barcode, has label |
| 4 | Date: Original release preferred |

---

## Cover Art

Album artwork served by **Cover Art Archive** (coverartarchive.org), not MusicBrainz directly.

| Size | URL Pattern |
|------|-------------|
| Original | `/release/{release_mbid}/front` |
| Thumbnail | `/release/{release_mbid}/front-250` |
| Medium | `/release/{release_mbid}/front-500` |
| Large | `/release/{release_mbid}/front-1200` |

Not all releases have cover art. Check availability via release metadata.

---

## Bulk Data Access

For large-scale ingestion, database dumps avoid rate limits.

| Source | Format | Frequency | Use Case |
|--------|--------|-----------|----------|
| JSON dumps | JSONL (gzipped) | 2x/week | Initial seeding |
| PostgreSQL dumps | SQL | 2x/week | Full mirror |
| Replication packets | Incremental | Hourly | Staying in sync |

### Recommended Strategy

| Phase | Method |
|-------|--------|
| Initial load | JSON dumps |
| On-demand | Live API with caching |
| Periodic refresh | JSON dumps monthly |

---

## Caching

| Entity | TTL | Rationale |
|--------|-----|-----------|
| Artist | 30 days | Rarely changes |
| Album | 30 days | Rarely changes |
| Track | 30 days | Rarely changes |
| Search results | 24 hours | New entries may appear |

---

## External ID Storage

Store in `*_external_ids` tables:

| Field | Value |
|-------|-------|
| `source` | `"musicbrainz"` |
| `source_id` | MBID (UUID) |
| `url` | `https://musicbrainz.org/{entity}/{mbid}` |

Enables:
- Cross-source deduplication
- Lookup by MBID from other services
- Link back for verification

---

## Go Client

Recommended: `go.uploadedlobster.com/musicbrainzws2`