Files

T

Alexander a1f6701bac feat: initial implementation of metadata aggregator

- gRPC service with MusicBrainz provider
- PostgreSQL schema with migrations
- Service layer with database-first caching
- Repository pattern for data access
- YAML configuration support
- Research documentation for 17 music metadata projects

2026-04-28 16:28:53 +02:00

11 KiB

Raw Blame History

AcoustID System Overview

Introduction

AcoustID is an open-source audio fingerprinting service that identifies music recordings by analyzing their acoustic characteristics. The system consists of two primary components working in tandem: a Python-based web service (acoustid-server) and a high-performance Zig-based fingerprint index (acoustid-index). Together, they provide a production-grade solution for matching audio fingerprints to MusicBrainz metadata.

System Components

acoustid-server (Python)

The server component handles all user-facing operations, database management, and business logic.

Repository: acoustid/acoustid-server
License: MIT
Language: Python 3.12+
Current Version: 26.3.1

Core Technologies:

Web Framework: Werkzeug/Flask (current) with migration to Starlette (future async)
ORM: SQLAlchemy 2.x with multi-database support
Database: PostgreSQL 17.4 (4 separate databases)
Cache/Queue: Redis for rate limiting and task queues
Message Queue: NATS with JetStream for async submission processing
ASGI Server: Uvicorn for async endpoints, Gunicorn for legacy

Key Dependencies:

acoustid-ext (C extension for Chromaprint)
Flask (current web framework)
Starlette (future async framework)
aiohttp (async HTTP client)
SQLAlchemy 2.x (ORM)
alembic (database migrations)
asyncpg (async PostgreSQL driver)
psycopg2 (sync PostgreSQL driver)
nats-py (NATS client)
mbdata (MusicBrainz data models)
msgspec (fast JSON/MessagePack)
zstd (compression)
gunicorn (WSGI server)
uvicorn (ASGI server)

Entry Point:

# Main CLI entry
python manage.py -> acoustid.cli:main()

# Available commands
python manage.py run web      # Web UI server
python manage.py run api      # API server
python manage.py run cron     # Scheduled tasks
python manage.py run worker   # Background worker
python manage.py run import   # Import fingerprints

File Locations:

Entry script: manage.py
CLI implementation: acoustid/cli.py
Server logic: acoustid/server.py
Worker logic: acoustid/worker.py
Cron jobs: acoustid/cron.py
Configuration: acoustid/config.py

acoustid-index (Zig)

The index component provides ultra-fast fingerprint search using advanced data structures and SIMD optimizations.

Repository: acoustid/acoustid-index
License: GPL-3.0
Language: Zig
Build System: Zig build system

Core Technologies:

HTTP Server: httpz (Zig HTTP library)
Data Structure: LSM-tree (Log-Structured Merge-tree) inverted index
Compression: StreamVByte SIMD compression for posting lists
Serialization: MessagePack for wire protocol
Metrics: Prometheus-compatible metrics endpoint

Key Dependencies:

httpz (HTTP server framework)
metrics (Prometheus metrics)
zul (Zig utility library)
msgpack (MessagePack serialization)
nats (NATS client)

Entry Point:

# Build and run
zig build run -- --dir /tmp --port 8080

# Binary name
fpindex

# CLI flags
--dir <path>          # Data directory for index storage
--port <number>       # HTTP server port (default: 6081)
--threads <number>    # Worker thread count
--log-level <level>   # Logging verbosity
--cluster <name>      # Cluster name for distributed setup
--nats-url <url>      # NATS server URL for clustering

File Locations:

Main entry: src/main.zig
HTTP server: src/server.zig
API handlers: src/api.zig
Multi-index manager: src/MultiIndex.zig
Core index: src/Index.zig
Index reader: src/IndexReader.zig
Segment management: src/segment.zig
Memory segment: src/MemorySegment.zig
File segment: src/FileSegment.zig
Write-ahead log: src/Oplog.zig
File format: src/filefmt.zig
Block compression: src/block.zig
SIMD compression: src/streamvbyte.zig
Metrics: src/metrics.zig

Build and Run

Server Build

# Install dependencies with uv
uv sync

# Build Chromaprint extension
# (handled automatically in Docker build)

# Run with docker-compose
docker compose up

Docker Compose Services:

nats: Message queue
redis: Cache and rate limiting
postgres: Database (custom pg17.4 image)
index: Fingerprint index service
api: API server
web: Web UI server
cron: Scheduled tasks
worker: Background job processor

Index Build

# Build binary
zig build

# Run with options
zig build run -- --dir /var/lib/acoustid-index --port 6081 --threads 4

Architecture Relationship

The two components work together in a client-server model:

Server receives fingerprint submissions and lookup requests via HTTP API
Server stores metadata in PostgreSQL
Server sends fingerprint data to Index via HTTP/MessagePack protocol
Index performs ultra-fast similarity search using LSM-tree
Index returns candidate fingerprint IDs to Server
Server enriches results with metadata from PostgreSQL and MusicBrainz
Server returns final results to client

Communication Protocols

Server to Index

Modern Protocol (fpstore.py):

HTTP POST to http://index:6081/:index/_search
Request body: MessagePack-encoded fingerprint query
Response: MessagePack-encoded list of candidate IDs with scores

Legacy Protocol (indexclient.py):

Raw TCP socket connection
Binary protocol with custom framing
Being phased out in favor of HTTP

Client to Server

Public API:

HTTP GET/POST to https://api.acoustid.org/v2/*
JSON/XML/JSONP responses
Rate-limited by API key and IP

Version Information

Server Version: 26.3.1

Semantic versioning
Tagged releases in Git
Version defined in acoustid/__init__.py

Index Version: No formal versioning yet

Tracked by Git commit hash
Breaking changes communicated via commit messages

Deployment Models

Production (acoustid.org)

Multi-server deployment
Separate API, web, worker, and cron processes
Dedicated PostgreSQL cluster (4 databases)
Redis cluster for caching
NATS cluster for message queue
Multiple index instances for load balancing

Self-Hosted (Docker Compose)

Single-host deployment
All services in containers
Shared PostgreSQL instance
Single Redis instance
Single NATS instance
Single index instance

Development (Local)

Python virtual environment with uv
Local PostgreSQL (or Docker)
Local Redis (or Docker)
Local NATS (or Docker)
Index built and run locally with Zig

Key Features

Server Features

Fingerprint Submission: Accept audio fingerprints with optional metadata
Fingerprint Lookup: Match fingerprints to known recordings
MusicBrainz Integration: Link fingerprints to MBIDs
User Management: API key generation and management
Rate Limiting: Multi-tier rate limiting (global, app, IP)
Batch Operations: Submit/lookup up to 20 fingerprints per request
Async Processing: Background workers for heavy operations
Health Checks: Multiple health endpoints for monitoring
Metrics: StatsD metrics for observability

Index Features

Fast Search: Sub-millisecond fingerprint matching
SIMD Optimization: StreamVByte compression for posting lists
LSM-Tree Storage: Efficient write and read performance
Background Merging: Automatic segment compaction
Snapshot Support: Point-in-time index snapshots
Cluster Support: Distributed index via NATS
Prometheus Metrics: Built-in metrics endpoint
HTTP API: RESTful API for all operations

Configuration

Server Configuration

Config File: acoustid.conf (INI format) Environment Variables: ACOUSTID_* prefix Secret Files: *_file suffix for file-based secrets

Example:

[database]
name = acoustid_app
user = acoustid
password_file = /run/secrets/db_password

[redis]
host = redis
port = 6379

[fingerprint_index]
host = index
port = 6081

Index Configuration

CLI Flags Only: No config file support Environment Variables: Limited support