Reorganize docs into v1 (beetfs) and v2 (new architecture)
docs/v1/ - Original beetfs documentation:
- analysis.md, components.md, data-flow.md, drawbacks.md
- features.md, modernization.md, rust-migration.md
- benchmark-plan.md, benchmark-results.md, e2e-test-plan.md
- README.md
docs/v2/ - New MusicFS architecture:
- requirements.md: Full requirements spec (FR-1 to FR-25, NFR-1 to NFR-14)
- P0: Multi-origin, plugins, CAS, control API
- P1: Search, album art, prefetch, metadata sources
- P3: HA, 10M+ files scalability
- architecture.md: Google BlueDoc style design document
- PlantUML diagrams for all components
- Design requirements with quantitative targets
- Alternatives considered, implementation plan
This commit is contained in:
Alexander
@@ -0,0 +1,899 @@
|
||||
# MusicFS: Design Doc
|
||||
|
||||
**Authors:** [TBD]
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-12
|
||||
**Reviewers:** [TBD]
|
||||
**Approvers:** [TBD]
|
||||
**Requirements:** [requirements.md](requirements.md)
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
MusicFS is a read-only FUSE filesystem that presents music libraries organized
|
||||
by metadata (artist/album/track) rather than physical file paths. It supports
|
||||
multiple origin storage backends (local, NFS, S3, SFTP), provides intelligent
|
||||
caching with delta synchronization, and exposes a plugin architecture for
|
||||
extensibility.
|
||||
|
||||
The system addresses limitations of the existing beetfs implementation:
|
||||
- O(N) mount time → O(1) lazy loading
|
||||
- Full file in RAM → streaming with content-addressable chunks
|
||||
- Single origin → federated multi-origin with failover
|
||||
- No offline support → cache-first with graceful degradation
|
||||
|
||||
Target users are media enthusiasts with large music collections (100K-10M+
|
||||
tracks) distributed across multiple storage systems who want a unified,
|
||||
metadata-organized view without modifying original files.
|
||||
|
||||
---
|
||||
|
||||
## 2. Background
|
||||
|
||||
### 2.1 Current State
|
||||
|
||||
The existing beetfs implementation is a Python 2.7 FUSE plugin for beets that:
|
||||
- Presents a virtual filesystem organized by metadata templates
|
||||
- Overlays metadata from beets database onto file headers
|
||||
- Supports metadata writes back to the beets database
|
||||
|
||||
### 2.2 Pain Points
|
||||
|
||||
| Problem | Impact |
|
||||
|---------|--------|
|
||||
| O(N) mount time (5-120s for large libraries) | Unusable for large collections |
|
||||
| Loads entire file into RAM on open | OOM risk, 50-100MB per file |
|
||||
| Python GIL limits concurrency | Poor performance under load |
|
||||
| No caching between sessions | Repeated work on every mount |
|
||||
| Single local origin only | Can't federate across storage |
|
||||
| No offline support | Unusable without origin access |
|
||||
| Critical bugs (nested methods, tree building) | Non-functional |
|
||||
|
||||
### 2.3 Related Systems
|
||||
|
||||
| System | Relationship |
|
||||
|--------|--------------|
|
||||
| [beets](https://beets.io/) | Source of inspiration; potential import source |
|
||||
| [rclone mount](https://rclone.org/commands/rclone_mount/) | Similar FUSE + remote storage; no metadata organization |
|
||||
| [Plex/Jellyfin](https://jellyfin.org/) | Media servers with metadata; not filesystem-based |
|
||||
|
||||
---
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1 Goals
|
||||
|
||||
| ID | Goal | Success Metric |
|
||||
|----|------|----------------|
|
||||
| G1 | O(1) mount time | <500ms regardless of library size |
|
||||
| G2 | Minimal memory footprint | <50MB idle, <500MB peak |
|
||||
| G3 | Support multiple origins | ≥2 origins with automatic failover |
|
||||
| G4 | Offline-first operation | Serve cached data when origin unavailable |
|
||||
| G5 | Delta synchronization | >90% bandwidth reduction vs full sync |
|
||||
| G6 | Plugin extensibility | Support custom origins, formats, metadata sources |
|
||||
| G7 | Full-text search | Sub-second search across 1M+ tracks |
|
||||
|
||||
### 3.2 Design Requirements
|
||||
|
||||
The following quantitative requirements drive architectural decisions. Full
|
||||
specification in [requirements.md](requirements.md).
|
||||
|
||||
#### 3.2.1 Latency Requirements
|
||||
|
||||
| Operation | Target | Maximum | Requirement |
|
||||
|-----------|--------|---------|-------------|
|
||||
| `stat()` cached | <1ms | 5ms | NFR-1.1 |
|
||||
| `readdir()` cached | <10ms | 50ms | NFR-1.2 |
|
||||
| `open()` cached | <5ms | 20ms | NFR-1.3 |
|
||||
| `read()` cached | <1ms | 5ms | NFR-1.4 |
|
||||
| `read()` cache miss (local) | <50ms | 200ms | NFR-1.5 |
|
||||
| `read()` cache miss (remote) | <200ms | 1000ms | NFR-1.6 |
|
||||
| Mount completion | <100ms | 500ms | NFR-1.7 |
|
||||
| Search query (1M files) | <500ms | 1000ms | FR-14 |
|
||||
|
||||
**Design Response:**
|
||||
- Lazy loading eliminates mount-time I/O → O(1) mount
|
||||
- In-memory LRU cache for hot metadata → <1ms stat
|
||||
- SQLite with indexes → O(log n) lookups
|
||||
- Async I/O via tokio → non-blocking operations
|
||||
|
||||
#### 3.2.2 Throughput Requirements
|
||||
|
||||
| Metric | Target | Requirement |
|
||||
|--------|--------|-------------|
|
||||
| Sequential read (cached) | >500 MB/s | NFR-2.1 |
|
||||
| Sequential read (local origin) | >200 MB/s | NFR-2.2 |
|
||||
| Metadata ops/sec | >1000 | NFR-2.3 |
|
||||
| Concurrent file handles | >1000 | NFR-2.4 |
|
||||
|
||||
**Design Response:**
|
||||
- Memory-mapped chunk files → kernel-optimized reads
|
||||
- No GIL (Rust) → true parallelism
|
||||
- Async FUSE ops → handle many concurrent requests
|
||||
|
||||
#### 3.2.3 Scalability Requirements
|
||||
|
||||
| Metric | Target | Stretch | Requirement |
|
||||
|--------|--------|---------|-------------|
|
||||
| Library size | 1M files | 10M files | NFR-3.1, NFR-3.5 |
|
||||
| Directory entries | 100K | 1M | NFR-3.2 |
|
||||
| Concurrent clients | 10 | 100+ | NFR-3.6 |
|
||||
| Mount time scaling | O(1) | O(1) | NFR-3.3 |
|
||||
|
||||
**Design Response:**
|
||||
- Lazy tree loading → mount time independent of size
|
||||
- SQLite indexes → O(log n) regardless of scale
|
||||
- Streaming readdir → handle large directories
|
||||
- Connection pooling → support many clients
|
||||
|
||||
#### 3.2.4 Resource Requirements
|
||||
|
||||
| Resource | Idle | Active (1K files) | Peak | Requirement |
|
||||
|----------|------|-------------------|------|-------------|
|
||||
| Memory | <50 MB | <200 MB | <500 MB | NFR-4.1-4.3 |
|
||||
| Per-file overhead | - | <1 KB | - | NFR-4.4 |
|
||||
| Metadata cache | - | 100 MB default | configurable | NFR-5.1 |
|
||||
| Content cache | - | 10 GB default | configurable | NFR-5.2 |
|
||||
|
||||
**Design Response:**
|
||||
- Streaming reads → never load full file in memory
|
||||
- Content-addressed chunks → bounded cache with LRU eviction
|
||||
- Metadata in SQLite → minimal per-file RAM overhead
|
||||
|
||||
#### 3.2.5 Efficiency Requirements
|
||||
|
||||
| Metric | Target | Requirement |
|
||||
|--------|--------|-------------|
|
||||
| Delta sync bandwidth reduction | >90% | NFR-6.4 |
|
||||
| Cache hit rate (warm) | >95% | Derived |
|
||||
| Deduplication ratio | >10% typical | FR-20 |
|
||||
|
||||
**Design Response:**
|
||||
- CDC chunking → stable boundaries, minimal re-transfer
|
||||
- Content-addressable storage → automatic deduplication
|
||||
- Prefetch engine → anticipate access patterns
|
||||
|
||||
#### 3.2.6 Reliability Requirements
|
||||
|
||||
| Scenario | Behavior | Requirement |
|
||||
|----------|----------|-------------|
|
||||
| Origin offline | Serve cached data | NFR-7.1 |
|
||||
| Network failure | Graceful degradation, no crash | NFR-7.2 |
|
||||
| Failed operation | Retry with backoff (100ms, 500ms, 2s) | NFR-7.3 |
|
||||
| Malformed audio | Skip file, log error, don't crash | NFR-7.4 |
|
||||
| Chunk corruption | Detect via checksum, re-fetch | NFR-8.1, NFR-8.4 |
|
||||
| Interrupted sync | Resume from last good state | NFR-8.3 |
|
||||
| Unclean unmount | Recover on next mount | NFR-8.2 |
|
||||
|
||||
**Design Response:**
|
||||
- Cache-first architecture → offline operation by default
|
||||
- Origin federation with health checks → survive single origin failure
|
||||
- xxHash checksums on all chunks → detect corruption
|
||||
- WAL mode SQLite → ACID transactions, crash recovery
|
||||
|
||||
#### 3.2.7 Concurrent Access Requirements
|
||||
|
||||
| Scenario | Limit | Latency Impact | Requirement |
|
||||
|----------|-------|----------------|-------------|
|
||||
| Simultaneous open files | >1000 handles | None | NFR-2.4 |
|
||||
| Parallel read ops | >100 concurrent | <2x p99 latency | Derived |
|
||||
| Multiple clients | >10 (target 100+) | Linear degradation | NFR-3.6 |
|
||||
| Readdir during sync | No blocking | Serve stale if needed | FR-9.2 |
|
||||
|
||||
**Design Response:**
|
||||
- Async I/O (tokio) → non-blocking operations
|
||||
- No GIL → true parallelism across cores
|
||||
- Read-write locks on cache → readers don't block readers
|
||||
- Stale-while-revalidate → serve cached during refresh
|
||||
|
||||
### 3.3 Non-Goals
|
||||
|
||||
| ID | Non-Goal | Rationale |
|
||||
|----|----------|-----------|
|
||||
| NG1 | Write to origin files | Read-only by design; preserves originals |
|
||||
| NG2 | Transcoding | Out of scope for MVP; plugin possible later |
|
||||
| NG3 | Video file support | Focus on audio; deferred to future |
|
||||
| NG4 | Distributed/clustered mode | Single-node for MVP; architecture supports later |
|
||||
| NG5 | Mobile app | CLI/daemon only; filesystem interface |
|
||||
|
||||
---
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1 High-Level Architecture
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
skinparam componentStyle rectangle
|
||||
|
||||
package "User Space" {
|
||||
[Media Players\n(mpv, VLC, Plex)] as Apps
|
||||
|
||||
package "MusicFS Daemon" {
|
||||
[FUSE Interface] as FUSE
|
||||
[Control API] as Control
|
||||
[Metrics] as Metrics
|
||||
|
||||
package "Core Services" {
|
||||
[Virtual Path\nResolver] as VPR
|
||||
[Event Bus] as Events
|
||||
[Search Engine\n(tantivy)] as Search
|
||||
}
|
||||
|
||||
package "Plugin Host" {
|
||||
[Origin\nPlugins] as OriginPlugins
|
||||
[Metadata\nPlugins] as MetaPlugins
|
||||
[Format\nPlugins] as FormatPlugins
|
||||
}
|
||||
|
||||
package "Storage Layer" {
|
||||
[Content-Addressable\nStore (CAS)] as CAS
|
||||
database "SQLite\n(metadata)" as SQLite
|
||||
database "sled\n(chunks)" as Sled
|
||||
}
|
||||
|
||||
[Origin\nFederation] as Federation
|
||||
}
|
||||
}
|
||||
|
||||
package "Origins (Read-Only)" {
|
||||
[Local FS] as Local
|
||||
[NFS] as NFS
|
||||
[S3] as S3
|
||||
[SFTP] as SFTP
|
||||
}
|
||||
|
||||
Apps --> FUSE : POSIX
|
||||
FUSE --> VPR
|
||||
VPR --> Events
|
||||
VPR --> Search
|
||||
VPR --> CAS
|
||||
CAS --> SQLite
|
||||
CAS --> Sled
|
||||
VPR --> Federation
|
||||
Federation --> OriginPlugins
|
||||
OriginPlugins --> Local
|
||||
OriginPlugins --> NFS
|
||||
OriginPlugins --> S3
|
||||
OriginPlugins --> SFTP
|
||||
Control --> Events
|
||||
Metrics --> Events
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
### 4.2 Component Overview
|
||||
|
||||
| Component | Responsibility | Technology |
|
||||
|-----------|---------------|------------|
|
||||
| FUSE Interface | Translate POSIX ops to internal calls | fuser (Rust) |
|
||||
| Virtual Path Resolver | Map virtual ↔ real paths | Custom |
|
||||
| Event Bus | Decouple components, enable observability | tokio broadcast |
|
||||
| Search Engine | Full-text metadata search | tantivy |
|
||||
| Plugin Host | Load/manage plugins | Native + WASM |
|
||||
| CAS | Content-addressed chunk storage | Custom + sled |
|
||||
| Origin Federation | Multi-origin routing with failover | Custom |
|
||||
|
||||
### 4.3 Detailed Design
|
||||
|
||||
#### 4.3.1 Virtual Path Resolution
|
||||
|
||||
The resolver maps metadata-based virtual paths to real origin paths.
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
participant "FUSE" as F
|
||||
participant "VirtualPathResolver" as VPR
|
||||
participant "MetadataIndex" as MI
|
||||
participant "TreeCache" as TC
|
||||
participant "OriginFederation" as OF
|
||||
|
||||
F -> VPR : lookup("/Metallica/72 Seasons/01.flac")
|
||||
VPR -> TC : get_cached(path)
|
||||
alt cache hit
|
||||
TC --> VPR : CachedEntry
|
||||
else cache miss
|
||||
VPR -> MI : query(artist="Metallica", album="72 Seasons", track=1)
|
||||
MI --> VPR : FileRecord { origin_id, real_path, metadata }
|
||||
VPR -> TC : store(path, entry)
|
||||
end
|
||||
VPR -> OF : resolve_origin(origin_id)
|
||||
OF --> VPR : OriginHandle
|
||||
VPR --> F : ResolvedPath { origin, real_path, inode }
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
**Path Template Grammar:**
|
||||
```
|
||||
template = segment ("/" segment)*
|
||||
segment = (literal | variable)+
|
||||
variable = "$" identifier
|
||||
identifier = "artist" | "album" | "title" | "track" | "year" | "genre"
|
||||
| "format" | "format_upper" | "disc"
|
||||
```
|
||||
|
||||
**Default Template:**
|
||||
```
|
||||
$artist/$album ($year) [$format_upper]/$track - $title.$format
|
||||
```
|
||||
|
||||
#### 4.3.2 Content-Addressable Store (CAS)
|
||||
|
||||
All file content is stored as content-addressed chunks, enabling deduplication
|
||||
and efficient delta sync.
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
package "Content-Addressable Store" {
|
||||
component "Chunk Manager" as CM
|
||||
component "CDC Chunker\n(FastCDC)" as CDC
|
||||
component "Hash Index\n(xxHash64)" as Hash
|
||||
|
||||
database "Chunk Files\n~/.cache/musicfs/chunks/" as Chunks
|
||||
database "Index DB\n(sled)" as Index
|
||||
|
||||
CM --> CDC : chunk data
|
||||
CDC --> Hash : compute hash
|
||||
Hash --> Index : store hash → location
|
||||
CM --> Chunks : write chunk file
|
||||
}
|
||||
|
||||
note right of CDC
|
||||
Avg chunk: 64KB
|
||||
Min: 16KB, Max: 256KB
|
||||
Stable boundaries for delta sync
|
||||
end note
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
**Chunk Storage Layout:**
|
||||
```
|
||||
~/.cache/musicfs/
|
||||
├── chunks/
|
||||
│ ├── aa/
|
||||
│ │ ├── aa1b2c3d4e5f6789... (64KB chunk)
|
||||
│ │ └── aa9f8e7d6c5b4a32...
|
||||
│ ├── ab/
|
||||
│ └── ... (256 subdirs for distribution)
|
||||
├── metadata.db (SQLite: file metadata, tree cache)
|
||||
├── search.idx/ (tantivy: full-text index)
|
||||
└── chunks.sled/ (sled: hash → chunk location)
|
||||
```
|
||||
|
||||
#### 4.3.3 Origin Federation
|
||||
|
||||
Multiple origins are managed with priority-based routing and health tracking.
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
participant "VirtualPathResolver" as VPR
|
||||
participant "OriginFederation" as OF
|
||||
participant "HealthChecker" as HC
|
||||
participant "Origin[Local]" as O1
|
||||
participant "Origin[NFS]" as O2
|
||||
participant "Origin[S3]" as O3
|
||||
|
||||
VPR -> OF : read(real_path, offset, size)
|
||||
OF -> OF : select_origin(priority, health)
|
||||
|
||||
alt Origin[Local] healthy (pri=1)
|
||||
OF -> O1 : read()
|
||||
O1 --> OF : data
|
||||
else Origin[Local] unhealthy, try NFS (pri=2)
|
||||
OF -> O2 : read()
|
||||
alt success
|
||||
O2 --> OF : data
|
||||
else failure
|
||||
OF -> O3 : read()
|
||||
O3 --> OF : data
|
||||
end
|
||||
end
|
||||
|
||||
OF --> VPR : data
|
||||
|
||||
note over HC
|
||||
Background health checks
|
||||
every 30s per origin
|
||||
end note
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
**Origin Configuration:**
|
||||
```toml
|
||||
[[origins]]
|
||||
id = "local"
|
||||
type = "local"
|
||||
path = "/mnt/nas/music"
|
||||
priority = 1
|
||||
|
||||
[[origins]]
|
||||
id = "backup"
|
||||
type = "s3"
|
||||
bucket = "music-backup"
|
||||
priority = 2
|
||||
```
|
||||
|
||||
#### 4.3.4 Plugin System
|
||||
|
||||
Plugins extend functionality without modifying core code.
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
interface "Plugin" {
|
||||
+name(): String
|
||||
+version(): Version
|
||||
+init(config)
|
||||
+shutdown()
|
||||
}
|
||||
|
||||
interface "OriginPlugin" {
|
||||
+list_dir(path): Vec<DirEntry>
|
||||
+read(path, offset, size): Vec<u8>
|
||||
+stat(path): FileStat
|
||||
+watch(path, callback): WatchHandle
|
||||
}
|
||||
|
||||
interface "MetadataPlugin" {
|
||||
+extract(data, format): Metadata
|
||||
+can_handle(format): bool
|
||||
}
|
||||
|
||||
interface "FormatPlugin" {
|
||||
+extensions(): Vec<String>
|
||||
+parse_header(data): AudioHeader
|
||||
+synthesize_header(metadata): Vec<u8>
|
||||
}
|
||||
|
||||
Plugin <|-- OriginPlugin
|
||||
Plugin <|-- MetadataPlugin
|
||||
Plugin <|-- FormatPlugin
|
||||
|
||||
class "LocalFSPlugin" implements OriginPlugin
|
||||
class "S3Plugin" implements OriginPlugin
|
||||
class "SymphoniaPlugin" implements MetadataPlugin
|
||||
class "FlacPlugin" implements FormatPlugin
|
||||
class "Mp3Plugin" implements FormatPlugin
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
**Plugin Loading:**
|
||||
1. **Built-in:** Compiled into binary (Local, S3, SFTP, symphonia)
|
||||
2. **Native:** Dynamic libraries (`.so`/`.dylib`) loaded at runtime
|
||||
3. **WASM:** Sandboxed plugins via wasmtime (future)
|
||||
|
||||
#### 4.3.5 Data Flow: Read Operation
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
|FUSE|
|
||||
start
|
||||
:receive read(path, offset, size);
|
||||
|
||||
|VirtualPathResolver|
|
||||
:resolve virtual path to real path;
|
||||
:lookup file metadata;
|
||||
|
||||
|CAS|
|
||||
:compute chunk range for [offset, offset+size];
|
||||
if (all chunks cached?) then (yes)
|
||||
:read from local chunk files;
|
||||
else (no)
|
||||
|OriginFederation|
|
||||
:select healthy origin by priority;
|
||||
:fetch missing byte range;
|
||||
|CAS|
|
||||
:chunk fetched data (CDC);
|
||||
:store chunks by hash;
|
||||
:update chunk manifest;
|
||||
endif
|
||||
|
||||
|EventBus|
|
||||
:emit FileAccessed event;
|
||||
|
||||
|FUSE|
|
||||
:return data to application;
|
||||
stop
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
#### 4.3.6 Data Schema
|
||||
|
||||
**Metadata Index (SQLite):**
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
id INTEGER PRIMARY KEY,
|
||||
origin_id TEXT NOT NULL,
|
||||
real_path TEXT NOT NULL,
|
||||
virtual_path TEXT NOT NULL,
|
||||
|
||||
-- Metadata (see FR-6 in requirements.md)
|
||||
title TEXT,
|
||||
artist TEXT,
|
||||
album TEXT,
|
||||
album_artist TEXT,
|
||||
genre TEXT,
|
||||
year INTEGER,
|
||||
track INTEGER,
|
||||
disc INTEGER,
|
||||
duration_ms INTEGER,
|
||||
bitrate INTEGER,
|
||||
sample_rate INTEGER,
|
||||
format TEXT,
|
||||
|
||||
-- Sync state
|
||||
origin_mtime INTEGER,
|
||||
origin_size INTEGER,
|
||||
content_hash TEXT,
|
||||
chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)]
|
||||
last_sync INTEGER,
|
||||
|
||||
UNIQUE(origin_id, real_path)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_virtual ON files(virtual_path);
|
||||
CREATE INDEX idx_artist_album ON files(artist, album);
|
||||
CREATE INDEX idx_content_hash ON files(content_hash);
|
||||
|
||||
CREATE TABLE artwork (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_id INTEGER REFERENCES files(id),
|
||||
art_type TEXT, -- 'front', 'back'
|
||||
chunk_hash TEXT, -- reference to CAS
|
||||
width INTEGER,
|
||||
height INTEGER,
|
||||
UNIQUE(file_id, art_type)
|
||||
);
|
||||
|
||||
CREATE TABLE collections (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT UNIQUE,
|
||||
query_json TEXT, -- smart collection query
|
||||
created_at INTEGER
|
||||
);
|
||||
```
|
||||
|
||||
#### 4.3.7 Control API
|
||||
|
||||
**Unix Socket Protocol (JSON-RPC 2.0):**
|
||||
|
||||
```json
|
||||
// Request: Get cache statistics
|
||||
{"jsonrpc": "2.0", "method": "cache.stats", "id": 1}
|
||||
|
||||
// Response
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"result": {
|
||||
"hits": 15234,
|
||||
"misses": 421,
|
||||
"hit_rate": 0.973,
|
||||
"chunks_stored": 84521,
|
||||
"chunks_unique": 71203,
|
||||
"dedup_ratio": 0.157,
|
||||
"size_bytes": 5368709120
|
||||
}
|
||||
}
|
||||
|
||||
// Request: Search
|
||||
{"jsonrpc": "2.0", "method": "search", "params": {"query": "metallica"}, "id": 2}
|
||||
|
||||
// Request: Refresh origin
|
||||
{"jsonrpc": "2.0", "method": "origin.rescan", "params": {"id": "local"}, "id": 3}
|
||||
```
|
||||
|
||||
**CLI Interface:**
|
||||
```bash
|
||||
musicfs mount /mnt/music # Mount filesystem
|
||||
musicfs status # Show daemon status
|
||||
musicfs cache stats # Cache statistics
|
||||
musicfs cache clear --origin=local # Clear cache for origin
|
||||
musicfs search "metallica heavy" # Search library
|
||||
musicfs origin list # List origins and health
|
||||
musicfs origin rescan local # Force rescan
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1 Security & Privacy
|
||||
|
||||
| Concern | Mitigation |
|
||||
|---------|------------|
|
||||
| Credential storage | Use system keyring (secret-service) or env vars; never in config file |
|
||||
| Credential exposure | Redact from logs; exclude from `/proc/cmdline` |
|
||||
| Cache at rest | Optional encryption via age/libsodium (P3 requirement) |
|
||||
| Plugin sandboxing | WASM plugins run in wasmtime sandbox; native plugins require trust |
|
||||
| Access control | Respect origin permissions; run as unprivileged user |
|
||||
| No PII handling | Filesystem metadata only; no user data collected |
|
||||
|
||||
### 5.2 Observability
|
||||
|
||||
**Metrics (Prometheus format):**
|
||||
```
|
||||
musicfs_fuse_ops_total{op="read"} 152341
|
||||
musicfs_fuse_ops_total{op="readdir"} 8234
|
||||
musicfs_fuse_latency_seconds{op="read",quantile="0.99"} 0.004
|
||||
musicfs_cache_hits_total 142107
|
||||
musicfs_cache_misses_total 10234
|
||||
musicfs_cache_size_bytes 5368709120
|
||||
musicfs_origin_health{origin="local"} 1
|
||||
musicfs_origin_health{origin="s3"} 0
|
||||
musicfs_sync_files_changed{origin="local"} 15
|
||||
```
|
||||
|
||||
**Logging Levels:**
|
||||
| Level | Content |
|
||||
|-------|---------|
|
||||
| ERROR | Unrecoverable failures, data corruption |
|
||||
| WARN | Recoverable failures, origin timeouts |
|
||||
| INFO | Mount/unmount, sync completion, config reload |
|
||||
| DEBUG | Cache hits/misses, origin selection |
|
||||
| TRACE | Individual FUSE operations, chunk I/O |
|
||||
|
||||
**Golden Signals Dashboard:**
|
||||
1. **Latency:** p50/p95/p99 for read, stat, readdir
|
||||
2. **Traffic:** FUSE ops/sec, bytes read/sec
|
||||
3. **Errors:** Origin failures, cache corruption
|
||||
4. **Saturation:** Cache fullness, open file handles
|
||||
|
||||
### 5.3 Scalability & Performance
|
||||
|
||||
**Expected Load:**
|
||||
| Metric | Target | Maximum |
|
||||
|--------|--------|---------|
|
||||
| Library size | 1M files | 10M files |
|
||||
| Concurrent clients | 10 | 100+ |
|
||||
| FUSE ops/sec | 1,000 | 10,000 |
|
||||
| Read throughput | 500 MB/s | 1 GB/s |
|
||||
|
||||
**Scaling Strategy:**
|
||||
- **Horizontal:** Not supported (single daemon per mountpoint)
|
||||
- **Vertical:** Increase cache size, add origins
|
||||
|
||||
**Resource Requirements:**
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| CPU | 1 core | 4 cores |
|
||||
| RAM | 256 MB | 2 GB |
|
||||
| Disk (cache) | 1 GB | 50 GB |
|
||||
| Network | 10 Mbps | 1 Gbps |
|
||||
|
||||
### 5.4 Testing Plan
|
||||
|
||||
| Test Type | Scope | Tools |
|
||||
|-----------|-------|-------|
|
||||
| Unit | Individual components | cargo test |
|
||||
| Integration | Component interaction | cargo test --features integration |
|
||||
| E2E | Full FUSE operations | pytest + real mount |
|
||||
| Performance | Latency, throughput | criterion.rs, custom benchmarks |
|
||||
| Stress | High load, large libraries | locust, custom generators |
|
||||
| Chaos | Origin failures, network issues | toxiproxy |
|
||||
|
||||
**Test Matrix:**
|
||||
```
|
||||
Origins: [local, s3, sftp] × [healthy, degraded, offline]
|
||||
Cache: [cold, warm, full]
|
||||
Library: [100, 10K, 1M, 10M] files
|
||||
Operations: [mount, readdir, stat, read, search]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1 Alternative A: Extend beetfs (Python)
|
||||
|
||||
**Description:** Fix bugs in existing beetfs, add features incrementally.
|
||||
|
||||
**Rejected Because:**
|
||||
- Python GIL fundamentally limits concurrency
|
||||
- Python 2.7 EOL; migration to Python 3 substantial
|
||||
- Architecture (full file in RAM) requires rewrite anyway
|
||||
- No async I/O support in fuse-python
|
||||
|
||||
### 6.2 Alternative B: Use rclone mount
|
||||
|
||||
**Description:** Use rclone's FUSE mount with VFS caching.
|
||||
|
||||
**Rejected Because:**
|
||||
- No metadata-based virtual path organization
|
||||
- No metadata overlay functionality
|
||||
- Limited plugin extensibility
|
||||
- Would require forking and heavy modification
|
||||
|
||||
### 6.3 Alternative C: Build as Plex/Jellyfin Plugin
|
||||
|
||||
**Description:** Extend existing media server with virtual filesystem view.
|
||||
|
||||
**Rejected Because:**
|
||||
- Tied to specific media server
|
||||
- Not a true filesystem (no POSIX interface)
|
||||
- Heavy runtime dependency
|
||||
- Different use case (streaming vs filesystem)
|
||||
|
||||
### 6.4 Alternative D: Go Implementation
|
||||
|
||||
**Description:** Implement in Go using go-fuse.
|
||||
|
||||
**Considered Trade-offs:**
|
||||
| Aspect | Rust | Go |
|
||||
|--------|------|-----|
|
||||
| Memory safety | Compile-time | GC pauses |
|
||||
| Concurrency | async/await, no GC | goroutines, GC |
|
||||
| FUSE library | fuser (mature) | go-fuse (mature) |
|
||||
| Learning curve | Steeper | Gentler |
|
||||
| Binary size | Smaller | Larger |
|
||||
|
||||
**Decision:** Rust chosen for zero-cost abstractions, no GC pauses during I/O,
|
||||
and better fit for systems programming.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### 7.1 Phase 1: MVP (4 weeks)
|
||||
|
||||
**Goal:** Basic functional filesystem with single origin.
|
||||
|
||||
| Week | Deliverables |
|
||||
|------|--------------|
|
||||
| 1 | Project setup, FUSE skeleton, local origin plugin |
|
||||
| 2 | Metadata extraction (symphonia), SQLite schema |
|
||||
| 3 | Virtual path resolver, tree cache, basic readdir/stat/read |
|
||||
| 4 | CAS implementation, chunk caching, integration tests |
|
||||
|
||||
**Exit Criteria:**
|
||||
- Mount and browse local music library
|
||||
- Play audio files through mounted filesystem
|
||||
- Cache persists across restarts
|
||||
|
||||
### 7.2 Phase 2: Delta Sync & Multi-Origin (3 weeks)
|
||||
|
||||
**Goal:** Efficient synchronization and origin federation.
|
||||
|
||||
| Week | Deliverables |
|
||||
|------|--------------|
|
||||
| 5 | CDC chunking (FastCDC), delta detection |
|
||||
| 6 | Origin federation, priority routing, health checks |
|
||||
| 7 | S3 origin plugin, SFTP origin plugin |
|
||||
|
||||
**Exit Criteria:**
|
||||
- Delta sync achieves >90% bandwidth reduction
|
||||
- Automatic failover between origins
|
||||
- Remote origins functional
|
||||
|
||||
### 7.3 Phase 3: Search & Smart Features (2 weeks)
|
||||
|
||||
**Goal:** Full-text search and intelligent caching.
|
||||
|
||||
| Week | Deliverables |
|
||||
|------|--------------|
|
||||
| 8 | tantivy integration, search indexing, `/.search/` virtual dir |
|
||||
| 9 | Smart collections, prefetch engine, access pattern learning |
|
||||
|
||||
**Exit Criteria:**
|
||||
- Search returns results in <1s for 1M tracks
|
||||
- Prefetch reduces cache misses by >50%
|
||||
|
||||
### 7.4 Phase 4: Plugin System & Polish (2 weeks)
|
||||
|
||||
**Goal:** Extensibility and production readiness.
|
||||
|
||||
| Week | Deliverables |
|
||||
|------|--------------|
|
||||
| 10 | Plugin host, plugin API stabilization, example plugins |
|
||||
| 11 | Control API, metrics, documentation, packaging |
|
||||
|
||||
**Exit Criteria:**
|
||||
- Custom origin plugin loadable at runtime
|
||||
- Prometheus metrics exported
|
||||
- systemd service functional
|
||||
|
||||
### 7.5 Rollout Strategy
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
|
||||
[*] --> Alpha
|
||||
Alpha --> Beta : Internal testing complete
|
||||
Beta --> GA : Community testing complete
|
||||
|
||||
state Alpha {
|
||||
[*] --> DevTesting
|
||||
DevTesting --> DogFood : Core features work
|
||||
}
|
||||
|
||||
state Beta {
|
||||
[*] --> LimitedRelease
|
||||
LimitedRelease --> PublicBeta : No critical bugs
|
||||
}
|
||||
|
||||
state GA {
|
||||
[*] --> Stable
|
||||
}
|
||||
|
||||
note right of Alpha : 2-4 weeks\nDevelopers only
|
||||
note right of Beta : 4-8 weeks\nEarly adopters
|
||||
note right of GA : Stable releases
|
||||
|
||||
@enduml
|
||||
```
|
||||
|
||||
**Feature Flags:**
|
||||
```toml
|
||||
[features]
|
||||
search_enabled = true
|
||||
smart_collections = false # Beta
|
||||
wasm_plugins = false # Experimental
|
||||
```
|
||||
|
||||
**Rollback:** Binary replacement + cache clear; no data migration needed.
|
||||
|
||||
---
|
||||
|
||||
## 8. Glossary & References
|
||||
|
||||
### 8.1 Glossary
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **CAS** | Content-Addressable Store; data stored/retrieved by hash |
|
||||
| **CDC** | Content-Defined Chunking; chunking with stable boundaries |
|
||||
| **FUSE** | Filesystem in Userspace; kernel interface for user-space filesystems |
|
||||
| **Origin** | Source storage backend (local, S3, NFS, etc.) |
|
||||
| **Virtual Path** | Metadata-derived path shown to users |
|
||||
| **Real Path** | Actual path on origin storage |
|
||||
|
||||
### 8.2 References
|
||||
|
||||
| Document | Link |
|
||||
|----------|------|
|
||||
| Requirements Specification | [requirements.md](requirements.md) |
|
||||
| beetfs (Original) | [beetsplug/beetFs.py](../../beetsplug/beetFs.py) |
|
||||
| beetfs Features | [v1/features.md](../v1/features.md) |
|
||||
| fuser (Rust FUSE) | https://github.com/cberner/fuser |
|
||||
| tantivy (Search) | https://github.com/quickwit-oss/tantivy |
|
||||
| symphonia (Audio) | https://github.com/pdrat/symphonia |
|
||||
| FastCDC | https://github.com/nlfiedler/fastcdc-rs |
|
||||
| wasmtime | https://wasmtime.dev/ |
|
||||
|
||||
### 8.3 Dependencies
|
||||
|
||||
| Crate | Version | Purpose |
|
||||
|-------|---------|---------|
|
||||
| fuser | 0.14+ | FUSE interface |
|
||||
| tokio | 1.x | Async runtime |
|
||||
| rusqlite | 0.31+ | SQLite bindings |
|
||||
| sled | 0.34+ | Embedded key-value store |
|
||||
| tantivy | 0.21+ | Full-text search |
|
||||
| symphonia | 0.5+ | Audio metadata extraction |
|
||||
| fastcdc | 3.x | Content-defined chunking |
|
||||
| xxhash-rust | 0.8+ | Fast hashing |
|
||||
| serde | 1.x | Serialization |
|
||||
| toml | 0.8+ | Configuration |
|
||||
| tracing | 0.1+ | Logging/instrumentation |
|
||||
| metrics | 0.22+ | Prometheus metrics |
|
||||
@@ -0,0 +1,649 @@
|
||||
# Music Library FUSE Filesystem - Requirements Specification
|
||||
|
||||
**Version**: 1.0
|
||||
**Date**: 2026-05-12
|
||||
**Status**: Draft
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
### 1.1 Purpose
|
||||
|
||||
This document specifies the requirements for a FUSE-based virtual filesystem that presents a music library organized by metadata. The system overlays metadata onto audio files without modifying originals and operates as a read-only client against the origin storage.
|
||||
|
||||
### 1.2 Scope
|
||||
|
||||
The system provides:
|
||||
- Virtual filesystem accessible via standard POSIX operations
|
||||
- Metadata-based directory structure (artist/album/track)
|
||||
- Local caching with delta synchronization
|
||||
- Support for local and remote origin storage
|
||||
|
||||
### 1.3 Definitions
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **Origin** | The source storage containing original audio files (local FS, NFS, S3, etc.) |
|
||||
| **Virtual path** | The metadata-derived path shown to users (e.g., `/Artist/Album/Track.flac`) |
|
||||
| **Real path** | The actual path on origin storage |
|
||||
| **Metadata overlay** | Serving synthesized file headers from cached metadata |
|
||||
| **CDC** | Content-Defined Chunking - algorithm for stable file segmentation |
|
||||
|
||||
---
|
||||
|
||||
## 2. System Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ User Applications │
|
||||
│ (mpv, Rhythmbox, Plex, etc.) │
|
||||
└─────────────────────────────┬───────────────────────────────────┘
|
||||
│ POSIX (read-only)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ FUSE Interface │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Plugin Host │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Origin │ │ Metadata │ │ Format │ │
|
||||
│ │ Plugins │ │ Plugins │ │ Plugins │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Core Services │
|
||||
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
|
||||
│ │ Virtual │ │ Event │ │ Search │ │ Control │ │
|
||||
│ │ Path │ │ Bus │ │ Index │ │ API │ │
|
||||
│ │ Resolver │ │ │ │ │ │ │ │
|
||||
│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Storage Layer │
|
||||
│ ┌─────────────────────────────────────────────────────────┐ │
|
||||
│ │ Content-Addressable Chunk Store │ │
|
||||
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
|
||||
│ │ │ Metadata │ │ Content │ │ Tree │ │ │
|
||||
│ │ │ Cache │ │ Chunks │ │ Cache │ │ │
|
||||
│ │ │ (SQLite) │ │ (CAS) │ │ │ │ │
|
||||
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
|
||||
│ └─────────────────────────────────────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────────────────┤
|
||||
│ Origin Federation │
|
||||
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
||||
│ │ Local │ │ NFS │ │ S3 │ │ SFTP │ │
|
||||
│ │ FS │ │ │ │ │ │ │ │
|
||||
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ read-only
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Origin Storage(s) │
|
||||
│ (original audio files) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Functional Requirements
|
||||
|
||||
### 3.1 Filesystem Operations
|
||||
|
||||
#### FR-1: Mount/Unmount
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-1.1 | The system SHALL mount as a FUSE filesystem at a user-specified mountpoint |
|
||||
| FR-1.2 | The system SHALL return control to the caller within 500ms of mount initiation |
|
||||
| FR-1.3 | The system SHALL unmount cleanly via `fusermount -u` |
|
||||
| FR-1.4 | The system SHALL release all resources (file handles, connections) on unmount |
|
||||
|
||||
#### FR-2: Directory Operations
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-2.1 | The system SHALL present files organized by metadata path format |
|
||||
| FR-2.2 | The system SHALL support configurable path templates (e.g., `$artist/$album/$track - $title.$format`) |
|
||||
| FR-2.3 | The system SHALL return directory listings via `readdir()` |
|
||||
| FR-2.4 | The system SHALL support nested directory traversal to arbitrary depth |
|
||||
| FR-2.5 | The system SHALL handle directories with 100,000+ entries |
|
||||
|
||||
#### FR-3: File Operations (Read)
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-3.1 | The system SHALL support `open()` for reading |
|
||||
| FR-3.2 | The system SHALL support `read()` with arbitrary offset and size |
|
||||
| FR-3.3 | The system SHALL support `seek()` operations for random access |
|
||||
| FR-3.4 | The system SHALL return file attributes via `stat()` / `fstat()` |
|
||||
| FR-3.5 | The system SHALL support concurrent reads from multiple processes |
|
||||
|
||||
#### FR-4: Read-Only Constraint
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-4.1 | The system SHALL NOT modify original files on the origin storage |
|
||||
| FR-4.2 | The system SHALL NOT push any changes to the origin server |
|
||||
| FR-4.3 | The system SHALL return `EROFS` (Read-only filesystem) for write operations |
|
||||
| FR-4.4 | The system SHALL return `EROFS` for `create()`, `mkdir()`, `unlink()`, `rmdir()` |
|
||||
| FR-4.5 | The system SHALL return `EROFS` for `rename()`, `chmod()`, `chown()`, `truncate()` |
|
||||
|
||||
### 3.2 Metadata Handling
|
||||
|
||||
#### FR-5: Metadata Overlay
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-5.1 | The system SHALL extract metadata from audio files on first access |
|
||||
| FR-5.2 | The system SHALL cache extracted metadata in a local database |
|
||||
| FR-5.3 | The system SHALL serve file headers with metadata from cache |
|
||||
| FR-5.4 | The system SHALL support FLAC Vorbis comments |
|
||||
| FR-5.5 | The system SHALL support MP3 ID3v2 tags |
|
||||
| FR-5.6 | The system SHOULD support additional formats (OGG, M4A, OPUS) |
|
||||
|
||||
#### FR-6: Metadata Fields
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-6.1 | The system SHALL extract and cache: title, artist, album, genre |
|
||||
| FR-6.2 | The system SHALL extract and cache: year, track number, disc number |
|
||||
| FR-6.3 | The system SHALL extract and cache: duration, bitrate, sample rate |
|
||||
| FR-6.4 | The system SHOULD extract: composer, album artist, lyrics |
|
||||
| FR-6.5 | The system SHALL handle missing metadata gracefully with defaults |
|
||||
|
||||
### 3.3 Caching
|
||||
|
||||
#### FR-7: Metadata Cache
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-7.1 | The system SHALL persist metadata cache across restarts |
|
||||
| FR-7.2 | The system SHALL store metadata in SQLite database |
|
||||
| FR-7.3 | The system SHALL index by both virtual path and real path |
|
||||
| FR-7.4 | The system SHALL invalidate cache entries when origin file changes |
|
||||
|
||||
#### FR-8: Content Cache
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-8.1 | The system SHALL cache file content in fixed-size chunks |
|
||||
| FR-8.2 | The system SHALL use content-defined chunking for cache efficiency |
|
||||
| FR-8.3 | The system SHALL store chunk hashes for delta detection |
|
||||
| FR-8.4 | The system SHALL evict chunks under memory/disk pressure |
|
||||
|
||||
#### FR-9: Directory Tree Cache
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-9.1 | The system SHALL cache directory listings locally |
|
||||
| FR-9.2 | The system SHALL serve `readdir()` from cache without origin access |
|
||||
| FR-9.3 | The system SHALL refresh tree cache based on configurable policy |
|
||||
| FR-9.4 | The system SHALL support forced refresh via signal or special file |
|
||||
|
||||
### 3.4 Synchronization
|
||||
|
||||
#### FR-10: Change Detection
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-10.1 | The system SHALL detect changes to origin files |
|
||||
| FR-10.2 | The system SHALL use inotify for local filesystem origins |
|
||||
| FR-10.3 | The system SHALL use polling for remote origins without push support |
|
||||
| FR-10.4 | The system SHALL compare mtime and size for change detection |
|
||||
| FR-10.5 | The system SHALL support content-hash verification on demand |
|
||||
|
||||
#### FR-11: Delta Sync
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-11.1 | The system SHALL download only changed portions of files |
|
||||
| FR-11.2 | The system SHALL use CDC to identify changed chunks |
|
||||
| FR-11.3 | The system SHALL preserve unchanged chunks in cache |
|
||||
| FR-11.4 | The system SHALL handle file additions and deletions |
|
||||
|
||||
### 3.5 Origin Support
|
||||
|
||||
#### FR-12: Origin Types
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-12.1 | The system SHALL support local filesystem as origin |
|
||||
| FR-12.2 | The system SHOULD support NFS mounted filesystems |
|
||||
| FR-12.3 | The system SHOULD support SMB/CIFS shares |
|
||||
| FR-12.4 | The system SHOULD support S3-compatible object storage |
|
||||
| FR-12.5 | The system SHOULD support SFTP servers |
|
||||
| FR-12.6 | The system SHALL provide pluggable origin interface |
|
||||
|
||||
#### FR-13: Multiple Origins [P0]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-13.1 | The system SHALL support multiple simultaneous origins |
|
||||
| FR-13.2 | The system SHALL present unified virtual tree across origins |
|
||||
| FR-13.3 | The system SHALL support origin priority/preference ordering |
|
||||
| FR-13.4 | The system SHALL handle duplicate files across origins |
|
||||
| FR-13.5 | The system SHALL support per-origin configuration |
|
||||
|
||||
### 3.6 Search & Discovery
|
||||
|
||||
#### FR-14: Full-Text Search [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-14.1 | The system SHALL index metadata for full-text search |
|
||||
| FR-14.2 | The system SHALL expose search via virtual directory (`/.search/query/`) |
|
||||
| FR-14.3 | The system SHALL support fuzzy matching |
|
||||
| FR-14.4 | The system SHOULD support search by audio fingerprint |
|
||||
|
||||
#### FR-15: Smart Collections [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-15.1 | The system SHALL support query-based virtual folders |
|
||||
| FR-15.2 | The system SHALL support saved searches as directories |
|
||||
| FR-15.3 | The system SHALL support dynamic playlists (recently played, most played) |
|
||||
| FR-15.4 | The system SHOULD support user-defined metadata fields |
|
||||
|
||||
### 3.7 Album Art
|
||||
|
||||
#### FR-16: Cover Art Handling [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-16.1 | The system SHALL extract embedded album art |
|
||||
| FR-16.2 | The system SHALL expose art as virtual files (`/Artist/Album/cover.jpg`) |
|
||||
| FR-16.3 | The system SHALL cache artwork separately from audio |
|
||||
| FR-16.4 | The system SHALL support multiple art sizes (thumbnail, medium, full) |
|
||||
| FR-16.5 | The system SHOULD fetch missing art from online sources |
|
||||
|
||||
### 3.8 Control & API
|
||||
|
||||
#### FR-17: Control Interface [P0]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-17.1 | The system SHALL expose control via Unix socket |
|
||||
| FR-17.2 | The system SHOULD expose REST/gRPC API |
|
||||
| FR-17.3 | The system SHALL support cache management commands (clear, refresh, stats) |
|
||||
| FR-17.4 | The system SHALL support runtime configuration changes |
|
||||
| FR-17.5 | The system SHALL support graceful shutdown with drain |
|
||||
|
||||
#### FR-18: Event System [P0]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-18.1 | The system SHALL emit events for file access |
|
||||
| FR-18.2 | The system SHALL support webhook notifications |
|
||||
| FR-18.3 | The system SHOULD support event streaming (SSE/WebSocket) |
|
||||
| FR-18.4 | The system SHALL log access patterns for analysis |
|
||||
|
||||
### 3.9 Caching Enhancements
|
||||
|
||||
#### FR-19: Intelligent Prefetching [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-19.1 | The system SHALL learn access patterns |
|
||||
| FR-19.2 | The system SHALL support playlist-aware prefetching |
|
||||
| FR-19.3 | The system SHOULD support time-based prefetching |
|
||||
| FR-19.4 | The system SHALL support manual prefetch hints (`/.prefetch/path/`) |
|
||||
|
||||
#### FR-20: Content-Addressable Storage [P0]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-20.1 | The system SHALL store chunks by content hash |
|
||||
| FR-20.2 | The system SHALL detect identical files across library |
|
||||
| FR-20.3 | The system SHALL report deduplication statistics |
|
||||
| FR-20.4 | The system SHALL enable cache sharing via content addressing |
|
||||
|
||||
### 3.10 Integration
|
||||
|
||||
#### FR-21: Metadata Sources [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-21.1 | The system SHOULD integrate with MusicBrainz |
|
||||
| FR-21.2 | The system SHOULD integrate with Discogs |
|
||||
| FR-21.3 | The system SHOULD integrate with Last.fm |
|
||||
| FR-21.4 | The system SHOULD support AcoustID fingerprinting |
|
||||
| FR-21.5 | The system SHALL support custom metadata plugins |
|
||||
|
||||
#### FR-22: Import & Migration [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-22.1 | The system SHALL import from beets database |
|
||||
| FR-22.2 | The system SHOULD import from iTunes/Apple Music library |
|
||||
| FR-22.3 | The system SHALL export library metadata |
|
||||
|
||||
### 3.11 Extensibility
|
||||
|
||||
#### FR-23: Plugin System [P0]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-23.1 | The system SHALL support loadable plugins |
|
||||
| FR-23.2 | The system SHALL define stable plugin API |
|
||||
| FR-23.3 | The system SHALL support plugins for: origins, metadata extractors, formats |
|
||||
| FR-23.4 | The system SHOULD support WASM plugins for sandboxed execution |
|
||||
| FR-23.5 | The system SHALL provide plugin lifecycle management (load, unload, reload) |
|
||||
|
||||
#### FR-24: Format Extensibility [P1]
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-24.1 | The system SHALL support pluggable codec modules |
|
||||
| FR-24.2 | The system SHOULD support audiobook formats (M4B, chapters) |
|
||||
| FR-24.3 | The system SHALL allow format plugins to register file extensions |
|
||||
|
||||
### 3.12 High Availability [P3]
|
||||
|
||||
#### FR-25: Resilience
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| FR-25.1 | The system SHOULD support active-passive failover |
|
||||
| FR-25.2 | The system SHOULD support read replicas |
|
||||
| FR-25.3 | The system SHALL support zero-downtime upgrades |
|
||||
| FR-25.4 | The system SHALL support cache backup/restore |
|
||||
| FR-25.5 | The system SHALL validate cache integrity on startup |
|
||||
|
||||
---
|
||||
|
||||
## 4. Non-Functional Requirements
|
||||
|
||||
### 4.1 Performance
|
||||
|
||||
#### NFR-1: Latency
|
||||
|
||||
| ID | Requirement | Target | Maximum |
|
||||
|----|-------------|--------|---------|
|
||||
| NFR-1.1 | `stat()` on cached file | <1ms | 5ms |
|
||||
| NFR-1.2 | `readdir()` on cached directory | <10ms | 50ms |
|
||||
| NFR-1.3 | `open()` on cached file | <5ms | 20ms |
|
||||
| NFR-1.4 | `read()` from cache | <1ms | 5ms |
|
||||
| NFR-1.5 | `read()` cache miss (local origin) | <50ms | 200ms |
|
||||
| NFR-1.6 | `read()` cache miss (remote origin) | <200ms | 1000ms |
|
||||
| NFR-1.7 | Mount completion | <100ms | 500ms |
|
||||
|
||||
#### NFR-2: Throughput
|
||||
|
||||
| ID | Requirement | Target |
|
||||
|----|-------------|--------|
|
||||
| NFR-2.1 | Sequential read throughput (cached) | >500 MB/s |
|
||||
| NFR-2.2 | Sequential read throughput (local origin) | >200 MB/s |
|
||||
| NFR-2.3 | Metadata operations per second | >1000 ops/s |
|
||||
| NFR-2.4 | Concurrent file handles | >1000 |
|
||||
|
||||
#### NFR-3: Scalability
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-3.1 | The system SHALL handle libraries with 1,000,000+ files |
|
||||
| NFR-3.2 | The system SHALL handle directories with 100,000+ entries |
|
||||
| NFR-3.3 | The system SHALL maintain O(1) mount time regardless of library size |
|
||||
| NFR-3.4 | The system SHALL maintain O(log n) lookup time for paths |
|
||||
| NFR-3.5 | The system SHOULD handle libraries with 10,000,000+ files [P3] |
|
||||
| NFR-3.6 | The system SHOULD support 100+ concurrent clients [P3] |
|
||||
| NFR-3.7 | The system SHOULD achieve <100μs cached stat for high-performance use [P3] |
|
||||
|
||||
### 4.2 Resource Usage
|
||||
|
||||
#### NFR-4: Memory
|
||||
|
||||
| ID | Requirement | Limit |
|
||||
|----|-------------|-------|
|
||||
| NFR-4.1 | Idle memory usage | <50 MB |
|
||||
| NFR-4.2 | Active usage (1000 files accessed) | <200 MB |
|
||||
| NFR-4.3 | Peak usage under load | <500 MB |
|
||||
| NFR-4.4 | Per-file metadata overhead | <1 KB |
|
||||
| NFR-4.5 | The system SHALL NOT load entire files into memory |
|
||||
|
||||
#### NFR-5: Disk
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-5.1 | Metadata cache size SHALL be configurable (default: 100 MB) |
|
||||
| NFR-5.2 | Content cache size SHALL be configurable (default: 10 GB) |
|
||||
| NFR-5.3 | The system SHALL evict cache entries under disk pressure |
|
||||
| NFR-5.4 | The system SHALL function with cache disabled (passthrough mode) |
|
||||
|
||||
#### NFR-6: Network
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-6.1 | The system SHALL minimize network round-trips via batching |
|
||||
| NFR-6.2 | The system SHALL use connection pooling for remote origins |
|
||||
| NFR-6.3 | The system SHALL support bandwidth limiting (configurable) |
|
||||
| NFR-6.4 | Delta sync SHALL achieve >90% bandwidth reduction vs full copy |
|
||||
|
||||
### 4.3 Reliability
|
||||
|
||||
#### NFR-7: Availability
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-7.1 | The system SHALL serve cached data when origin is unavailable |
|
||||
| NFR-7.2 | The system SHALL gracefully degrade with network failures |
|
||||
| NFR-7.3 | The system SHALL retry failed operations with exponential backoff |
|
||||
| NFR-7.4 | The system SHALL not crash on malformed audio files |
|
||||
|
||||
#### NFR-8: Data Integrity
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-8.1 | The system SHALL verify chunk integrity via checksums |
|
||||
| NFR-8.2 | The system SHALL use ACID transactions for cache database |
|
||||
| NFR-8.3 | The system SHALL recover from interrupted synchronization |
|
||||
| NFR-8.4 | The system SHALL detect and report cache corruption |
|
||||
|
||||
### 4.4 Usability
|
||||
|
||||
#### NFR-9: Configuration
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-9.1 | The system SHALL support configuration via file (TOML/YAML) |
|
||||
| NFR-9.2 | The system SHALL support configuration via command-line arguments |
|
||||
| NFR-9.3 | The system SHALL support configuration via environment variables |
|
||||
| NFR-9.4 | The system SHALL provide sensible defaults for all options |
|
||||
|
||||
#### NFR-10: Observability
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-10.1 | The system SHALL log operations at configurable verbosity |
|
||||
| NFR-10.2 | The system SHALL expose metrics (cache hit rate, latency, etc.) |
|
||||
| NFR-10.3 | The system SHALL support health check endpoint/signal |
|
||||
| NFR-10.4 | The system SHOULD support integration with Prometheus/StatsD |
|
||||
|
||||
### 4.5 Compatibility
|
||||
|
||||
#### NFR-11: Platform Support
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-11.1 | The system SHALL run on Linux (kernel 4.x+) |
|
||||
| NFR-11.2 | The system SHOULD run on macOS (via macFUSE) |
|
||||
| NFR-11.3 | The system SHALL require FUSE kernel module |
|
||||
| NFR-11.4 | The system SHALL run without root privileges (user-space FUSE) |
|
||||
|
||||
#### NFR-12: Application Compatibility
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-12.1 | The system SHALL work with standard media players (mpv, VLC, etc.) |
|
||||
| NFR-12.2 | The system SHALL work with media servers (Plex, Jellyfin) |
|
||||
| NFR-12.3 | The system SHALL work with file managers (Nautilus, Dolphin) |
|
||||
| NFR-12.4 | The system SHALL correctly report file sizes and timestamps |
|
||||
|
||||
### 4.6 Security
|
||||
|
||||
#### NFR-13: Access Control
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-13.1 | The system SHALL respect origin file permissions |
|
||||
| NFR-13.2 | The system SHALL run as unprivileged user |
|
||||
| NFR-13.3 | The system SHALL support credential storage for remote origins |
|
||||
| NFR-13.4 | The system SHALL NOT expose credentials in logs or process list |
|
||||
|
||||
### 4.7 Maintainability
|
||||
|
||||
#### NFR-14: Code Quality
|
||||
|
||||
| ID | Requirement |
|
||||
|----|-------------|
|
||||
| NFR-14.1 | The system SHALL be implemented in a memory-safe language |
|
||||
| NFR-14.2 | The system SHALL have no global interpreter lock (no Python/Ruby) |
|
||||
| NFR-14.3 | The system SHALL use async I/O for concurrent operations |
|
||||
| NFR-14.4 | The system SHALL have modular architecture with pluggable components |
|
||||
|
||||
---
|
||||
|
||||
## 5. Constraints
|
||||
|
||||
### 5.1 Technical Constraints
|
||||
|
||||
| ID | Constraint |
|
||||
|----|------------|
|
||||
| C-1 | Must use FUSE for filesystem interface |
|
||||
| C-2 | Must not require kernel module development |
|
||||
| C-3 | Must work with existing audio file formats (no transcoding) |
|
||||
| C-4 | Cache database must be portable (no external database server) |
|
||||
|
||||
### 5.2 Operational Constraints
|
||||
|
||||
| ID | Constraint |
|
||||
|----|------------|
|
||||
| C-5 | Client is read-only; no writes propagate to origin |
|
||||
| C-6 | Must function offline with cached data |
|
||||
| C-7 | Must not corrupt origin files under any circumstances |
|
||||
|
||||
---
|
||||
|
||||
## 6. Assumptions
|
||||
|
||||
| ID | Assumption |
|
||||
|----|------------|
|
||||
| A-1 | Origin storage is accessible via supported protocol |
|
||||
| A-2 | Audio files contain valid metadata headers |
|
||||
| A-3 | Sufficient local disk space for caching is available |
|
||||
| A-4 | FUSE kernel module is installed and accessible |
|
||||
| A-5 | Network connectivity is intermittent but generally available |
|
||||
|
||||
---
|
||||
|
||||
## 7. Dependencies
|
||||
|
||||
| ID | Dependency | Purpose |
|
||||
|----|------------|---------|
|
||||
| D-1 | FUSE library (fuser/libfuse) | Filesystem interface |
|
||||
| D-2 | SQLite | Metadata and tree cache |
|
||||
| D-3 | Audio parsing library (symphonia) | Metadata extraction |
|
||||
| D-4 | Async runtime (tokio) | Concurrent I/O |
|
||||
| D-5 | CDC library (fastcdc) | Content chunking |
|
||||
| D-6 | Full-text search (tantivy) | Search index [P1] |
|
||||
| D-7 | Image processing (image) | Album art thumbnails [P1] |
|
||||
| D-8 | HTTP client (reqwest) | Remote origins, metadata APIs |
|
||||
| D-9 | WASM runtime (wasmtime) | Plugin sandboxing [P0] |
|
||||
| D-10 | Hash library (xxhash/blake3) | Content addressing [P0] |
|
||||
|
||||
---
|
||||
|
||||
## 8. Acceptance Criteria
|
||||
|
||||
### 8.1 Functional Acceptance
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-1 | Mount filesystem and browse directories via `ls` |
|
||||
| AC-2 | Play audio file through mounted filesystem with media player |
|
||||
| AC-3 | Seek within audio file without full download |
|
||||
| AC-4 | Directory listing completes without network access (when cached) |
|
||||
| AC-5 | Confirm write operations return EROFS |
|
||||
| AC-6 | Detect and sync changes from origin within configured interval |
|
||||
|
||||
### 8.2 Performance Acceptance
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-7 | Mount completes in <500ms for library of any size |
|
||||
| AC-8 | Cached stat() completes in <5ms (p99) |
|
||||
| AC-9 | Memory stays under 500MB with 10,000 files accessed |
|
||||
| AC-10 | Tag-only change syncs <10KB of data |
|
||||
|
||||
### 8.3 Reliability Acceptance
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-11 | Filesystem remains accessible when origin is offline |
|
||||
| AC-12 | No data corruption after unclean unmount |
|
||||
| AC-13 | Recovers automatically when origin comes back online |
|
||||
|
||||
### 8.4 Multi-Origin Acceptance [P0]
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-14 | Configure and mount multiple origins simultaneously |
|
||||
| AC-15 | Browse unified tree showing content from all origins |
|
||||
| AC-16 | Access same file from preferred origin when duplicated |
|
||||
|
||||
### 8.5 Search & Discovery Acceptance [P1]
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-17 | Search for tracks by partial artist/album/title match |
|
||||
| AC-18 | Browse smart collection (e.g., "Jazz from 1960s") |
|
||||
| AC-19 | View album art via virtual cover.jpg file |
|
||||
|
||||
### 8.6 Plugin Acceptance [P0]
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-20 | Load custom origin plugin at runtime |
|
||||
| AC-21 | Control daemon via Unix socket (cache stats, refresh) |
|
||||
| AC-22 | Receive webhook on file access event |
|
||||
|
||||
### 8.7 Deduplication Acceptance [P0]
|
||||
|
||||
| ID | Criterion |
|
||||
|----|-----------|
|
||||
| AC-23 | Identical chunks stored once regardless of file count |
|
||||
| AC-24 | Deduplication stats visible via control API |
|
||||
|
||||
---
|
||||
|
||||
## 9. Appendix
|
||||
|
||||
### 9.1 Comparison with beetfs
|
||||
|
||||
| Requirement Area | beetfs | This Specification |
|
||||
|------------------|--------|-------------------|
|
||||
| Mount time | O(N), 5-120s | O(1), <500ms (NFR-1.7) |
|
||||
| Memory per file | Full file size | <1KB (NFR-4.4) |
|
||||
| Write to origin | Yes (DB updates) | No (FR-4.1, FR-4.2) |
|
||||
| Delta sync | None | Required (FR-11) |
|
||||
| Remote origins | None | Required (FR-12) |
|
||||
| Offline access | No | Required (NFR-7.1) |
|
||||
| Cache persistence | No | Required (FR-7.1) |
|
||||
|
||||
### 9.2 Path Template Variables
|
||||
|
||||
| Variable | Description | Example |
|
||||
|----------|-------------|---------|
|
||||
| `$artist` | Track artist | "Metallica" |
|
||||
| `$album` | Album name | "72 Seasons" |
|
||||
| `$title` | Track title | "Lux Æterna" |
|
||||
| `$track` | Track number (zero-padded) | "03" |
|
||||
| `$disc` | Disc number | "1" |
|
||||
| `$year` | Release year | "2023" |
|
||||
| `$genre` | Genre | "Metal" |
|
||||
| `$format` | File extension | "flac" |
|
||||
| `$format_upper` | File extension (uppercase) | "FLAC" |
|
||||
|
||||
### 9.3 Error Codes
|
||||
|
||||
| Operation | Error | Code |
|
||||
|-----------|-------|------|
|
||||
| Any write operation | Read-only filesystem | EROFS (30) |
|
||||
| File not found | No such file | ENOENT (2) |
|
||||
| Origin unavailable | I/O error | EIO (5) |
|
||||
| Permission denied | Access denied | EACCES (13) |
|
||||
Reference in New Issue
Block a user