fujin/MusicFS
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove old docs

Browse Source
This commit is contained in:
Alexander
2026-06-27 16:46:06 +02:00
parent a8262ac8f8
commit 53e6dbaef0
48 changed files with 0 additions and 30526 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api/search.md
-199
View File
@@ -1,199 +0,0 @@
# Search API Documentation
## Overview
MusicFS provides two search interfaces:
1. **FUSE Virtual Directory** - `/.search/query/` for file manager integration
2. **gRPC API** - `Search` and `SearchStream` RPCs for programmatic access (planned)
---
## FUSE Search Interface
### Endpoint: `/.search/{query}/`
Browse search results as symlinks in a virtual directory.
### Happy Path
1. User navigates to `/.search/metallica/`
2. FUSE returns directory listing of symlinks
3. Each symlink points to absolute path: `/mnt/music/Metallica/Album/Track.flac`
4. User can open symlink directly in media player
**Example:**
```bash
$ ls -la /mnt/musicfs/.search/metallica/
001. Metallica - Enter Sandman.flac -> /mnt/musicfs/Metallica/Black Album/Enter Sandman.flac
002. Metallica - Battery.flac -> /mnt/musicfs/Metallica/Master of Puppets/Battery.flac
```
### Error Cases
| Scenario | Behavior | FUSE Error |
|----------|----------|------------|
| Empty query | Empty directory | (none) |
| No results | Empty directory | (none) |
| Query too long (>256 chars) | Truncated | (none) |
| Invalid UTF-8 in query | EINVAL | `libc::EINVAL` |
| Index corrupted | ENOENT | `libc::ENOENT` |
| Index writer shutdown | EIO | `libc::EIO` |
### Cache Behavior
- Results cached for 5 minutes (TTL)
- Maximum 1000 cached queries (LRU eviction)
- Cache miss triggers tantivy query
---
## gRPC Search API
> **Note:** gRPC API is planned for implementation. See architecture docs for design.
### `Search(SearchRequest) -> SearchResponse`
Single request/response search.
#### Request Schema
```protobuf
message SearchRequest {
string query = 1; // Required: tantivy query string
optional uint32 limit = 2; // Default: 100, max: 10000
optional uint32 offset = 3; // Default: 0, for pagination
optional string origin_id = 4; // Filter by origin (optional)
}
```
#### Response Schema
```protobuf
message SearchResponse {
repeated SearchResult results = 1;
uint64 total_matches = 2; // Approximate total
uint32 query_time_ms = 3; // Query execution time
}
message SearchResult {
int64 file_id = 1;
string virtual_path = 2;
optional string artist = 3;
optional string album = 4;
optional string title = 5;
float score = 6; // Relevance score
map<string, string> highlights = 7; // Matched fragments
}
```
### Error Cases
| Scenario | gRPC Status | Details |
|----------|-------------|---------|
| Empty query | `INVALID_ARGUMENT` | "Query cannot be empty" |
| Malformed query syntax | `INVALID_ARGUMENT` | tantivy parse error message |
| limit > 10000 | `INVALID_ARGUMENT` | "Limit exceeds maximum (10000)" |
| Index unavailable | `UNAVAILABLE` | "Search index not ready" |
| Index corrupted | `INTERNAL` | "Search index corrupted" |
| Timeout (>5s) | `DEADLINE_EXCEEDED` | Client-specified deadline |
---
## Query Syntax
MusicFS uses tantivy query syntax with custom fuzzy support.
### Supported Operators
| Operator | Example | Description |
|----------|---------|-------------|
| Term | `metallica` | Match in any default field |
| Field | `artist:metallica` | Match specific field |
| Phrase | `"enter sandman"` | Exact phrase match |
| Fuzzy | `metalica~1` | 1-character edit distance |
| Boolean | `metallica AND 1991` | Combine conditions |
| Range | `year:[1980 TO 1989]` | Numeric range |
### Searchable Fields
| Field | Type | Notes |
|-------|------|-------|
| `artist` | TEXT | Full-text searchable, default field |
| `album` | TEXT | Full-text searchable, default field |
| `album_artist` | TEXT | Full-text searchable, default field |
| `title` | TEXT | Full-text searchable, default field |
| `genre` | TEXT | Full-text searchable, default field |
| `composer` | TEXT | Full-text searchable, default field |
| `year` | u64 | Range queries only |
### Fuzzy Query Implementation
Fuzzy queries use the `term~N` syntax where N is the maximum edit distance (0-2).
When a fuzzy query is detected:
1. Query is parsed to extract term and distance
2. `FuzzyTermQuery` is created for each default field
3. Results are combined with `BooleanQuery` (OR semantics)
Example: `metalica~1` matches "Metallica" (edit distance 1).
---
## Performance
| Metric | Target | Notes |
|--------|--------|-------|
| Query latency (1M tracks) | <500ms | tantivy optimized |
| Index throughput | >1000 files/sec | Batch commits recommended |
| Memory per 1M tracks | <500MB | mmap-based index |
---
## Architecture
### Index Schema
```rust
pub struct SearchSchema {
file_id: Field, // INDEXED | STORED - for deletion
virtual_path: Field, // STORED - symlink target
artist: Field, // TEXT | STORED
album: Field, // TEXT | STORED
album_artist: Field, // TEXT | STORED
title: Field, // TEXT | STORED
genre: Field, // TEXT | STORED
composer: Field, // TEXT | STORED
year: Field, // INDEXED | STORED
duration_ms: Field, // STORED
bitrate: Field, // STORED
sample_rate: Field, // STORED
}
```
### Writer Pattern
Uses `Arc<RwLock<IndexWriter>>` per tantivy best practices:
- `add_document()` and `delete_term()` require READ lock
- `commit()` requires WRITE lock
- Single writer, multiple concurrent indexers
### Event Integration
The `Indexer` subscribes to `EventBus` for:
- `FileAdded` - Index new file via `MetadataLookup`
- `FileRemoved` - Remove from index by file_id
- `FileModified` - Update index entry
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_search_basic` | Unit | Basic search returns results |
| `test_search_fuzzy` | Unit | Typo tolerance (FR-14.3) |
| `test_search_genre` | Unit | Field-specific search |
| `test_index_persistence` | Unit | Index survives restart |
| `test_remove_file` | Unit | Deletion works correctly |
| `test_index_batch` | Unit | Batch indexing via Indexer |
| `test_search_ops_*` | Unit | FUSE SearchOps integration |
docs/api/smart-features.md
-315
View File
@@ -1,315 +0,0 @@
# Smart Features API Documentation
## Overview
MusicFS Week 9 introduces three intelligent features:
1. **Smart Collections** - Dynamic playlists based on queries, time ranges, and listening patterns
2. **Artwork Extraction & Caching** - Extract and serve album art in multiple sizes
3. **Predictive Prefetching** - Learn listening patterns to preload likely-next tracks
---
## Smart Collections
### CollectionStore
Manages persistent smart collections using SQLite.
```rust
pub struct CollectionStore {
db: rusqlite::Connection,
}
pub struct Collection {
pub id: i64,
pub name: String,
pub query: CollectionQuery,
pub created_at: SystemTime,
pub updated_at: SystemTime,
}
```
### CollectionQuery Types
| Query Type | Description | Example |
|------------|-------------|---------|
| `Match(String)` | tantivy search query | `"artist:Metallica"` |
| `DateRange { start, end }` | Files added within range | Last 30 days |
| `RecentlyAdded(days)` | Files added in last N days | `RecentlyAdded(7)` |
| `RecentlyPlayed(days)` | Files played in last N days | `RecentlyPlayed(30)` |
| `MostPlayed(limit)` | Top N most played tracks | `MostPlayed(100)` |
| `Genre(String)` | All tracks matching genre | `"Progressive Rock"` |
| `Compound(Vec)` | AND combination of queries | Multiple conditions |
### API
```rust
impl CollectionStore {
fn create(&self, name: &str, query: CollectionQuery) -> Result<i64, CollectionError>;
fn get(&self, id: i64) -> Result<Option<Collection>, CollectionError>;
fn list(&self) -> Result<Vec<Collection>, CollectionError>;
fn update(&self, id: i64, name: &str, query: CollectionQuery) -> Result<(), CollectionError>;
fn delete(&self, id: i64) -> Result<(), CollectionError>;
fn evaluate(&self, id: i64, index: &SearchIndex, patterns: &PatternStore) -> Result<Vec<FileId>, CollectionError>;
}
```
### FUSE Integration (Planned)
Collections will appear as virtual directories under `/.collections/`:
```bash
$ ls /mnt/musicfs/.collections/
Recent Additions/
Most Played/
80s Metal/
$ ls /mnt/musicfs/.collections/Most\ Played/
001. Track1.flac -> /mnt/musicfs/Artist/Album/Track1.flac
002. Track2.flac -> /mnt/musicfs/Artist/Album/Track2.flac
```
---
## Artwork Extraction & Caching
### ArtworkExtractor
Extracts embedded artwork from audio files.
```rust
pub struct Artwork {
pub data: Vec<u8>,
pub mime_type: String,
pub art_type: ArtType,
pub width: u32,
pub height: u32,
}
pub enum ArtType {
Front,
Back,
Other,
}
pub enum ArtSize {
Thumbnail, // 150x150 max
Medium, // 300x300 max
Full, // Original size
}
```
### API
```rust
impl ArtworkExtractor {
fn extract(&self, path: &Path) -> Result<Vec<Artwork>, ArtworkError>;
fn extract_first(&self, path: &Path) -> Result<Option<Artwork>, ArtworkError>;
fn resize(data: &[u8], size: ArtSize) -> Result<Vec<u8>, ArtworkError>;
}
```
### ArtworkCache
Caches artwork in CAS (Content-Addressable Storage).
```rust
impl ArtworkCache {
async fn store(&self, file_id: i64, artwork: &Artwork) -> Result<ChunkHash, ArtworkError>;
async fn get(&self, file_id: i64, art_type: &str, size: ArtSize) -> Result<Option<Vec<u8>>, ArtworkError>;
async fn has(&self, file_id: i64, art_type: &str) -> Result<bool, ArtworkError>;
}
```
### Size Specifications
| Size | Max Dimension | Use Case |
|------|---------------|----------|
| Thumbnail | 150px | List views, grids |
| Medium | 300px | Detail panels |
| Full | Original | High-res display |
### Caching Strategy
1. Original artwork stored in CAS with content hash
2. SQLite maps `(file_id, art_type)` → `chunk_hash`
3. Resizing performed on-demand, not cached (saves storage)
4. Max input size: 10MB (reject larger images)
---
## Predictive Prefetching
### Access Patterns (PatternStore)
Tracks file access history to predict next tracks.
```rust
pub struct AccessPattern {
pub file_id: FileId,
pub timestamp: SystemTime,
pub context: AccessContext,
pub hour_of_day: u8,
}
pub struct AccessContext {
pub album_id: Option<i64>,
pub track_number: Option<u32>,
pub artist: Option<String>,
}
```
### Pattern Learning
| Pattern Type | Description | Use Case |
|--------------|-------------|----------|
| Sequential | A → B → C transitions | Album playback |
| Time-based | Hour-of-day preferences | Morning playlist |
| Frequency | Most played tracks | Popular content |
### API
```rust
impl PatternStore {
fn record(&self, file_id: FileId, context: AccessContext) -> Result<(), PatternError>;
fn predict_next(&self, current: FileId, limit: usize) -> Vec<FileId>;
fn predict_for_time(&self, hour: u8, limit: usize) -> Vec<FileId>;
fn recently_played(&self, days: u32) -> Result<Vec<FileId>, PatternError>;
fn most_played(&self, limit: u32) -> Result<Vec<FileId>, PatternError>;
}
```
### PrefetchEngine
Background engine that listens for file access events and prefetches predicted content.
```rust
pub struct PrefetchConfig {
pub lookahead: usize, // How many tracks to prefetch (default: 3)
pub max_concurrent: usize, // Concurrent prefetch limit (default: 2)
pub cooldown: Duration, // Delay between prefetch bursts (default: 100ms)
pub enabled: bool, // Master switch
}
```
### Architecture
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ EventBus │────▶│ PrefetchEngine │────▶│ ContentFetcher │
│ (FileAccessed) │ │ (predictions) │ │ (CAS storage) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│
▼
┌─────────────────┐
│ PatternStore │
│ (SQLite DB) │
└─────────────────┘
```
### FUSE Interface
Virtual directory `/.prefetch/` exposes prefetch status and hints:
```bash
$ cat /mnt/musicfs/.prefetch/status
MusicFS Prefetch Status
=======================
running: true
in_flight: 2
most_played: [42, 57, 103, 89, 12]
$ ls /mnt/musicfs/.prefetch/
status
hint_0042
hint_0057
hint_0103
$ cat /mnt/musicfs/.prefetch/hint_0042
57
103
89
```
---
## Performance Targets
| Feature | Metric | Target |
|---------|--------|--------|
| Collection evaluation | Latency | <100ms for 100k files |
| Artwork extraction | Throughput | >10 files/sec |
| Artwork resize | Latency | <50ms per image |
| Pattern prediction | Latency | <10ms |
| Prefetch hit rate | Accuracy | >70% for sequential play |
---
## Error Handling
### CollectionError
| Error | Description |
|-------|-------------|
| `Database(rusqlite::Error)` | SQLite operation failed |
| `NotFound` | Collection ID doesn't exist |
| `InvalidQuery` | Query failed to serialize |
| `Search(SearchError)` | tantivy query failed |
| `Pattern(PatternError)` | Pattern lookup failed |
### ArtworkError
| Error | Description |
|-------|-------------|
| `Database(rusqlite::Error)` | Cache DB operation failed |
| `Cas(CasError)` | CAS storage operation failed |
| `InvalidHash` | Stored hash is malformed |
| `NotFound` | Artwork not in cache |
| `ImageTooLarge(usize)` | Input exceeds 10MB limit |
| `InvalidImage` | Cannot decode image data |
| `ResizeFailed` | Image resize operation failed |
### PatternError
| Error | Description |
|-------|-------------|
| `Database(rusqlite::Error)` | SQLite operation failed |
---
## Configuration
### Default Settings
```toml
[prefetch]
enabled = true
lookahead = 3
max_concurrent = 2
cooldown_ms = 100
[artwork]
max_input_size_mb = 10
thumbnail_size = 150
medium_size = 300
[patterns]
max_history_days = 30
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_collection_crud` | Unit | Create, read, update, delete |
| `test_collection_evaluate_match` | Unit | Match query evaluation |
| `test_collection_persistence` | Unit | Collections survive restart |
| `test_artwork_extract_flac` | Unit | FLAC artwork extraction |
| `test_artwork_cache_store_get` | Unit | Cache round-trip |
| `test_artwork_resize` | Unit | Resize produces valid output |
| `test_pattern_prediction` | Unit | Sequential pattern learning |
| `test_pattern_persistence` | Unit | Patterns survive restart |
| `test_prefetch_config_defaults` | Unit | Default config values |
| `test_prefetch_ops_*` | Unit | FUSE PrefetchOps integration |
docs/templates/bluedoc.md
Vendored
-96
View File
@@ -1,96 +0,0 @@
# [Project Name]: Design Doc
**Authors:** [Author Name(s)]
**Status:** [Draft / In-Review / Approved / Obsolete]
**Last Updated:** YYYY-MM-DD
**Reviewers:** [List of reviewers, usually @usernames]
**Approvers:** [List of final decision makers]
**Document Link:** [Link to this file or rendered version]
---
## 1. Abstract
A high-level summary (1–3 paragraphs) of what the project is, what problem it solves, and the proposed solution. This should be readable by a non-expert.
## 2. Background
Context for why this project exists.
- What is the current state?
- What are the pain points?
- Are there existing systems that this will replace or interact with?
- Include links to relevant PRDs (Product Requirement Documents) or previous design docs.
## 3. Goals & Non-Goals
Clarity on scope is critical to prevent scope creep.
### 3.1. Goals
* **Primary Goal:** The most important outcome.
* Metric-driven goals (e.g., "Reduce latency by 20%").
* Functional requirements (e.g., "Allow users to edit comments").
### 3.2. Non-Goals
* Features that might seem related but are explicitly out of scope.
* Future improvements that are deferred.
## 4. Proposed Design
The "meat" of the document. Start with the high-level architecture and zoom in.
### 4.1. High-Level Architecture
Provide a high-level diagram or description of how the system fits together.
> *Tip: Use Mermaid.js or link to an embedded image.*
### 4.2. Detailed Design
Go into specific components, APIs, and data models.
* **API Definitions:** Describe new endpoints, Protobuf definitions, or CLI commands.
* **Data Schema:** Database tables, key-value structures, or file formats.
* **Workflows:** Step-by-step logic for complex operations (e.g., auth flow).
## 5. Cross-Cutting Concerns
Google design docs place heavy emphasis on these "standard" reviews.
### 5.1. Security & Privacy
- How is data encrypted?
- What are the access control lists (ACLs)?
- Does this handle PII (Personally Identifiable Information)?
### 5.2. Observability (Monitoring & Logging)
- What metrics will be exported (e.g., RPC error rates, latency)?
- What logging is required for debugging?
- What are the "Golden Signals" for the dashboard?
### 5.3. Scalability & Performance
- What are the expected QPS (Queries Per Second)?
- How does the system scale (Horizontal vs. Vertical)?
- What are the resource requirements (CPU, RAM, Storage)?
### 5.4. Testing Plan
- Unit tests, integration tests, and end-to-end tests.
- Strategy for load testing or "chaos" testing.
## 6. Alternatives Considered
*A BlueDoc is not just about the chosen path, but why others were rejected.*
* **Alternative A:** Briefly describe it and why it was rejected (e.g., "Too complex," "High latency").
* **Alternative B:** Why "Doing Nothing" is not an option.
## 7. Implementation Plan
- **Phase 1:** Minimum Viable Product (MVP).
- **Phase 2:** Feature parity or migrations.
- **Rollout/Rollback:** How will the feature be toggled? (e.g., feature flags).
## 8. Glossary / References
- Links to external libraries.
- Definitions for project-specific acronyms.
---
### Markdown Style Tips (Google Conventions):
1. **Line Length:** Google's internal style guide suggests a soft limit of **80 characters** per line for source Markdown to make it easier to review in code-diff tools.
2. **Headings:** Use `#` for title, `##` for sections, and `###` for subsections.
3. **TOC:** If your environment supports it, use `[TOC]` at the top to generate a Table of Contents.
4. **Diagrams:** Use **Mermaid** blocks if using GitHub/GitLab, otherwise link to a stable SVG/PNG.
```mermaid
graph TD;
A-->B;
A-->C;
B-->D;
C-->D;
```
docs/templates/greendoc.md
Vendored
-56
View File
@@ -1,56 +0,0 @@
# [Project Name]: Design One-Pager (GreenDoc)
**Author(s):** [Name]
**Status:** [Draft / Approved / Shipped]
**Last Updated:** YYYY-MM-DD
**Estimated Effort:** [e.g., 2 weeks, 1 sprint]
---
## 1. Summary
A 2–3 sentence overview of the change. What are you doing and why?
## 2. Problem Statement
Describe the specific pain point or "broken" state this project addresses.
* *Example: Currently, users cannot filter their search history by date, leading to high latency in manual lookup.*
## 3. Proposed Solution
Explain the high-level logic of the fix or feature.
* What is the specific code change or configuration update?
* How does it interact with existing systems?
* *Note: Use a single simple diagram if the logic is non-trivial.*
## 4. Risks & Trade-offs
Even small changes have risks. Address them upfront.
* **Performance:** Will this increase memory usage?
* **Complexity:** Does this add a new dependency?
* **Backwards Compatibility:** Will this break existing clients?
* **Alternatives:** Why did you choose this over a "quicker" or "better" fix?
## 5. Success Criteria (Metrics)
How will you know this worked?
* [ ] Primary metric (e.g., "Feature usage > 5%")
* [ ] Guardrail metric (e.g., "Latency does not increase by > 10ms")
## 6. Implementation & Rollout
A brief bulleted list of the steps to ship.
1. Feature flag implementation.
2. Canary to 1% of users.
3. Full rollout.
---
### Comparison: BlueDoc vs. GreenDoc
| Feature | BlueDoc (Standard) | GreenDoc (One-Pager) |
| :--- | :--- | :--- |
| **Scope** | Major systems, new services. | Small features, optimizations, bug fixes. |
| **Length** | 5–20+ pages. | 1–2 pages. |
| **Review** | Cross-functional committees (SRE, Security). | Peer-level or Team Lead review. |
| **Focus** | Long-term scalability and architecture. | Immediate impact and implementation. |
### Markdown Tips for GreenDocs:
* **Be Brutally Concise:** If a section requires more than three paragraphs, consider upgrading to a **BlueDoc**.
* **Checklists:** Use `[ ]` to show remaining work or requirements.
* **Inline Links:** Link directly to the relevant code files or bug tracker (Jira/Buganizer) to keep the doc self-contained.
docs/v1/README.md
-118
View File
@@ -1,118 +0,0 @@
# beetfs - Reverse Engineered Documentation
> **Status**: Archived project (2010-2013), Python 2, fuse-python API
> **Fork**: git@github.com:LichHunter/beetfs.git
> **Original**: https://github.com/jbaiter/beetfs
## Overview
beetfs is a FUSE filesystem that presents audio files with **metadata from a database** while **passing through audio data unchanged** from original files. This enables transparent metadata modification without touching the underlying files.
### The Core Concept
```
┌─────────────────────────────────────────────────────────────────────┐
│ APPLICATION (VLC, Jellyfin, etc.) │
│ │
│ read("/mount/Artist/Album/track.flac") │
└─────────────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ beetfs (FUSE Layer) │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ FileHandler │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ if offset < header_boundary: │ │ │
│ │ │ return MODIFIED_HEADER (from beets database) │ │ │
│ │ │ else: │ │ │
│ │ │ return ORIGINAL_AUDIO (from real file on disk) │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
│ │
┌───────────┘ └───────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Beets Database │ │ Original File │
│ (SQLite - tags) │ │ (untouched) │
│ │ │ │
│ title: "Fixed" │ │ [FLAC header] │
│ artist: "Corr" │ │ [Audio frames] │
│ album: "Right" │ │ │
└───────────────────┘ └───────────────────┘
```
## Key Features
| Feature | Description |
|---------|-------------|
| **Metadata Overlay** | Returns tags from database, not from file |
| **Audio Passthrough** | Original audio data served unchanged |
| **Write Interception** | Tag edits saved to database, not to file |
| **Virtual Organization** | Presents files in template-based directory structure |
| **Format Support** | FLAC (full), MP3 (partial - read-only) |
## File Structure
```
beetfs/
├── beetsplug/
│ ├── __init__.py # Package initialization
│ └── beetFs.py # ALL code (~1144 lines)
├── README.rst # Original readme
└── COPYING # GPLv3 license
```
## Quick Architecture Summary
| Component | Lines | Purpose |
|-----------|-------|---------|
| `beetFs` (plugin) | 188-191 | Beets plugin hook |
| `mount()` | 119-183 | CLI entry point, builds virtual tree |
| `FSNode` | 390-436 | Virtual directory tree node |
| `FileHandler` | 439-565 | **CORE**: Metadata interpolation |
| `InterpolatedFLAC` | 274-388 | FLAC header generation |
| `InterpolatedID3` | 200-271 | ID3 tag generation (incomplete) |
| `beetFileSystem` | 622-1144 | FUSE operations implementation |
| `Stat` | 568-619 | File stat structure |
## Documentation Index
1. **[Architecture Overview](./architecture.md)** - System design and component interaction
2. **[Components Deep Dive](./components.md)** - Detailed component analysis
3. **[Data Flow](./data-flow.md)** - Read/write operation flows
4. **[Performance Analysis](./analysis.md)** - Latency, memory footprint, I/O patterns
5. **[Drawbacks & Limitations](./drawbacks.md)** - Known issues and missing features
6. **[Modernization Guide](./modernization.md)** - Notes for updating to Python 3
## Critical Issues Summary
| Issue | Severity | Impact |
|-------|----------|--------|
| Full file loaded into RAM | 🔴 Critical | OOM on large libraries |
| MP3 support disabled | 🔴 Critical | Only FLAC works |
| Python 2 only | 🔴 Critical | EOL, security risk |
| Single-threaded | 🟡 Major | Poor concurrency |
| 4 of 17 metadata fields | 🟡 Major | Limited functionality |
See [drawbacks.md](./drawbacks.md) for complete list (27 identified issues).
## Dependencies (Original)
```
beets >= 1.0
fuse-python (Python 2 FUSE bindings)
mutagen (audio metadata library)
```
## Usage (Original)
```bash
# As beets plugin
beet mount /path/to/mountpoint
```
## License
GPLv3 - See COPYING file
docs/v1/analysis.md
-263
View File
@@ -1,263 +0,0 @@
# beetfs Performance Analysis
## Executive Summary
beetfs has significant performance limitations due to its 2010-era design assumptions. The primary issues are **full file loading into RAM** and **blocking I/O on file open**.
---
## 1. Latency Analysis
### Operation Latencies
| Operation | Time Complexity | Typical Latency | Notes |
|-----------|-----------------|-----------------|-------|
| **File Open** | O(file_size) | 50ms - 1s+ | Reads entire file into memory |
| **File Read** | O(1) | <1ms | Pure memory slice |
| **File Write** | O(file_size) | 100ms - 2s+ | Reconstructs + DB write |
| **Directory List** | O(n) | <10ms | In-memory tree traversal |
| **getattr** | O(depth) | <1ms | Tree navigation + stat |
### File Open Breakdown
The file open operation is the critical bottleneck:
```
Time breakdown for opening 50MB FLAC file:
┌────────────────────────────────────────────────────────────┐
│ 1. open() syscall │ ~1ms │
│ 2. file_object.read() - load entire file │ ~100-200ms │
│ 3. InterpolatedFLAC() - parse FLAC │ ~20-50ms │
│ 4. Inject DB metadata │ ~1ms │
│ 5. get_header() - generate new header │ ~10-20ms │
│ 6. Seek to audio offset │ ~1ms │
│ 7. Read audio into music_data │ ~100-200ms │
├────────────────────────────────────────────────────────────┤
│ TOTAL │ ~230-470ms │
└────────────────────────────────────────────────────────────┘
```
**Code Evidence** (lines 461-483):
```python
# Step 2-5: Load and parse entire file
self.inf = InterpolatedFLAC(self.file_object.read()) # FULL FILE READ
self.inf["title"] = self.item.title
# ...
self.header = self.inf.get_header(self.real_path)
# Step 6-7: Cache all audio data
self.file_object.seek(self.music_offset)
self.music_data = self.file_object.read() # ANOTHER FULL READ
```
### Read Operation (Post-Open)
After file is opened, reads are fast:
```python
def read(self, size, offset):
if offset < self.bound:
return self.header[offset:offset+size] # Memory slice: O(1)
else:
return self.music_data[offset - len(self.header):...] # Memory slice: O(1)
```
### Write Operation
Writes to header area trigger expensive reconstruction:
```
Time breakdown for tag write:
┌────────────────────────────────────────────────────────────┐
│ 1. Reconstruct filedata in memory │ ~10-50ms │
│ 2. Parse as InterpolatedFLAC │ ~20-50ms │
│ 3. Extract tag values │ ~1ms │
│ 4. lib.store() + lib.save() (SQLite) │ ~10-50ms │
│ 5. Regenerate header │ ~10-20ms │
├────────────────────────────────────────────────────────────┤
│ TOTAL │ ~50-170ms │
└────────────────────────────────────────────────────────────┘
```
---
## 2. Memory Footprint
### Per-File Memory Usage
```
┌─────────────────────────────────────────────────────────────────────┐
│ FileHandler Memory Layout │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ self.music_data (bytes) │ │
│ │ Size: file_size - original_header_size │ │
│ │ Typical: 95-99% of file size │ │
│ │ Example: 48.5 MB for 50 MB file │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ self.header (bytes) │ │
│ │ Size: Generated FLAC header with DB metadata │ │
│ │ Typical: 4 KB - 64 KB (depends on metadata + padding) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ self.inf (InterpolatedFLAC) │ │
│ │ Size: Parsed metadata blocks + internal state │ │
│ │ Typical: 10 KB - 100 KB │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Other attributes │ │
│ │ path, real_path, item reference, format, etc. │ │
│ │ Typical: ~1 KB │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
├─────────────────────────────────────────────────────────────────────┤
│ TOTAL per file: ~1.0x - 1.1x original file size │
└─────────────────────────────────────────────────────────────────────┘
```
### Memory Scaling
| Scenario | Files Open | Avg File Size | RAM Usage |
|----------|------------|---------------|-----------|
| Single track playback | 1 | 30 MB | ~32 MB |
| Album playback (gapless) | 2-3 | 30 MB | ~65-100 MB |
| Album fully opened | 10 | 30 MB | ~320 MB |
| Jellyfin library scan | 50-100 | 30 MB | **1.6 - 3.2 GB** |
| Full library scan | 1000 | 30 MB | **32 GB** (OOM) |
### Global Memory
```python
# Directory tree structure
directory_structure = FSNode({}, {})
# Memory: O(number_of_items)
# Typical: 1-10 MB for libraries with 10,000-100,000 tracks
# Open file handles
self.files = {} # Dict[str, FileHandler]
# Memory: Sum of all FileHandler instances
# Unbounded - grows with concurrent opens
```
---
## 3. I/O Patterns
### Current (Inefficient)
```
File Open:
Disk → [Read ALL] → RAM (music_data)
→ RAM (inf object)
→ RAM (header)
File Read:
RAM (header or music_data) → Application
Total I/O: 1x-2x file size on open, 0 on read
```
### Optimal (Not Implemented)
```
File Open:
Disk → [Read header only] → RAM (small)
File Read:
If header region:
RAM (header) → Application
If audio region:
Disk → [Seek + Read chunk] → Application
Total I/O: ~64KB on open, on-demand reads
```
---
## 4. Concurrency
### Current Model
```python
server.multithreaded = 0 # Single-threaded
```
**Implications:**
- All FUSE operations serialized
- One slow file open blocks everything
- No benefit from multi-core CPUs
### Impact on Use Cases
| Use Case | Impact |
|----------|--------|
| Single player (VLC) | Acceptable - one file at a time |
| Media server scan | Severe - sequential processing |
| Multiple clients | Severe - requests queue up |
| Concurrent reads | Moderate - reads are fast once open |
---
## 5. Benchmarks (Theoretical)
Based on code analysis, not actual measurements:
### File Open Time vs Size
```
File Size Open Time (HDD) Open Time (SSD)
────────────────────────────────────────────────
10 MB 50-100 ms 20-50 ms
30 MB 150-300 ms 50-100 ms
50 MB 250-500 ms 100-200 ms
100 MB 500-1000 ms 200-400 ms
200 MB 1000-2000 ms 400-800 ms
```
### Memory vs Concurrent Opens
```
Open Files RAM Usage (30MB avg)
─────────────────────────────────────
1 ~32 MB
5 ~160 MB
10 ~320 MB
25 ~800 MB
50 ~1.6 GB
100 ~3.2 GB
```
---
## 6. Comparison with Alternatives
| Metric | beetfs | Direct File | NFS | FUSE passthrough |
|--------|--------|-------------|-----|------------------|
| Open latency | 200-500ms | <10ms | 10-50ms | <10ms |
| Read latency | <1ms | <1ms | 1-10ms | <1ms |
| Memory/file | ~1x size | ~0 | ~0 | ~0 |
| Metadata source | Database | File | File | File |
| Modify original | No | Yes | Yes | Yes |
---
## 7. Recommendations
### For Current Usage
1. **Limit concurrent opens** - Don't scan full library
2. **Use SSDs** - Reduces open latency by 2-3x
3. **Increase RAM** - Expect 1x file size per open
4. **Avoid large files** - 24-bit/192kHz FLACs are problematic
### For Modernization
1. **Implement lazy loading** - Read audio on demand
2. **Add file handle caching** - Keep headers, release audio
3. **Enable multi-threading** - Parallelize opens
4. **Add memory limits** - Evict old FileHandlers
docs/v1/benchmark-plan.md
-403
View File
@@ -1,403 +0,0 @@
# beetfs Benchmark Plan
## Executive Summary
Benchmark suite to measure beetfs FUSE filesystem performance across mount time, metadata operations, file I/O, and memory usage. Focus on realistic music library workloads.
## Critical Performance Findings (Pre-Benchmark)
### Architecture Bottlenecks Identified
| Bottleneck | Location | Impact |
|------------|----------|--------|
| **Full file load into RAM** | `FileHandler.__init__` line 481 | 50-100MB per open FLAC |
| **Mount-time bulk load** | `mount()` line 143 | O(N) for N library items |
| **GIL serialization** | Python 2.7 | Single-core limit for metadata ops |
| **Per-file DB lookup** | `getattr()`, `access()` | SQLite query per stat call |
### Expected Performance Characteristics
| Operation | Expected Performance | Bottleneck |
|-----------|---------------------|------------|
| Mount (10K items) | 5-30 seconds | `lib.items()` + FSNode construction |
| readdir | Fast (in-memory dict) | None |
| getattr (file) | Slow (~1ms) | DB lookup + real file stat |
| open (first) | Very slow | Full file read into RAM |
| read | Fast | Memory-to-memory copy |
| Memory (10 open files) | 500MB-1GB | FileHandler caches entire files |
---
## Benchmark Tools
### Primary Tools
| Tool | Purpose | Install |
|------|---------|---------|
| **fio** | I/O throughput, IOPS, latency | `nix-shell -p fio` |
| **mdtest** | Metadata operations (stat, readdir) | `nix-shell -p ior` |
| **hyperfine** | Mount time, command timing | `nix-shell -p hyperfine` |
| **time** | Basic timing | builtin |
| **/usr/bin/time -v** | Memory usage (maxrss) | builtin |
### Measurement Scripts
All benchmarks use synthetic FLAC files (5-10MB) to avoid I/O variance from real storage.
---
## Benchmark Categories
### 1. Mount Time Scaling
**Goal**: Measure how mount time scales with library size.
**Method**:
```bash
# Create libraries with N items: 100, 1K, 10K, 50K, 100K
hyperfine --warmup 1 --runs 5 \
'beet mount /mnt/beetfs && sleep 1 && fusermount -u /mnt/beetfs'
```
**Metrics**:
- Time to mount (seconds)
- Memory usage at mount completion (RSS)
**Expected scaling**: O(N) - linear with library size
**Test matrix**:
| Library Size | Expected Mount Time | Expected Memory |
|--------------|--------------------:|----------------:|
| 100 items | <1s | ~50MB |
| 1,000 items | 1-3s | ~60MB |
| 10,000 items | 5-15s | ~100MB |
| 50,000 items | 30-60s | ~300MB |
| 100,000 items | 60-120s | ~500MB |
---
### 2. Metadata Operations (stat/readdir)
**Goal**: Measure getattr and readdir performance - critical for music players that scan libraries.
#### 2a. Single stat latency
```bash
# Measure single stat call latency
hyperfine --warmup 10 --runs 100 \
'stat /mnt/beetfs/Artist/Album/01-Track.flac'
```
**Target**: <5ms average, <20ms p99
#### 2b. Bulk stat (library scan simulation)
```bash
# Stat all files in library
hyperfine --warmup 1 --runs 5 \
'find /mnt/beetfs -type f -exec stat {} + > /dev/null'
```
**Metrics**:
- Total time for N files
- stat operations per second
- p50, p95, p99 latency
**Target**: >500 stat/s (Python FUSE baseline)
#### 2c. Directory listing
```bash
# List directory with N entries
hyperfine --warmup 3 --runs 10 \
'ls /mnt/beetfs/Artist/Album/'
```
**Test matrix**:
| Directory entries | Target time |
|------------------:|------------:|
| 10 | <50ms |
| 100 | <100ms |
| 1,000 | <500ms |
---
### 3. File Open Performance
**Goal**: Measure file open latency - the critical bottleneck due to full file load.
#### 3a. First open (cold)
```bash
# Clear any caches, then open file
echo 3 > /proc/sys/vm/drop_caches
hyperfine --warmup 0 --runs 10 \
'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null'
```
**Test matrix**:
| File size | Expected open time |
|----------:|-------------------:|
| 5MB | 50-200ms |
| 20MB | 200-500ms |
| 50MB | 500ms-1s |
| 100MB | 1-2s |
#### 3b. Cached open (warm)
```bash
# File already opened once
hyperfine --warmup 5 --runs 50 \
'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null'
```
**Target**: <10ms (should hit FileHandler cache)
---
### 4. Read Throughput
**Goal**: Measure sequential and random read performance.
#### 4a. Sequential read
```bash
fio --name=seq_read \
--filename=/mnt/beetfs/Artist/Album/01-Track.flac \
--rw=read --bs=1M --direct=0 \
--ioengine=sync --numjobs=1 \
--runtime=30 --time_based
```
**Metrics**: MB/s throughput
**Target**: >100 MB/s (memory-backed after first read)
#### 4b. Random read (simulates seeking in audio player)
```bash
fio --name=rand_read \
--filename=/mnt/beetfs/Artist/Album/01-Track.flac \
--rw=randread --bs=64k --direct=0 \
--ioengine=sync --numjobs=1 \
--runtime=30 --time_based
```
**Metrics**: IOPS, latency histogram
---
### 5. Memory Usage
**Goal**: Measure memory consumption under load.
#### 5a. Idle memory (mounted, no activity)
```bash
# Mount and measure RSS
beet mount /mnt/beetfs &
sleep 5
ps -o rss= -p $(pgrep -f beetfs)
```
#### 5b. Memory per open file
```bash
# Open N files, measure memory growth
for i in 1 5 10 20; do
# Open $i files simultaneously
cat /mnt/beetfs/Artist/Album/0{1..$i}*.flac > /dev/null &
ps -o rss= -p $(pgrep -f beetfs)
done
```
**Expected**: ~file_size × open_files (FileHandler caches entire file)
#### 5c. Memory leak detection
```bash
# Repeatedly open/close files, check for memory growth
for i in {1..100}; do
cat /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null
done
# Compare RSS before and after
```
---
### 6. Concurrent Access
**Goal**: Measure performance under parallel access (multiple processes).
```bash
# Parallel stat operations
hyperfine --warmup 1 --runs 5 \
'seq 1 100 | xargs -P 4 -I {} stat /mnt/beetfs/Artist/Album/0{}-Track.flac'
```
**Metrics**:
- Throughput scaling with parallelism (1, 2, 4, 8 workers)
- Latency degradation
**Expected**: Limited scaling due to Python GIL
---
### 7. Realistic Workloads
#### 7a. Music player library scan
Simulates: Rhythmbox/Clementine scanning library at startup
```bash
# Recursive stat + readdir
time find /mnt/beetfs -type f -name "*.flac" -exec stat {} + | wc -l
```
#### 7b. Album playback
Simulates: Playing 12-track album sequentially
```bash
# Open each file, read 1MB (simulate buffering), close
for f in /mnt/beetfs/Artist/Album/*.flac; do
dd if="$f" of=/dev/null bs=1M count=1 2>/dev/null
done
```
#### 7c. Metadata edit
Simulates: Editing tags in Picard/Kid3
```bash
# Open file, write to header region, close
# (Requires write support to be functional)
```
---
## Baseline Comparisons
### Reference Filesystems
| Filesystem | Purpose |
|------------|---------|
| **ext4 (local)** | Best-case baseline |
| **fuse-passthrough** | FUSE overhead baseline |
| **sshfs** | Network FUSE comparison |
### Comparison Method
Run identical benchmarks on:
1. Real music files on ext4
2. Same files via FUSE passthrough
3. Same files via beetfs
Calculate overhead: `(beetfs_time - ext4_time) / ext4_time × 100%`
---
## Test Environment
### Hardware Requirements
- CPU: 4+ cores (to test GIL impact)
- RAM: 8+ GB (for large library tests)
- Storage: SSD recommended (reduces I/O variance)
### Software Requirements
```nix
# Add to flake.nix devShell
buildInputs = [
fio
hyperfine
# ior # includes mdtest
];
```
### Cache Control
```bash
# Clear all caches before cold benchmarks
sync
echo 3 > /proc/sys/vm/drop_caches
# Disable kernel FUSE caching for accurate measurements
mount -o entry_timeout=0,attr_timeout=0,negative_timeout=0
```
---
## Success Criteria
### Minimum Viable Performance
| Metric | Minimum | Target | Excellent |
|--------|--------:|-------:|----------:|
| Mount time (10K items) | <60s | <15s | <5s |
| stat latency (avg) | <20ms | <5ms | <1ms |
| stat throughput | >100/s | >500/s | >2000/s |
| File open (50MB, cold) | <5s | <1s | <200ms |
| Read throughput | >50 MB/s | >200 MB/s | >500 MB/s |
| Memory (idle, 10K items) | <500MB | <100MB | <50MB |
| Memory per open file | <2× file size | <1.5× | <1.1× |
### Regression Detection
Any benchmark result >20% worse than baseline triggers investigation.
---
## Implementation Notes
### Test Data Generation
Use existing test infrastructure from `tests/conftest.py`:
- `create_synthetic_flac()` - generates valid FLAC files
- `BeetFSTestCase` - creates isolated beets library
### Benchmark Script Structure
```
beetfs/
├── benchmarks/
│ ├── run_all.sh # Master script
│ ├── bench_mount.sh # Mount time tests
│ ├── bench_metadata.sh # stat/readdir tests
│ ├── bench_io.sh # Read/write throughput
│ ├── bench_memory.sh # Memory profiling
│ └── results/ # Output directory
│ ├── mount_scaling.csv
│ ├── stat_latency.csv
│ └── ...
```
### Output Format
```csv
# Example: mount_scaling.csv
library_size,mount_time_ms,memory_rss_kb,timestamp
100,450,52000,2024-01-15T10:30:00
1000,2100,61000,2024-01-15T10:31:00
10000,12500,98000,2024-01-15T10:33:00
```
---
## Known Limitations
1. **Python 2.7 GIL**: Cannot achieve true parallelism - expect flat scaling beyond 1 core
2. **FileHandler memory**: Each open file = full file in RAM - will OOM with many large files
3. **No lazy loading**: All library items loaded at mount - slow for large libraries
4. **SQLite single-writer**: Concurrent writes will serialize
## Optimization Opportunities (Post-Benchmark)
Based on benchmark results, consider:
1. **Lazy FSNode construction** - Build tree on first access, not mount
2. **Memory-mapped file access** - mmap instead of full read
3. **LRU cache for FileHandler** - Evict old files instead of holding all
4. **Metadata caching** - Cache getattr results, invalidate on DB change
5. **Batch DB queries** - Prefetch metadata for directory listings
docs/v1/benchmark-results.md
-101
View File
@@ -1,101 +0,0 @@
# beetfs Benchmark Results
**Date**: 2026-05-12
**Status**: ❌ ALL BENCHMARKS BLOCKED BY BUGS
## Executive Summary
Benchmarks cannot complete due to critical bugs in beetfs. The implementation is non-functional for any library with content.
## Results
| Benchmark | Status | Mean | Error |
|-----------|--------|------|-------|
| mount_time | ❌ FAIL | N/A | Directory tree building bug |
| readdir | ❌ FAIL | N/A | Directory tree building bug |
| stat_latency | ❌ FAIL | N/A | Directory tree building bug |
| enoent_lookup | ❌ FAIL | N/A | Directory tree building bug |
| file_open | ❌ FAIL | N/A | Directory tree building bug |
| read_throughput | ❌ FAIL | N/A | Directory tree building bug |
| memory_usage | ❌ FAIL | N/A | Directory tree building bug |
## Blocking Bugs
### Bug #1: Nested Methods (Lines 758-1144)
All FUSE operations (`readdir`, `open`, `read`, `write`, etc.) are indented inside the `access()` method, making them local functions instead of class methods.
**Impact**: Even if mount succeeds, all file operations return `ENOSYS (Function not implemented)`.
**Fix Required**: Dedent lines 758-1144 by 8 spaces.
### Bug #2: Directory Tree Building (Lines 403-414)
`FSNode.adddir()` calls `getnode()` which assumes parent directories already exist. When building the tree for a new library, parent directories haven't been created yet.
**Error**:
```
KeyError: u'Bench Artist'
File "beetFs.py", line 403, in getnode
return self.getnode(elements, root=root.dirs[topdir])
```
**Impact**: Mount crashes when library contains any tracks.
**Fix Required**: `adddir()` must create parent directories recursively before adding child.
### Bug #3: Empty Library Only
The only working configuration is mounting with an empty beets library:
- `test_mount_empty_library`: ✅ PASS
- Any library with tracks: ❌ CRASH
## Test Environment
- **Python**: 2.7.15
- **OS**: Linux (NixOS)
- **Test data**: 10 synthetic FLAC files (5 MB each)
- **Beets**: 1.4.9
## Benchmark Configuration
```python
num_tracks = 10
track_size_mb = 5
mount_runs = 3
stat_runs = 20
readdir_runs = 10
```
## Raw Results
See `benchmarks/results/benchmark_results.json` for full JSON output.
## Next Steps
1. **Fix Bug #2** (directory tree building) - allows mount with content
2. **Fix Bug #1** (nested methods) - allows FUSE operations to work
3. **Re-run benchmarks** - get actual performance numbers
## Conclusion
**beetfs is currently non-functional** for real-world use. Both bugs must be fixed before performance can be measured. The test infrastructure and benchmark suite are ready; only the implementation needs repair.
---
## Appendix: E2E Test Results (For Reference)
From the e2e test suite (74 tests):
| Category | Passed | Failed | Errors |
|----------|--------|--------|--------|
| Smoke tests | 4 | 3 | 0 |
| Nested bug detection | 3 (confirmed bug) | 10 | 0 |
| Readdir | 0 | 10 | 0 |
| Stat | 0 | 8 | 0 |
| Read | 0 | 11 | 0 |
| Write | 0 | 7 | 0 |
| Error handling | 0 | 7 | 3 |
| **Total** | **12** | **56** | **3** |
The 12 passing tests are infrastructure tests and tests that verify the bugs exist.
docs/v1/components.md
-550
View File
@@ -1,550 +0,0 @@
# beetfs Components Deep Dive
## Component Overview
```
┌─────────────────────────────────────────────────────────────────────────┐
│ beetFs.py │
│ ┌─────────────────────────────────────────────────────────────────────┐│
│ │ PLUGIN LAYER ││
│ │ beetFs (BeetsPlugin) beetFs_command (Subcommand) ││
│ │ mount() template_mapping() ││
│ └─────────────────────────────────────────────────────────────────────┘│
│ ┌─────────────────────────────────────────────────────────────────────┐│
│ │ VIRTUAL FILESYSTEM ││
│ │ FSNode beetFileSystem (fuse.Fuse) ││
│ │ Stat ││
│ └─────────────────────────────────────────────────────────────────────┘│
│ ┌─────────────────────────────────────────────────────────────────────┐│
│ │ METADATA INTERPOLATION ││
│ │ FileHandler InterpolatedFLAC ││
│ │ InterpolatedID3 ││
│ └─────────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────────────┘
```
---
## 1. Plugin Layer
### 1.1 beetFs (BeetsPlugin)
**Location**: Lines 188-191
```python
class beetFs(BeetsPlugin):
""" The beets plugin hook."""
def commands(self):
return [beetFs_command]
```
**Purpose**: Registers beetfs as a beets plugin, exposing the `mount` subcommand.
### 1.2 beetFs_command
**Location**: Lines 47, 185
```python
beetFs_command = Subcommand('mount', help='Mount a beets filesystem')
beetFs_command.func = mount
```
**Purpose**: CLI subcommand definition for `beet mount`.
### 1.3 mount() Function
**Location**: Lines 119-183
```python
def mount(lib, config, opts, args):
# 1. Validate arguments
if not args:
raise beets.ui.UserError('no mountpoint specified')
# 2. Parse path template
global structure_split
structure_split = PATH_FORMAT.split("/")
global structure_depth
structure_depth = len(structure_split)
# 3. Store library reference
global library
library = lib
# 4. Build virtual directory tree
global directory_structure
directory_structure = FSNode({}, {})
# 5. Iterate all library items
for item in lib.items():
mapping = template_mapping(lib, item)
# ... build tree ...
directory_structure.addfile(sub_elements, filename, item.id)
# 6. Create and run FUSE server
server = beetFileSystem(...)
server.main()
```
**Key Variables Set**:
| Variable | Type | Purpose |
|----------|------|---------|
| `structure_split` | `List[str]` | Path template components |
| `structure_depth` | `int` | Number of path levels |
| `library` | `Library` | Beets library reference |
| `directory_structure` | `FSNode` | Root of virtual tree |
### 1.4 template_mapping() Function
**Location**: Lines 82-116
```python
def template_mapping(lib, item):
"""Builds a template substitution map from beets item."""
mapping = {}
for key in METADATA_KEYS:
value = getattr(item, key)
# Sanitize value for filesystem paths
if isinstance(value, basestring):
value = re.sub(r'[\\/:]|^\.', '_', value)
elif key in ('track', 'tracktotal', 'disc', 'disctotal'):
value = '%02i' % value # Zero-pad numbers
mapping[key] = value
# Add format info
format_ = os.path.splitext(item.path)[1][1:]
mapping['format'] = format_
mapping['format_upper'] = format_.upper()
# Default values for missing fields
if mapping['artist'] == '':
mapping['artist'] = 'Unknown Artist'
# ... etc
return mapping
```
**Template Variables Available**:
| Variable | Source | Example |
|----------|--------|---------|
| `$artist` | `item.artist` | "Pink Floyd" |
| `$album` | `item.album` | "The Wall" |
| `$title` | `item.title` | "Comfortably Numb" |
| `$year` | `item.year` | "1979" |
| `$track` | `item.track` | "06" |
| `$format` | file extension | "flac" |
| `$format_upper` | file extension | "FLAC" |
---
## 2. Virtual Filesystem Layer
### 2.1 FSNode Class
**Location**: Lines 390-436
```python
class FSNode(object):
"""A directory node in the virtual filesystem tree."""
def __init__(self, dirs, files):
self.dirs = dirs # Dict[str, FSNode] - subdirectories
self.files = files # Dict[str, int] - filename → beets item ID
```
**Methods**:
| Method | Purpose | Signature |
|--------|---------|-----------|
| `getnode()` | Navigate to nested node | `getnode(elements, root=None) → FSNode` |
| `adddir()` | Add a directory | `adddir(elements, directory, root=None)` |
| `addfile()` | Add a file entry | `addfile(elements, filename, id, root=None)` |
| `listdir()` | List contents | `listdir(elements, directories, root=None) → List[str]` |
**Example Tree Navigation**:
```python
# Path: /Artist/Album/track.flac
# structure_split = ["$artist", "$album ($year) [$format_upper]", "$track - $artist - $title.$format"]
elements = ["Artist", "Album (2020) [FLAC]"]
node = directory_structure.getnode(elements)
# node.files = {"01 - Artist - Track.flac": 42, ...}
item_id = node.files["01 - Artist - Track.flac"]
# item_id = 42
```
### 2.2 Stat Class
**Location**: Lines 568-619
```python
class Stat(fuse.Stat):
DIRSIZE = 4096
def __init__(self, st_mode, st_size, st_nlink=1, st_uid=None, st_gid=None,
dt_atime=None, dt_mtime=None, dt_ctime=None):
self.st_mode = st_mode
self.st_ino = 0
self.st_dev = 0
self.st_nlink = st_nlink
self.st_uid = st_uid or os.getuid()
self.st_gid = st_gid or os.getgid()
self.st_size = st_size
# ... timestamps ...
```
**Purpose**: Represents file/directory metadata for FUSE stat operations.
### 2.3 beetFileSystem Class
**Location**: Lines 622-1144
```python
class beetFileSystem(fuse.Fuse):
"""Main FUSE filesystem implementation."""
def __init__(self, *args, **kwargs):
logging.basicConfig(filename="LOG", level=logging.INFO)
super(beetFileSystem, self).__init__(*args, **kwargs)
def fsinit(self):
"""Called after filesystem is mounted."""
self.lib = library
self.files = {} # Dict[path, FileHandler]
```
**FUSE Operations Implemented**:
| Operation | Lines | Purpose |
|-----------|-------|---------|
| `fsinit()` | 630-636 | Post-mount initialization |
| `fsdestroy()` | 638-639 | Pre-unmount cleanup |
| `statfs()` | 641-646 | Filesystem statistics |
| `getattr()` | 648-707 | Get file/dir attributes |
| `access()` | 723-756 | Check permissions |
| `readdir()` | 931-975 | List directory contents |
| `open()` | 988-1021 | Open file |
| `read()` | 1077-1106 | Read file data |
| `write()` | 1108-1135 | Write file data |
| `release()` | 1049-1059 | Close file |
**Not Implemented (return EOPNOTSUPP)**:
- `mknod()`, `mkdir()`, `unlink()`, `rmdir()`
- `symlink()`, `link()`, `rename()`
- `chmod()`, `chown()`, `truncate()`
---
## 3. Metadata Interpolation Layer
### 3.1 FileHandler Class
**Location**: Lines 439-565
This is the **core component** that implements metadata overlay.
```python
class FileHandler(object):
def __init__(self, path, lib):
self.path = path # Virtual path
self.lib = lib # Beets library
# Resolve virtual path to real file
pathsplit = path[1:].split('/')
self.item = self.lib.get_item(id=directory_structure
.getnode(pathsplit[0:structure_depth-1])
.files[pathsplit[structure_depth-1]])
self.real_path = self.item.path
# Open real file
self.file_object = open(self.real_path, 'r+')
self.instance_count = 1
# Determine format
self.format = os.path.splitext(path)[1][1:].lower()
if self.format == "flac":
# Load file into interpolated FLAC object
self.inf = InterpolatedFLAC(self.file_object.read())
# INJECT DATABASE METADATA
self.inf["title"] = self.item.title
self.inf["album"] = self.item.album
self.inf["artist"] = self.item.artist
self.inf["genre"] = self.item.genre
# Generate new header with DB metadata
self.header = self.inf.get_header(self.real_path)
self.bound = len(self.header)
self.music_offset = self.inf.offset()
elif self.format == "mp3":
self.bound = 0 # MP3 interpolation disabled
self.music_offset = 0
# Cache audio data
self.file_object.seek(self.music_offset)
self.music_data = self.file_object.read()
self.file_object.close()
```
**Key Attributes**:
| Attribute | Type | Purpose |
|-----------|------|---------|
| `path` | `str` | Virtual path (e.g., `/Artist/Album/track.flac`) |
| `real_path` | `str` | Actual file path on disk |
| `item` | `Item` | Beets library item (has DB metadata) |
| `format` | `str` | File format ("flac", "mp3") |
| `inf` | `InterpolatedFLAC` | Mutagen object with injected metadata |
| `header` | `bytes` | Generated header with DB tags |
| `bound` | `int` | Byte offset where header ends |
| `music_offset` | `int` | Byte offset in original file where audio starts |
| `music_data` | `bytes` | Cached audio data |
| `instance_count` | `int` | Reference count for file handles |
### 3.2 FileHandler.read() Method
**Location**: Lines 497-517
```python
def read(self, size, offset):
# Case 1: Reading within header boundary
if offset < self.bound:
if offset + size < len(self.header):
# Entire read is within header
return self.header[offset:offset+size]
else:
# Read spans header and audio
ret = self.header[offset:len(self.header)]
ret = ret + self.music_data[0:size - (len(self.header) - offset)]
return ret
# Case 2: Reading audio data only
return self.music_data[offset - len(self.header):offset - len(self.header) + size]
```
**Read Logic Diagram**:
```
Virtual File Layout:
┌────────────────────────────────────────────────────────────────┐
│ 0 bound EOF │
│ ├─────────┼────────────────────────────────────────────────┤ │
│ │ HEADER │ AUDIO DATA │ │
│ │ (from │ (from self.music_data) │ │
│ │ self. │ │ │
│ │ header) │ │ │
│ └─────────┴────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
Read scenarios:
1. offset=0, size=100, bound=500 → Return header[0:100]
2. offset=400, size=200, bound=500 → Return header[400:500] + music[0:100]
3. offset=600, size=100, bound=500 → Return music[100:200]
```
### 3.3 FileHandler.write() Method
**Location**: Lines 519-565
```python
def write(self, offset, buf):
# Only handle writes to header area
if offset < self.bound:
# Reconstruct full file in memory
filedata = self.header + self.music_data
# Patch in new data
filedata = filedata[0:offset] + buf + filedata[offset + len(buf):]
if self.format == "flac":
# Parse the patched data
self.inf = InterpolatedFLAC(filedata)
# EXTRACT new tag values and save to DB
self.item.title = str(self.inf["title"][0]).encode('utf-8')
self.item.album = str(self.inf["album"][0]).encode('utf-8')
self.item.artist = str(self.inf["artist"][0]).encode('utf-8')
self.item.genre = str(self.inf["genre"][0]).encode('utf-8')
# Persist to beets database
self.lib.store(self.item)
self.lib.save()
# Regenerate header with updated values
self.inf["title"] = self.item.title
self.inf["album"] = self.item.album
self.inf["artist"] = self.item.artist
self.inf["genre"] = self.item.genre
self.header = self.inf.get_header(self.real_path)
self.bound = len(self.header)
return len(buf)
```
**Write Flow**:
```
1. App writes new tag data to header region
│
▼
2. Patch header + music_data with new bytes
│
▼
3. Parse patched data as FLAC
│
▼
4. Extract tag values from parsed FLAC
│
▼
5. Update beets Item with new values
│
▼
6. lib.store(item) + lib.save() → SQLite
│
▼
7. Regenerate header for subsequent reads
```
### 3.4 InterpolatedFLAC Class
**Location**: Lines 274-388
```python
class InterpolatedFLAC(FLAC):
"""Custom FLAC handler that can load from bytes and generate headers."""
def load(self, filedata):
"""Load FLAC from byte string instead of file."""
self.metadata_blocks = []
self.tags = None
self.filedata = filedata
self.fileobj = BytesIO(filedata)
self.__check_header(self.fileobj)
while self.__read_metadata_block(self.fileobj):
pass
# Verify audio frame starts correctly
if self.fileobj.read(2) not in ["\xff\xf8", "\xff\xf9"]:
raise FLACNoHeaderError("End of metadata did not start audio")
def get_header(self, filename=None):
"""Generate FLAC header with current metadata."""
# Add padding block
self.metadata_blocks.append(Padding('\x00' * 1020))
MetadataBlock.group_padding(self.metadata_blocks)
# Calculate available space
header = self.__check_header(self.fileobj)
available = self.__find_audio_offset(self.fileobj) - header
data = MetadataBlock.writeblocks(self.metadata_blocks)
# Adjust padding to match available space
if len(data) > available:
# Reduce padding
padding = self.metadata_blocks[-1]
padding.length -= (len(data) - available)
data = MetadataBlock.writeblocks(self.metadata_blocks)
elif len(data) < available:
# Increase padding
self.metadata_blocks[-1].length += (available - len(data))
data = MetadataBlock.writeblocks(self.metadata_blocks)
self.__offset = len("fLaC" + data)
return "fLaC" + data
def offset(self):
"""Return byte offset where audio data starts."""
return self.__offset
```
**FLAC Structure**:
```
┌──────────────────────────────────────────────────────────────────┐
│ "fLaC" │ STREAMINFO │ VORBIS_COMMENT │ ... │ PADDING │ AUDIO... │
│ (4B) │ block │ block │ │ block │ │
└──────────────────────────────────────────────────────────────────┘
│◄──────── metadata_blocks ─────────►│
│ │
└──── get_header() returns this ─────┘
```
### 3.5 InterpolatedID3 Class
**Location**: Lines 200-271
```python
class InterpolatedID3(ID3):
"""Custom ID3 handler for MP3 files."""
def save(self, filename=None, v1=0):
"""Save ID3 tags to file."""
# Sort frames by importance
order = ["TIT2", "TPE1", "TRCK", "TALB", "TPOS", "TDRC", "TCON"]
# ... write header ...
```
**Note**: MP3 support is **incomplete** in the current implementation. The `FileHandler.__init__` sets `self.bound = 0` for MP3, effectively disabling interpolation.
---
## 4. Supported Metadata Fields
**Location**: Lines 55-77
```python
METADATA_RW_FIELDS = [
('title', 'text'),
('artist', 'text'),
('album', 'text'),
('genre', 'text'),
('composer', 'text'),
('grouping', 'text'),
('year', 'int'),
('month', 'int'),
('day', 'int'),
('track', 'int'),
('tracktotal', 'int'),
('disc', 'int'),
('disctotal', 'int'),
('lyrics', 'text'),
('comments', 'text'),
('bpm', 'int'),
('comp', 'bool'),
]
```
**Actually Implemented** (in FileHandler):
| Field | Read | Write |
|-------|------|-------|
| `title` | ✅ | ✅ |
| `artist` | ✅ | ✅ |
| `album` | ✅ | ✅ |
| `genre` | ✅ | ✅ |
| Others | ❌ | ❌ |
---
## 5. Error Handling
**Error Codes Used**:
| Code | Constant | Usage |
|------|----------|-------|
| 2 | `ENOENT` | File/directory not found |
| 13 | `EACCES` | Permission denied |
| 1 | `EPERM` | Operation not permitted |
| 95 | `EOPNOTSUPP` | Operation not supported |
**Exception Handling Pattern**:
```python
def getattr(self, path):
try:
# ... logic ...
except Exception as e:
logging.error(e)
return -errno.ENOENT
```
docs/v1/data-flow.md
-412
View File
@@ -1,412 +0,0 @@
# beetfs Data Flow
## Overview
This document details the complete data flow for read and write operations in beetfs.
---
## 1. Initialization Flow
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ beet mount /mountpoint │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ mount(lib, config, opts, args) │
│ │
│ 1. Parse PATH_FORMAT into structure_split │
│ PATH_FORMAT = "$artist/$album ($year) [$format_upper]/..." │
│ structure_split = ["$artist", "$album ($year) [$format_upper]", ...] │
│ structure_depth = 3 │
│ │
│ 2. Store global library reference │
│ library = lib │
│ │
│ 3. Create empty virtual directory tree │
│ directory_structure = FSNode({}, {}) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ for item in lib.items(): │
│ │
│ For each item in beets library: │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ 1. Build template mapping │ │
│ │ mapping = { │ │
│ │ 'artist': 'Pink Floyd', │ │
│ │ 'album': 'The Wall', │ │
│ │ 'year': '1979', │ │
│ │ 'format_upper': 'FLAC', │ │
│ │ 'track': '01', │ │
│ │ 'title': 'In The Flesh?', │ │
│ │ } │ │
│ │ │ │
│ │ 2. Substitute template for each level │ │
│ │ level_subbed[0] = "Pink Floyd" │ │
│ │ level_subbed[1] = "The Wall (1979) [FLAC]" │ │
│ │ level_subbed[2] = "01 - Pink Floyd - In The Flesh?.flac" │ │
│ │ │ │
│ │ 3. Add directories to tree │ │
│ │ directory_structure.adddir([], "Pink Floyd") │ │
│ │ directory_structure.adddir(["Pink Floyd"], "The Wall (1979)...") │ │
│ │ │ │
│ │ 4. Add file entry (filename → item.id) │ │
│ │ directory_structure.addfile( │ │
│ │ ["Pink Floyd", "The Wall (1979) [FLAC]"], │ │
│ │ "01 - Pink Floyd - In The Flesh?.flac", │ │
│ │ item.id # e.g., 42 │ │
│ │ ) │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem FUSE Server │
│ │
│ server = beetFileSystem(...) │
│ server.multithreaded = 0 │
│ server.main() ← Enters FUSE event loop │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 2. File Open Flow
```
Application: open("/mount/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The Flesh?.flac")
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem.open(path, flags) │
│ Lines 988-1021 │
│ │
│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The..." │
│ flags = os.O_RDONLY (or O_RDWR) │
│ │
│ if path in self.files: │
│ # File already open - increment reference count │
│ self.files[path].open() │
│ return self.files[path] │
│ else: │
│ # Create new FileHandler │
│ self.files[path] = FileHandler(path, self.lib) │
│ return self.files[path] │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ FileHandler.__init__(path, lib) │
│ Lines 440-483 │
│ │
│ Step 1: Resolve virtual path to beets item │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ pathsplit = ["Pink Floyd", "The Wall (1979) [FLAC]", │ │
│ │ "01 - Pink Floyd - In The Flesh?.flac"] │ │
│ │ │ │
│ │ # Navigate to parent directory in virtual tree │ │
│ │ node = directory_structure.getnode(pathsplit[0:2]) │ │
│ │ # node.files = {"01 - Pink Floyd - In The Flesh?.flac": 42, ...} │ │
│ │ │ │
│ │ # Get beets item by ID │ │
│ │ item_id = node.files[pathsplit[2]] # 42 │ │
│ │ self.item = lib.get_item(id=42) │ │
│ │ self.real_path = self.item.path │ │
│ │ # e.g., "/mnt/music/torrents/pink_floyd_wall.flac" │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 2: Open real file and detect format │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.file_object = open(self.real_path, 'r+') │ │
│ │ self.format = "flac" # from file extension │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 3: Create InterpolatedFLAC with database metadata │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.inf = InterpolatedFLAC(self.file_object.read()) │ │
│ │ │ │
│ │ # INJECT DATABASE METADATA (this is the key operation!) │ │
│ │ self.inf["title"] = self.item.title # "In The Flesh?" │ │
│ │ self.inf["album"] = self.item.album # "The Wall" │ │
│ │ self.inf["artist"] = self.item.artist # "Pink Floyd" │ │
│ │ self.inf["genre"] = self.item.genre # "Progressive Rock" │ │
│ │ │ │
│ │ # Generate header with injected metadata │ │
│ │ self.header = self.inf.get_header(self.real_path) │ │
│ │ self.bound = len(self.header) # e.g., 8192 bytes │ │
│ │ self.music_offset = self.inf.offset() │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 4: Cache audio data │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.file_object.seek(self.music_offset) │ │
│ │ self.music_data = self.file_object.read() # All audio data │ │
│ │ self.file_object.close() │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 3. File Read Flow
```
Application: read(fd, buffer, 4096) # offset managed by kernel
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem.read(path, size, offset, fh) │
│ Lines 1077-1106 │
│ │
│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - ..." │
│ size = 4096 │
│ offset = 0 (first read) or previous offset + bytes_read │
│ fh = FileHandler instance │
│ │
│ return self.files[path].read(size, offset) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ FileHandler.read(size, offset) │
│ Lines 497-517 │
│ │
│ Variables: │
│ self.bound = 8192 (header size) │
│ self.header = bytes (generated FLAC header with DB metadata) │
│ self.music_data = bytes (original audio frames) │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────┼───────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Case 1: Header Only │ │ Case 2: Span Both │ │ Case 3: Audio Only │
│ offset < bound │ │ offset < bound │ │ offset >= bound │
│ offset+size < bound │ │ offset+size >= bound│ │ │
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
│ Example: │ │ Example: │ │ Example: │
│ offset=0 │ │ offset=8000 │ │ offset=10000 │
│ size=4096 │ │ size=4096 │ │ size=4096 │
│ bound=8192 │ │ bound=8192 │ │ bound=8192 │
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
│ Return: │ │ Return: │ │ Return: │
│ header[0:4096] │ │ header[8000:8192] │ │ music_data[ │
│ │ │ + music_data[0:3904]│ │ 1808:5904] │
│ (DB metadata!) │ │ │ │ │
│ │ │ (mixed) │ │ (original audio) │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
Visual representation of virtual file:
0 bound (8192) EOF
│ │ │
▼ ▼ ▼
┌───────────────────────┬────────────────────────────────────────────┐
│ HEADER │ AUDIO DATA │
│ (self.header) │ (self.music_data) │
│ │ │
│ Contains: │ Contains: │
│ - "fLaC" magic │ - Original FLAC frames │
│ - STREAMINFO block │ - Unchanged from disk │
│ - VORBIS_COMMENT │ │
│ with DB values: │ │
│ title, artist, │ │
│ album, genre │ │
│ - PADDING block │ │
└───────────────────────┴────────────────────────────────────────────┘
▲ ▲
│ │
From InterpolatedFLAC From original file
with injected DB tags (passed through)
```
---
## 4. File Write Flow
```
Application: write(fd, "TITLE=New Title\0", 16) # Hypothetical tag edit
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem.write(path, buf, offset, fh) │
│ Lines 1108-1135 │
│ │
│ return self.files[path].write(offset, buf) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ FileHandler.write(offset, buf) │
│ Lines 519-565 │
│ │
│ if offset >= self.bound: │
│ # Write is in audio area - DISCARD │
│ return # Do nothing, audio is read-only │
│ │
│ # Write is in header area - process tag update │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Step 1: Reconstruct full virtual file in memory │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ filedata = self.header + self.music_data │ │
│ │ │ │
│ │ # Patch in new data │ │
│ │ filedata = filedata[0:offset] + buf + filedata[offset + len(buf):] │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 2: Parse patched data as FLAC │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.inf = InterpolatedFLAC(filedata) │ │
│ │ # This parses the FLAC structure and extracts Vorbis comments │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 3: Extract tag values from parsed FLAC │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.item.title = str(self.inf["title"][0]).encode('utf-8') │ │
│ │ self.item.album = str(self.inf["album"][0]).encode('utf-8') │ │
│ │ self.item.artist = str(self.inf["artist"][0]).encode('utf-8') │ │
│ │ self.item.genre = str(self.inf["genre"][0]).encode('utf-8') │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 4: Save to beets database │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.lib.store(self.item) # Update item in library │ │
│ │ self.lib.save() # Persist to SQLite │ │
│ │ │ │
│ │ # NOTE: Original file on disk is NEVER touched! │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ Step 5: Regenerate header for subsequent reads │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ self.inf["title"] = self.item.title │ │
│ │ self.inf["album"] = self.item.album │ │
│ │ self.inf["artist"] = self.item.artist │ │
│ │ self.inf["genre"] = self.item.genre │ │
│ │ │ │
│ │ self.header = self.inf.get_header(self.real_path) │ │
│ │ self.bound = len(self.header) │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
│ │
│ return len(buf) # Success │
└─────────────────────────────────────────────────────────────────────────────┘
Write data flow summary:
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Application │ │ beetfs │ │ Beets │ │ Original │
│ writes │────▶│ parses │────▶│ database │ │ file │
│ new tags │ │ extracts │ │ updated │ │ UNTOUCHED │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
```
---
## 5. File Release Flow
```
Application: close(fd)
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem.release(path, flags, fh) │
│ Lines 1049-1059 │
│ │
│ if self.files[path].release(): │
│ # Reference count reached 0, clean up │
│ del self.files[path] │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ FileHandler.release() │
│ Lines 489-495 │
│ │
│ self.instance_count -= 1 │
│ │
│ if self.instance_count == 0: │
│ return True # OK to delete │
│ else: │
│ return False # Still in use │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 6. Directory Listing Flow
```
Application: ls /mount/Pink\ Floyd/
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ beetFileSystem.readdir(path, offset, dh) │
│ Lines 931-975 │
│ │
│ path = "/Pink Floyd" │
│ pathsplit = ["Pink Floyd"] │
│ │
│ yield fuse.Direntry(".") │
│ yield fuse.Direntry("..") │
│ │
│ # len(pathsplit) == 1, structure_depth - 1 == 2 │
│ # So we're listing directories (albums), not files │
│ │
│ for dirname in directory_structure.listdir(pathsplit, True): │
│ yield fuse.Direntry(dirname.encode('utf-8')) │
│ # "The Wall (1979) [FLAC]" │
│ # "Animals (1977) [FLAC]" │
│ # etc. │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## 7. Complete Request Lifecycle
```
┌──────────────────────────────────────────────────────────────────────────────┐
│ COMPLETE LIFECYCLE │
│ │
│ 1. User mounts: beet mount /mnt/music │
│ ├─ Build virtual tree from beets library │
│ └─ Start FUSE event loop │
│ │
│ 2. Application opens file: open("/mnt/music/Artist/Album/track.flac") │
│ ├─ Resolve virtual path to beets item ID │
│ ├─ Load original file into memory │
│ ├─ Inject database metadata into FLAC structure │
│ ├─ Generate new header with DB tags │
│ └─ Cache audio data │
│ │
│ 3. Application reads file: read(fd, buf, 4096) │
│ ├─ If reading header region → return header (DB metadata) │
│ ├─ If reading audio region → return cached audio (original) │
│ └─ If spanning both → return combined data │
│ │
│ 4. Application writes tags: write(fd, new_tags, offset) │
│ ├─ If audio region → discard (read-only) │
│ ├─ If header region: │
│ │ ├─ Parse new tag values │
│ │ ├─ Update beets database │
│ │ └─ Regenerate header │
│ └─ Original file NEVER modified │
│ │
│ 5. Application closes file: close(fd) │
│ ├─ Decrement reference count │
│ └─ Clean up if count == 0 │
│ │
│ 6. User unmounts: fusermount -u /mnt/music │
│ └─ fsdestroy() called, cleanup │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
```
docs/v1/drawbacks.md
-479
View File
@@ -1,479 +0,0 @@
# beetfs Drawbacks & Limitations
## Overview
This document catalogs all identified issues, limitations, and missing features in beetfs. Issues are categorized by severity and type.
---
## Critical Issues (🔴)
### 1. Full File Loading into Memory
**Location**: Lines 463, 480-481
```python
self.inf = InterpolatedFLAC(self.file_object.read()) # Entire file
# ...
self.music_data = self.file_object.read() # Audio portion again
```
**Impact**:
- Memory usage = O(file_size) per open file
- 50MB FLAC = ~50MB RAM
- Library scan of 100 files = 5GB+ RAM
- Out-of-memory crashes on large libraries
**Fix Required**: Implement lazy loading with seek-based reads.
---
### 2. MP3 Support Disabled
**Location**: Lines 475-477
```python
elif self.format == "mp3":
self.bound = 0 # disable interpolation for now
self.music_offset = 0 # disable interpolation for now
```
**Impact**:
- MP3 files return original metadata, not database metadata
- Breaks the core promise of metadata overlay
- MP3 is still one of the most common formats
**Fix Required**: Implement `InterpolatedID3` header generation.
---
### 3. Python 2 Only
**Location**: Throughout
```python
except fuse.FuseError, e: # Python 2 syntax
if isinstance(value, basestring): # Removed in Python 3
return reduce(lambda a, b: (a << 8) + ord(b), string, 0L) # Long literals
```
**Impact**:
- Python 2 EOL was January 2020
- Security vulnerabilities unfixed
- No modern library support
- Cannot run on Python 3 without migration
**Fix Required**: Full Python 3 migration (see modernization.md).
---
### 4. Deprecated FUSE Library
**Location**: Line 25, 51
```python
import fuse
fuse.fuse_python_api = (0, 2)
```
**Impact**:
- fuse-python is unmaintained
- Missing modern FUSE features (FUSE 3.x)
- Compatibility issues with recent kernels
- No async support
**Fix Required**: Migrate to pyfuse3 or llfuse.
---
### 5. Single-Threaded Execution
**Location**: Line 178
```python
server.multithreaded = 0
```
**Impact**:
- All operations serialized
- One slow open blocks all other operations
- Cannot utilize multiple CPU cores
- Poor performance under concurrent access
**Fix Required**: Enable multithreading with proper locking.
---
## Major Issues (🟡)
### 6. Limited Metadata Fields
**Location**: Lines 466-469, 540-547
```python
# Only these 4 fields are actually used:
self.inf["title"] = self.item.title
self.inf["album"] = self.item.album
self.inf["artist"] = self.item.artist
self.inf["genre"] = self.item.genre
```
**Defined but not implemented** (lines 55-77):
- `composer`, `grouping`
- `year`, `month`, `day`
- `track`, `tracktotal`
- `disc`, `disctotal`
- `lyrics`, `comments`
- `bpm`, `comp`
- `albumartist` (not even defined)
**Impact**:
- Track numbers not from database
- Album artist not supported
- Year/date not interpolated
- Cover art not handled
---
### 7. No File Handle Caching/Eviction
**Location**: Lines 1004-1018
```python
if path in self.files:
self.files[path].open()
else:
self.files[path] = FileHandler(path, self.lib)
```
**Missing**:
- No maximum cache size
- No LRU eviction
- No memory pressure handling
- Files stay in memory until explicitly closed
**Impact**:
- Memory grows unbounded
- No protection against OOM
- Applications that open-then-close still leave data cached
---
### 8. Blocking Database Operations
**Location**: Lines 549-550
```python
self.lib.store(self.item)
self.lib.save()
```
**Impact**:
- SQLite operations in FUSE thread
- Write operations block all reads
- No transaction batching
- Potential deadlocks with beets
---
### 9. No Library Hot Reload
**Issue**: Virtual directory tree built once at mount time.
**Location**: Lines 142-172
```python
for item in lib.items():
# Build tree...
```
**Impact**:
- New files added to beets library not visible
- Deleted files still appear (ENOENT on access)
- Metadata changes in beets not reflected until remount
- Must unmount/remount to see changes
---
### 10. Static Path Format
**Location**: Lines 44-45
```python
PATH_FORMAT = ("$artist/$album ($year) [$format_upper]/"
"$track - $artist - $title.$format")
```
**Impact**:
- Cannot customize organization
- Hard-coded template
- No configuration option
- Incompatible with different organizational preferences
---
### 11. No Extended Attribute Support
**Location**: Not implemented
**Impact**:
- Cannot store/retrieve xattrs
- Some applications use xattrs for metadata
- macOS Finder metadata lost
- Linux capabilities not supported
---
### 12. No Symlink Support
**Location**: Lines 758-765
```python
def readlink(self, path):
return -errno.EOPNOTSUPP
```
**Impact**:
- Cannot create symlinks in mount
- Some applications expect symlink support
- Cannot link to external files
---
### 13. Silent Error Swallowing
**Location**: Lines 705-707, 1019-1021, 1103-1104
```python
except Exception as e:
logging.error(e)
return -errno.ENOENT # Always returns same error
```
**Impact**:
- All errors appear as "file not found"
- Hard to debug issues
- No distinction between permission, I/O, parse errors
- Lost stack traces in many cases
---
## Minor Issues (🟢)
### 14. Global State
**Location**: Lines 125-140
```python
global structure_split
global structure_depth
global library
global directory_structure
```
**Impact**:
- Cannot mount multiple instances
- Difficult to unit test
- Tight coupling between components
- No dependency injection
---
### 15. Hard-coded Log File
**Location**: Lines 624-625
```python
LOG_FILENAME = "LOG"
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO,)
```
**Impact**:
- Log file created in current directory
- No log rotation
- No configurable log level
- Fills disk on busy systems
---
### 16. Reference Count Manual Management
**Location**: Lines 485-495
```python
def open(self):
self.instance_count = self.instance_count + 1
def release(self):
if self.instance_count > 0:
self.instance_count = self.instance_count - 1
```
**Issues**:
- Race conditions possible if multithreaded
- No context manager support
- Manual counting error-prone
- Off-by-one potential
---
### 17. Inefficient Directory Building
**Location**: Lines 153-172
```python
for level in range(0, structure_depth - 1):
if level-1 in level_subbed:
sub_elements.append(level_subbed[level-1])
directory_structure.adddir(sub_elements, level_subbed[level])
```
**Issues**:
- Rebuilds path for every item
- O(items × depth) complexity
- String allocations in inner loop
- Could use trie-based insertion
---
### 18. No Cover Art Handling
**Issue**: Cover art embedded in FLAC not addressed.
**Impact**:
- Cover art from original file used, not database
- Cannot replace/add cover art through overlay
- PICTURE metadata blocks passed through unchanged
---
### 19. No Cue Sheet Support
**Issue**: Cue sheets not handled specially.
**Impact**:
- `.cue` files point to original file paths
- Cannot play cue-referenced tracks correctly
- Split-by-cue not supported
---
### 20. File Size Mismatch Potential
**Issue**: Virtual file size differs from physical if header size changes.
**Location**: Lines 675-688
```python
statinfo = os.stat(item)
st = Stat(st_mode=statinfo.st_mode,
st_size=statinfo.st_size, # Original size, not virtual!
...)
```
**Impact**:
- `stat()` returns original file size
- If generated header is larger/smaller, size is wrong
- Some applications may fail on size mismatch
- Range requests could break
---
## Missing Features
### Essential
| Feature | Status | Notes |
|---------|--------|-------|
| MP3 metadata interpolation | ❌ Disabled | Code exists but disabled |
| OGG/Opus support | ❌ Missing | No implementation |
| AAC/M4A support | ❌ Missing | No implementation |
| Lazy file loading | ❌ Missing | Full file loaded |
| Memory management | ❌ Missing | No limits or eviction |
| Configuration file | ❌ Missing | Hard-coded values |
### Nice to Have
| Feature | Status | Notes |
|---------|--------|-------|
| Cover art interpolation | ❌ Missing | Would need PICTURE block handling |
| ReplayGain from database | ❌ Missing | Tags not interpolated |
| Lyrics from database | ❌ Missing | Listed in fields, not implemented |
| Watch mode (hot reload) | ❌ Missing | No inotify integration |
| Multiple mount points | ❌ Missing | Global state prevents |
| Remote database | ❌ Missing | Local beets only |
| Read-only mode | ❌ Missing | Always allows writes |
| Custom path templates | ❌ Missing | Hard-coded PATH_FORMAT |
---
## Security Considerations
### 1. No Input Validation
**Location**: Throughout
```python
pathsplit = path[1:].split('/')
item_id = node.files[pathsplit[structure_depth-1]] # No bounds check
```
**Risk**: Path traversal, injection attacks unlikely but possible.
### 2. Database Credentials Exposed
**Issue**: Uses beets library directly with stored credentials.
**Risk**: Low - local access only.
### 3. No Permission Enforcement
**Location**: Lines 749-756
```python
if flags | os.R_OK:
pass # TODO: actually check the file permissions
if flags | os.W_OK:
pass
```
**Risk**: All users can read/write through mount.
---
## Compatibility Issues
| Component | Issue |
|-----------|-------|
| **Jellyfin** | May scan entire library, causing OOM |
| **Plex** | Same library scan issue |
| **Navidrome** | Expects certain tag fields not implemented |
| **mpd** | Works for playback, database features limited |
| **macOS** | fuse-python macOS support questionable |
| **Docker** | FUSE in containers requires privileged mode |
---
## Summary Table
| Category | Critical | Major | Minor |
|----------|----------|-------|-------|
| Performance | 2 | 4 | 2 |
| Functionality | 2 | 5 | 4 |
| Code Quality | 2 | 2 | 4 |
| **Total** | **6** | **11** | **10** |
---
## Prioritized Fix List
1. 🔴 **Memory**: Implement lazy loading (Critical for usability)
2. 🔴 **Python 3**: Migrate to Python 3 (Required for any changes)
3. 🔴 **FUSE lib**: Switch to pyfuse3/llfuse (Required for Python 3)
4. 🔴 **MP3**: Enable MP3 interpolation (Core functionality)
5. 🟡 **Metadata**: Implement all fields (Feature completeness)
6. 🟡 **Threading**: Enable multithreading (Performance)
7. 🟡 **Config**: Add configuration file (Usability)
8. 🟡 **Hot reload**: Watch for library changes (Usability)
9. 🟢 **Globals**: Remove global state (Code quality)
10. 🟢 **Logging**: Configurable logging (Operations)
docs/v1/e2e-test-plan.md
-493
View File
@@ -1,493 +0,0 @@
# beetfs E2E Test Plan
> **Reviewed by Oracle** - Critical bug discovered, plan updated accordingly
## Test Results (Latest Run)
```
Tests run: 74
Passed: 12
Failures: 56
Errors: 3
Skipped: 3
Duration: ~103 seconds
```
### Bugs Detected by Tests
| Bug | Tests Affected | Description |
|-----|----------------|-------------|
| **Nested Methods** | 56 | Lines 758-1144 indented inside `access()` - FUSE operations unreachable |
| **Directory Tree Building** | 3 | `KeyError` in `FSNode.getnode()` when adding files |
| **Unmount** | 1 | Filesystem not unmounting cleanly |
### Passing Tests (12)
- `test_fuse_available` - FUSE/fusermount detected
- `test_library_fixture_created` - SQLite DB and music dir created
- `test_temp_directory_created` - Temp dirs set up correctly
- `test_mount_empty_library` - **Mount works with empty library!**
- `test_list_empty_root` - Empty root returns empty list
- `test_list_root_returns_list` - Returns list type
- `test_access_empty_path` - Handles empty path
- Plus 5 nested bug detection tests (confirming bug exists)
## Executive Summary
E2E tests for beetfs FUSE filesystem using real music files from qBittorrent container. No mocks - actual filesystem operations against mounted beetfs.
### Critical Finding
**BUG DISCOVERED**: Lines 758-1144 in `beetFs.py` are indented inside `access()` method, making these FUSE operations unreachable as class methods:
- `readdir`, `open`, `read`, `write`, `mkdir`, `unlink`, `rmdir`, `symlink`, `link`, `rename`, `chmod`, `chown`, `truncate`, `opendir`, `releasedir`, `fsyncdir`, `create`, `fgetattr`, `release`, `fsync`, `flush`, `ftruncate`
Tests will expose this immediately - write `test_readdir.py` first.
---
## Test Environment
| Component | Status | Details |
|-----------|--------|---------|
| Real Music | Available | Metallica "72 Seasons" (12 FLAC, 650MB) at `/home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/` |
| Synthetic Music | Create | 5-10MB FLACs for most tests (avoid RAM explosion) |
| Beets Config | Create | `~/.config/beets/config.yaml` for test isolation |
| Beets Library | Empty | Needs import of test files |
| Python | 2.7.15 | Via Nix flake (nixpkgs-18.09) |
| Test Framework | unittest | stdlib, no external deps for Py2.7 |
---
## Test Architecture
```
beetfs/tests/
├── __init__.py
├── conftest.py # Test fixtures, beets library setup, synthetic FLAC creation
├── test_smoke.py # Mount/unmount lifecycle (run FIRST)
├── test_nested_bug.py # Verify the indentation bug (run SECOND)
├── test_readdir.py # Directory listing operations
├── test_read.py # File reading with metadata overlay (CORE FEATURE)
├── test_stat.py # getattr, fgetattr, statfs
├── test_write.py # Metadata write operations
├── test_error_handling.py # ENOENT, EOPNOTSUPP scenarios
├── test_edge_cases.py # Unicode, concurrent opens, special chars
├── test_integration.py # Real 650MB files (skip by default)
└── fixtures/
├── synthetic/ # Generated 5-10MB test FLACs
└── real -> /home/fujin/.local/share/docker/volumes/containers_downloads/_data/
```
---
## Test Tiers
### Tier 1: Unit-ish (Synthetic FLACs, ~500KB each)
- Fast execution
- No memory issues (FileHandler loads entire file to RAM)
- Run on every commit
### Tier 2: Integration (Subset of real files, 1-2 tracks)
- Uses real Metallica FLACs
- Tests real-world metadata
- Run before merge
### Tier 3: E2E (All 12 tracks, 650MB)
- Full album processing
- Memory stress testing
- Run via `E2E=1 python -m unittest discover`
- Skip by default
---
## Test Isolation Strategy
| Resource | Strategy | Rationale |
|----------|----------|-----------|
| Audio Files | **Symlinks** for reads | beetfs NEVER writes to source files, only to beets DB |
| Beets DB | **Copy per test** | Writes mutate DB; need isolation |
| Mount Point | **Fresh tempdir** | Each test gets clean mount |
| Global State | **Fresh subprocess** | `library`, `directory_structure` are module globals |
---
## Implementation Order
> Reordered per Oracle recommendation: smoke → nested-bug → read → write → errors → edge
### Phase 1: Infrastructure (Day 1 AM)
1. Create `tests/` directory structure
2. Implement `BeetFSTestCase` base class with:
- Subprocess timeout via `threading.Timer` (Py2.7 compatible)
- Mount wait polling (`os.path.ismount()`)
- Proper cleanup (`fusermount -u`)
3. Create synthetic FLAC generator using ffmpeg + flac CLI
4. Setup isolated beets config and library
### Phase 2: Bug Detection (Day 1 PM)
5. `test_smoke.py` - Mount/unmount lifecycle
6. `test_nested_bug.py` - Verify `readdir`, `open` are callable (will fail, exposing bug)
### Phase 3: Core Tests (Day 2)
7. `test_readdir.py` - Directory listing
8. `test_read.py` - **Metadata overlay verification** (critical)
9. `test_stat.py` - File/directory attributes
### Phase 4: Write & Errors (Day 3)
10. `test_write.py` - Metadata modification, DB persistence
11. `test_error_handling.py` - ENOENT, EOPNOTSUPP
### Phase 5: Edge Cases (Day 3-4)
12. `test_edge_cases.py` - Unicode, concurrent opens, special chars
13. `test_integration.py` - Real 650MB files (optional tier)
---
## Test Categories
### 1. Smoke Tests (`test_smoke.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_mount_success` | Mount beetfs | `os.path.ismount()` returns True |
| `test_unmount_clean` | Unmount | Process exits 0, dir accessible |
| `test_mount_empty_library` | Mount with 0 items | Mounts successfully, root empty |
| `test_mount_invalid_path` | Mount to non-existent | Fails gracefully |
| `test_fsinit_called` | Check initialization | No crash on mount |
### 2. Nested Methods Bug (`test_nested_bug.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_readdir_exists` | `hasattr(beetFileSystem, 'readdir')` | True (currently False!) |
| `test_open_exists` | `hasattr(beetFileSystem, 'open')` | True (currently False!) |
| `test_read_exists` | `hasattr(beetFileSystem, 'read')` | True (currently False!) |
| `test_readdir_callable` | `os.listdir(mount)` | Returns list (currently fails!) |
### 3. Directory Operations (`test_readdir.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_list_root` | `os.listdir(mount)` | Returns artist directories |
| `test_list_artist` | `os.listdir(mount/artist)` | Returns album directories |
| `test_list_album` | `os.listdir(mount/artist/album)` | Returns track files |
| `test_path_format` | Check structure | Matches `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format` |
| `test_unicode_paths` | Non-ASCII chars | Handles "Lux Aeterna" correctly |
### 4. Read Operations (`test_read.py`) - CORE FEATURE
| Test | Operation | Expected |
|------|-----------|----------|
| `test_read_header_overlay` | Read + parse with mutagen | Tags match DB, not file |
| `test_read_audio_passthrough` | Compare audio bytes | Identical to original after header |
| `test_read_full_file` | Read entire file | Header from DB + audio from file |
| `test_metadata_artist` | Check artist tag | DB value, not file value |
| `test_metadata_title` | Check title tag | DB value, not file value |
| `test_metadata_album` | Check album tag | DB value, not file value |
| `test_metadata_genre` | Check genre tag | DB value, not file value |
| `test_original_unchanged` | Read original file | Original metadata intact |
#### Metadata Overlay Verification Pattern
```python
import mutagen.flac
from io import BytesIO
def test_read_header_overlay(self):
# Setup: Import file, modify DB metadata
# beet import /path/to/file
# beet modify artist="DB Artist" # File has "Original Artist"
# Read mounted file as bytes
with open(os.path.join(self.mount_dir, 'DB Artist/...'), 'rb') as f:
mounted_data = f.read()
# Parse with mutagen
flac = mutagen.flac.FLAC(BytesIO(mounted_data))
# Verify overlay worked
self.assertEqual(flac['artist'][0], 'DB Artist') # From DB
self.assertNotEqual(flac['artist'][0], 'Original Artist') # Not from file
```
### 5. Stat Operations (`test_stat.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_stat_file` | `os.stat(file)` | Valid stat with size, mtime |
| `test_stat_directory` | `os.stat(dir)` | Directory mode (S_IFDIR) |
| `test_statfs` | `os.statvfs(mount)` | Valid filesystem stats |
| `test_access_read` | `os.access(file, R_OK)` | True |
| `test_access_write` | `os.access(file, W_OK)` | True (header writable) |
### 6. Write Operations (`test_write.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_write_title` | Modify title in header | DB updated, file unchanged |
| `test_write_artist` | Modify artist | DB updated |
| `test_write_album` | Modify album | DB updated |
| `test_write_genre` | Modify genre | DB updated |
| `test_write_audio_discarded` | Write at offset > bound | Silently discarded |
| `test_write_persistence` | Write -> unmount -> remount | Changes persisted in DB |
| `test_write_mp3_noop` | Write to MP3 header | No error, but no effect (bound=0) |
### 7. Error Handling (`test_error_handling.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_enoent_file` | Read non-existent | `OSError(ENOENT)` |
| `test_enoent_dir` | List non-existent | `OSError(ENOENT)` |
| `test_eopnotsupp_mkdir` | `os.mkdir()` | `OSError(EOPNOTSUPP)` |
| `test_eopnotsupp_unlink` | `os.unlink()` | `OSError(EOPNOTSUPP)` |
| `test_eopnotsupp_rename` | `os.rename()` | `OSError(EOPNOTSUPP)` |
| `test_eopnotsupp_symlink` | `os.symlink()` | `OSError(EOPNOTSUPP)` |
### 8. Edge Cases (`test_edge_cases.py`)
| Test | Operation | Expected |
|------|-----------|----------|
| `test_special_chars_sanitized` | Path with `?/` | Sanitized via `sanitize()` |
| `test_concurrent_opens` | Open same file twice | `instance_count` increments |
| `test_concurrent_release` | Release after double open | File stays cached until count=0 |
| `test_unicode_metadata` | Non-ASCII in artist/title | Handled correctly |
| `test_empty_metadata` | None/empty fields | Doesn't crash |
| `test_mp3_no_interpolation` | Read MP3 | Returns original file (no overlay) |
### 9. Integration (`test_integration.py`)
| Test | Env Var | Expected |
|------|---------|----------|
| `test_real_album_listing` | `E2E=1` | Lists all 12 Metallica tracks |
| `test_real_file_read` | `E2E=1` | Reads 67MB file successfully |
| `test_memory_usage` | `E2E=1` | Documents but doesn't fail on high RAM |
---
## Test Infrastructure Code
### Base Test Class (Python 2.7 Compatible)
```python
# tests/conftest.py
import unittest
import subprocess
import tempfile
import shutil
import os
import time
import threading
class BeetFSTestCase(unittest.TestCase):
"""Base class for beetfs e2e tests - Python 2.7 compatible"""
MOUNT_TIMEOUT = 30 # seconds
@classmethod
def setUpClass(cls):
"""Check FUSE availability"""
try:
with open(os.devnull, 'w') as devnull:
subprocess.check_call(['which', 'fusermount'],
stdout=devnull, stderr=devnull)
except subprocess.CalledProcessError:
raise unittest.SkipTest("fusermount not available")
def setUp(self):
self.mount_dir = tempfile.mkdtemp(prefix='beetfs_test_')
self.fs_process = None
def mount_beetfs(self, library_path=None):
"""Mount beetfs in background with timeout"""
cmd = ['python', '-c',
'from beetsplug.beetFs import mount; mount()']
# Add mount point and other args as needed
self.fs_process = subprocess.Popen(
cmd,
stdout=open(os.devnull, 'w'),
stderr=subprocess.STDOUT
)
# Python 2.7 timeout workaround
timer = threading.Timer(self.MOUNT_TIMEOUT, self._timeout_kill)
timer.start()
try:
self._wait_for_mount()
finally:
timer.cancel()
def _timeout_kill(self):
if self.fs_process and self.fs_process.poll() is None:
self.fs_process.kill()
def _wait_for_mount(self):
"""Wait for filesystem to be mounted"""
start = time.time()
while time.time() - start < self.MOUNT_TIMEOUT:
if os.path.ismount(self.mount_dir):
return
if self.fs_process.poll() is not None:
self.fail("Filesystem process terminated prematurely")
time.sleep(0.1)
self.fail("Mount timeout after {} seconds".format(self.MOUNT_TIMEOUT))
def tearDown(self):
"""Cleanup: unmount and kill process"""
if self.fs_process:
with open(os.devnull, 'w') as devnull:
subprocess.call(['fusermount', '-z', '-u', self.mount_dir],
stdout=devnull, stderr=devnull)
self.fs_process.terminate()
# Wait for termination (Py2.7 compatible)
start = time.time()
while time.time() - start < 5:
if self.fs_process.poll() is not None:
break
time.sleep(0.1)
else:
self.fs_process.kill()
shutil.rmtree(self.mount_dir, ignore_errors=True)
```
### Synthetic FLAC Generator
```python
# tests/conftest.py (continued)
import subprocess
import tempfile
import os
def create_synthetic_flac(duration_sec=5, artist="Test Artist",
title="Test Track", album="Test Album"):
"""Create minimal FLAC with known metadata (~500KB for 5s silence)"""
wav_fd, wav_path = tempfile.mkstemp(suffix='.wav')
os.close(wav_fd)
flac_path = wav_path.replace('.wav', '.flac')
try:
# Generate silence WAV
subprocess.check_call([
'ffmpeg', '-f', 'lavfi', '-i',
'anullsrc=r=44100:cl=stereo', '-t', str(duration_sec),
'-y', wav_path
], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT)
# Convert to FLAC with metadata
subprocess.check_call([
'flac', '--best',
'-T', 'ARTIST={}'.format(artist),
'-T', 'TITLE={}'.format(title),
'-T', 'ALBUM={}'.format(album),
'-o', flac_path, wav_path
], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT)
return flac_path
finally:
if os.path.exists(wav_path):
os.unlink(wav_path)
```
---
## Dependencies to Add to flake.nix
```nix
# In devShell buildInputs, add:
pkgs.ffmpeg # For synthetic FLAC generation
pkgs.flac # For FLAC encoding
# pythonEnv already has mutagen for verification
```
---
## Risks & Mitigations
| Risk | Impact | Mitigation |
|------|--------|------------|
| Memory explosion | High | Use 5-10MB synthetic FLACs, skip 650MB tests by default |
| Nested methods bug | Critical | Tests will expose; fix required before other tests pass |
| Python 2.7 EOL | Medium | Nix provides isolated environment |
| Global state pollution | Medium | Fresh subprocess per test |
| FUSE permissions | Low | Run as regular user, skip privileged tests |
| Concurrent access | Low | Single-threaded mode, sequential tests |
---
## Success Criteria
1. **All smoke tests pass** - beetfs mounts and unmounts cleanly
2. **Nested bug exposed and fixed** - All FUSE methods callable
3. **Metadata overlay verified** - Reads return DB metadata, not file metadata
4. **Writes update DB** - Metadata changes persist
5. **Errors handled gracefully** - Correct errno for unsupported ops
6. **No crashes on edge cases** - Unicode, special chars, concurrent access
---
## Findings from Test Execution
### Bug #1: Nested Methods (CRITICAL)
**Location**: `beetFs.py` lines 758-1144
**Problem**: All FUSE operation methods are indented inside the `access()` method, making them local functions instead of class methods.
**Evidence**:
```python
def access(self, path, flags): # Line 723 - correct class method
...
return 0
def readdir(self, path, ...): # Line 931 - WRONG! Nested inside access()
...
def open(self, path, flags): # Line 988 - Also nested
...
def read(self, path, ...): # Line 1077 - Also nested
...
```
**Symptom**: `os.listdir()` returns `OSError: [Errno 38] Function not implemented`
**Fix Required**: Dedent lines 758-1144 by 8 spaces to make them class methods.
### Bug #2: Directory Tree Building
**Location**: `beetFs.py` lines 403-414 (`FSNode.getnode()` and `FSNode.adddir()`)
**Problem**: When adding files to the directory structure, the code assumes parent directories already exist.
**Evidence**:
```
KeyError: u'Test Artist'
File "beetFs.py", line 403, in getnode
return self.getnode(elements, root=root.dirs[topdir])
```
**Symptom**: Mount fails when library contains tracks.
### Bug #3: Unmount Not Clean
**Problem**: After unmounting, `os.path.ismount()` still returns `True`.
**Likely Cause**: FUSE process not terminating properly, or lazy unmount not completing.
---
## Notes from Oracle Review
1. **MP3 is not "readonly"** - metadata overlay is disabled (`bound=0`), but reads still work
2. **Write returns None for MP3** - no explicit return in MP3 path (falls through)
3. **Path format is hardcoded** - tests must match `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format`
4. **basestring vs str** - use `isinstance(x, basestring)` for Py2.7 string checks
5. **Global variables** - `library`, `directory_structure` must be reset between tests (use subprocesses)
docs/v1/features.md
-249
View File
@@ -1,249 +0,0 @@
# beetfs Feature Set
## Overview
beetfs is a FUSE filesystem plugin for [beets](https://beets.io/) that presents your music library as a virtual filesystem organized by metadata. Files appear with paths derived from their database metadata, and reading file headers returns metadata from the beets database rather than the actual file tags.
**Author**: Martin Eve (2010)
**License**: GPLv3
**Python**: 2.7 (uses fuse-python)
## Core Features
### 1. Virtual Metadata-Based Directory Structure
Files are presented in a configurable path format based on beets database fields:
```
$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format
```
**Example**:
```
/mnt/beetfs/
├── Metallica/
│ └── 72 Seasons (2023) [FLAC]/
│ ├── 01 - Metallica - 72 Seasons.flac
│ ├── 02 - Metallica - Shadows Follow.flac
│ └── ...
├── Pink Floyd/
│ └── The Dark Side of the Moon (1973) [FLAC]/
│ └── ...
```
**Available template variables**:
- `$artist`, `$album`, `$title`, `$genre`, `$composer`, `$grouping`
- `$year`, `$month`, `$day`
- `$track`, `$tracktotal`, `$disc`, `$disctotal`
- `$format`, `$format_upper` (file extension)
- `$lyrics`, `$comments`, `$bpm`, `$comp`
### 2. Metadata Overlay (Read)
When you read a file through beetfs, the **metadata header is synthesized from the beets database**, not read from the actual file on disk.
**How it works**:
1. Open file → beetfs reads the real file from disk
2. Parse the audio format header (FLAC/MP3)
3. Replace metadata fields with values from beets database
4. Return synthesized header + original audio data
**Supported fields for overlay**:
- `title`, `artist`, `album`, `genre` (FLAC only currently)
**Use case**: Your files may have inconsistent or wrong tags, but beetfs presents them with the corrected metadata from your beets library.
### 3. Metadata Passthrough (Write)
When you write to file headers through beetfs, the **changes are saved to the beets database**, not to the actual file.
**How it works**:
1. Application writes new metadata to file header region
2. beetfs intercepts the write
3. Parses the new metadata values
4. Updates the beets database (`lib.store()`, `lib.save()`)
5. Regenerates the synthesized header
**Result**: Tag editors (Picard, Kid3, etc.) can edit metadata through beetfs, and changes persist in the beets database without modifying the original files.
### 4. Format Support
| Format | Read | Metadata Overlay | Write to DB |
|--------|------|------------------|-------------|
| FLAC | ✅ | ✅ Full | ✅ |
| MP3 | ✅ | ❌ Disabled | ❌ |
| Other | ❌ | ❌ | ❌ |
**FLAC Implementation**:
- Uses `InterpolatedFLAC` class extending mutagen
- Reconstructs Vorbis comment block with DB values
- Preserves audio data and other metadata blocks
**MP3 Implementation**:
- Passthrough only (no interpolation)
- `self.bound = 0` disables header replacement
### 5. File Caching
Open files are cached in `FileHandler` objects:
- First open: Load entire file into memory, parse headers
- Subsequent opens: Reuse cached `FileHandler`
- Reference counting for multiple opens
- Release when reference count reaches zero
**Memory impact**: Each open file consumes ~filesize RAM.
## FUSE Operations
### Implemented (Functional)
| Operation | Description |
|-----------|-------------|
| `getattr` | File/directory stat (size, mode, timestamps) |
| `access` | Permission checking |
| `opendir` | Open directory for listing |
| `readdir` | List directory contents |
| `releasedir` | Close directory |
| `open` | Open file for reading/writing |
| `read` | Read file contents |
| `write` | Write to file (header region only) |
| `release` | Close file |
| `fgetattr` | Stat with file handle |
| `statfs` | Filesystem statistics |
### Not Implemented (Return EOPNOTSUPP)
| Operation | Reason |
|-----------|--------|
| `create` | Read-only structure |
| `mknod` | Read-only structure |
| `mkdir` | Read-only structure |
| `unlink` | Read-only structure |
| `rmdir` | Read-only structure |
| `symlink` | Not needed |
| `link` | Not needed |
| `rename` | Would break DB consistency |
| `chmod` | Metadata-only FS |
| `chown` | Metadata-only FS |
| `truncate` | Would corrupt audio |
| `utime` | Metadata-only FS |
## Usage
### Mount
```bash
beet mount /mnt/beetfs
```
### Unmount
```bash
fusermount -u /mnt/beetfs
```
### Example Session
```bash
# Mount the filesystem
beet mount /mnt/music
# Browse by artist
ls /mnt/music/
# Metallica/ Pink Floyd/ The Beatles/ ...
# List an album
ls "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/"
# 01 - Metallica - 72 Seasons.flac
# 02 - Metallica - Shadows Follow.flac
# ...
# Play through any music player
mpv "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/01 - Metallica - 72 Seasons.flac"
# Edit tags (changes go to beets DB)
kid3 "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/"
# Unmount
fusermount -u /mnt/music
```
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ User Applications │
│ (mpv, Rhythmbox, Kid3, etc.) │
└─────────────────────────┬───────────────────────────────────┘
│ POSIX calls (open, read, write)
▼
┌─────────────────────────────────────────────────────────────┐
│ Linux Kernel │
│ FUSE module │
└─────────────────────────┬───────────────────────────────────┘
│ /dev/fuse
▼
┌─────────────────────────────────────────────────────────────┐
│ beetfs │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
│ │ FSNode Tree │ │ FileHandler │ │ InterpolatedFLAC │ │
│ │ (in-memory) │ │ (cache) │ │ (header synth) │ │
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
└────────┬────────────────┬───────────────────┬───────────────┘
│ │ │
▼ ▼ ▼
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Beets DB │ │ Real Files │ │ Mutagen │
│ (SQLite) │ │ (on disk) │ │ (parsing) │
└─────────────┘ └─────────────────┘ └─────────────────┘
```
## Limitations
### Current Bugs (Non-Functional)
1. **Nested Methods Bug**: Lines 758-1144 are indented inside `access()`, making FUSE operations unreachable
2. **Directory Tree Bug**: `FSNode.adddir()` crashes when building tree for non-empty library
### Design Limitations
1. **Memory Usage**: Entire file loaded into RAM on open
2. **Mount Time**: O(N) - loads all library items at mount
3. **No Lazy Loading**: Full directory tree built upfront
4. **Single Format**: Only FLAC has full metadata overlay
5. **No Real File Modification**: Writes only update DB, not actual files
6. **Python 2.7 GIL**: Single-threaded performance
### Not Supported
- Creating/deleting files or directories
- Moving/renaming files
- Modifying audio content
- Album art / embedded images
- Multi-value tags
- Non-ASCII in some edge cases
## Configuration
Currently hardcoded. Potential configuration points:
| Setting | Current Value | Description |
|---------|---------------|-------------|
| `PATH_FORMAT` | `$artist/$album ($year)...` | Directory structure template |
| `METADATA_RW_FIELDS` | 17 fields | Fields available for read/write |
| Caching | Always on | FileHandler caching behavior |
| Threading | Disabled | `multithreaded = 0` |
## Dependencies
- Python 2.7
- fuse-python
- beets 1.4.x
- mutagen (FLAC/MP3 parsing)
## See Also
- [e2e-test-plan.md](e2e-test-plan.md) - Test strategy and bug documentation
- [benchmark-plan.md](benchmark-plan.md) - Performance measurement methodology
- [benchmark-results.md](benchmark-results.md) - Current benchmark status
docs/v1/modernization.md
-459
View File
@@ -1,459 +0,0 @@
# beetfs Modernization Guide
## Current State Analysis
### Technical Debt
| Issue | Severity | Location |
|-------|----------|----------|
| Python 2 syntax | 🔴 Critical | Throughout |
| fuse-python (deprecated) | 🔴 Critical | Lines 25, 51 |
| `basestring` usage | 🔴 Critical | Line 89 |
| `reduce` without import | 🟡 Medium | Line 197 |
| `0755` octal syntax | 🟡 Medium | Lines 654, 700 |
| `print` as statement | 🟡 Medium | N/A (not used) |
| `except Exception, e` | 🔴 Critical | Line 181 |
| Long integers (`0L`) | 🟡 Medium | Line 197 |
| Global state | 🟡 Medium | Lines 125-140 |
| Memory-heavy design | 🟡 Medium | Line 481 |
### Dependencies to Update
| Original | Replacement | Notes |
|----------|-------------|-------|
| `fuse-python` | `pyfuse3` or `llfuse` | Modern FUSE bindings |
| `beets` (old API) | `beets >= 1.6` | Check API compatibility |
| `mutagen` | `mutagen >= 1.45` | Mostly compatible |
| Python 2.7 | Python 3.9+ | Full migration needed |
---
## Migration Steps
### Phase 1: Python 3 Compatibility
#### 1.1 Fix Syntax Issues
```python
# BEFORE (Python 2)
except fuse.FuseError, e:
log.error(str(e))
# AFTER (Python 3)
except fuse.FuseError as e:
log.error(str(e))
```
```python
# BEFORE
if isinstance(value, basestring):
# AFTER
if isinstance(value, str):
```
```python
# BEFORE
return reduce(lambda a, b: (a << 8) + ord(b), string, 0L)
# AFTER
from functools import reduce
return reduce(lambda a, b: (a << 8) + b, string, 0)
```
```python
# BEFORE
mode = stat.S_IFDIR | 0755
# AFTER
mode = stat.S_IFDIR | 0o755
```
#### 1.2 Fix String/Bytes Handling
```python
# BEFORE - implicit string/bytes mixing
self.header = self.inf.get_header(self.real_path)
return self.header[offset:offset+size]
# AFTER - explicit bytes handling
self.header: bytes = self.inf.get_header(self.real_path)
return self.header[offset:offset+size]
```
```python
# BEFORE
self.item.title = str(self.inf["title"][0]).encode('utf-8')
# AFTER
self.item.title = self.inf["title"][0] # Already str in Python 3
```
#### 1.3 Fix Dictionary Methods
```python
# BEFORE
return node.dirs.keys()
# AFTER
return list(node.dirs.keys()) # If list is needed
# or just
return node.dirs.keys() # If iteration is sufficient
```
---
### Phase 2: FUSE Library Migration
#### Option A: pyfuse3 (Recommended)
Modern, async-capable FUSE bindings.
```python
# BEFORE (fuse-python)
import fuse
fuse.fuse_python_api = (0, 2)
class beetFileSystem(fuse.Fuse):
def read(self, path, size, offset):
return data
# AFTER (pyfuse3)
import pyfuse3
import trio
class BeetFS(pyfuse3.Operations):
async def read(self, fh, offset, size):
return data
async def main():
fs = BeetFS()
fuse_options = set(pyfuse3.default_options)
fuse_options.add('fsname=beetfs')
pyfuse3.init(fs, mountpoint, fuse_options)
try:
await pyfuse3.main()
finally:
pyfuse3.close()
trio.run(main)
```
**Key Differences**:
| fuse-python | pyfuse3 |
|-------------|---------|
| `read(path, size, offset)` | `read(fh, offset, size)` |
| Synchronous | Async (trio) |
| Return data directly | Return bytes |
| Path-based | File handle based |
#### Option B: llfuse (Alternative)
Lower-level, synchronous.
```python
import llfuse
class BeetFS(llfuse.Operations):
def read(self, fh, offset, size):
return data
def main():
fs = BeetFS()
llfuse.init(fs, mountpoint, options)
try:
llfuse.main()
finally:
llfuse.close()
```
#### Option C: fusepy (Simple)
Simple wrapper, but less maintained.
```python
from fuse import FUSE, Operations
class BeetFS(Operations):
def read(self, path, size, offset, fh):
return data
FUSE(BeetFS(), mountpoint, foreground=True)
```
---
### Phase 3: Architecture Improvements
#### 3.1 Remove Global State
```python
# BEFORE - Global variables
global structure_split
global structure_depth
global library
global directory_structure
# AFTER - Instance variables
class BeetFS:
def __init__(self, lib: Library, path_format: str):
self.lib = lib
self.path_format = path_format
self.structure_split = path_format.split("/")
self.structure_depth = len(self.structure_split)
self.directory_structure = FSNode({}, {})
self._build_tree()
```
#### 3.2 Reduce Memory Usage
```python
# BEFORE - Load entire audio into memory
self.music_data = self.file_object.read() # Could be 100MB+
# AFTER - Lazy loading with mmap or seek
class FileHandler:
def __init__(self, path, lib):
self.real_path = self._resolve_path(path)
self.file_object = open(self.real_path, 'rb')
self._header = None # Lazy load
self._music_offset = None
@property
def header(self) -> bytes:
if self._header is None:
self._header = self._generate_header()
return self._header
def read(self, size: int, offset: int) -> bytes:
if offset < len(self.header):
# Header region - return from generated header
if offset + size <= len(self.header):
return self.header[offset:offset+size]
else:
# Span header and audio
header_part = self.header[offset:]
audio_offset = 0
audio_size = size - len(header_part)
audio_part = self._read_audio(audio_offset, audio_size)
return header_part + audio_part
else:
# Audio region - read directly from file
audio_offset = offset - len(self.header)
return self._read_audio(audio_offset, size)
def _read_audio(self, offset: int, size: int) -> bytes:
self.file_object.seek(self._music_offset + offset)
return self.file_object.read(size)
```
#### 3.3 Add Type Hints
```python
from typing import Dict, List, Optional, Tuple
from pathlib import Path
class FSNode:
def __init__(self, dirs: Dict[str, 'FSNode'], files: Dict[str, int]):
self.dirs: Dict[str, FSNode] = dirs
self.files: Dict[str, int] = files
def getnode(self, elements: List[str], root: Optional['FSNode'] = None) -> 'FSNode':
...
def addfile(self, elements: List[str], filename: str, item_id: int) -> None:
...
```
#### 3.4 Add MP3 Support
```python
class FileHandler:
def __init__(self, path: str, lib: Library):
self.format = Path(path).suffix[1:].lower()
if self.format == "flac":
self._handler = FLACHandler(self.real_path, self.item)
elif self.format == "mp3":
self._handler = MP3Handler(self.real_path, self.item)
elif self.format in ("ogg", "opus"):
self._handler = OggHandler(self.real_path, self.item)
else:
raise UnsupportedFormatError(f"Format {self.format} not supported")
class FLACHandler:
def generate_header(self, item: Item) -> bytes:
inf = InterpolatedFLAC(self.file_data)
inf["title"] = item.title
inf["album"] = item.album
inf["artist"] = item.artist
inf["genre"] = item.genre
return inf.get_header()
class MP3Handler:
def generate_header(self, item: Item) -> bytes:
# Implement ID3v2 header generation
id3 = InterpolatedID3()
id3.add(TIT2(encoding=3, text=item.title))
id3.add(TPE1(encoding=3, text=item.artist))
id3.add(TALB(encoding=3, text=item.album))
id3.add(TCON(encoding=3, text=item.genre))
# Calculate padding to match original header size
...
return id3.render()
```
---
### Phase 4: Testing
#### 4.1 Unit Tests
```python
import pytest
from beetfs import FSNode, FileHandler
class TestFSNode:
def test_adddir(self):
root = FSNode({}, {})
root.adddir([], "Artist")
assert "Artist" in root.dirs
def test_addfile(self):
root = FSNode({}, {})
root.adddir([], "Artist")
root.addfile(["Artist"], "track.flac", 42)
assert root.dirs["Artist"].files["track.flac"] == 42
def test_getnode(self):
root = FSNode({}, {})
root.adddir([], "Artist")
root.adddir(["Artist"], "Album")
node = root.getnode(["Artist", "Album"])
assert node is not None
class TestFileHandler:
def test_read_header(self, mock_flac_file, mock_beets_item):
handler = FileHandler("/Artist/Album/track.flac", mock_lib)
data = handler.read(100, 0)
assert data.startswith(b"fLaC")
def test_read_audio(self, mock_flac_file, mock_beets_item):
handler = FileHandler("/Artist/Album/track.flac", mock_lib)
data = handler.read(100, handler.bound + 100)
# Should be audio data from original file
assert data == mock_flac_file.audio_data[100:200]
```
#### 4.2 Integration Tests
```python
import subprocess
import tempfile
import os
class TestFUSEMount:
def test_mount_unmount(self, beets_library):
with tempfile.TemporaryDirectory() as mountpoint:
# Mount
proc = subprocess.Popen(
["beet", "mount", mountpoint],
stdout=subprocess.PIPE
)
time.sleep(1)
# Verify mount
assert os.path.ismount(mountpoint)
# List files
files = os.listdir(mountpoint)
assert len(files) > 0
# Unmount
subprocess.run(["fusermount", "-u", mountpoint])
proc.wait()
```
---
### Phase 5: Standalone Mode (Optional)
Remove beets dependency for use as standalone metadata overlay.
```python
class StandaloneFS:
"""Metadata overlay without beets dependency."""
def __init__(self,
source_dir: Path,
metadata_db: Path,
path_format: str):
self.source_dir = source_dir
self.db = sqlite3.connect(metadata_db)
self.path_format = path_format
self._build_tree()
def _build_tree(self):
"""Build virtual tree from source directory and metadata DB."""
for audio_file in self.source_dir.rglob("*.flac"):
# Get metadata from DB or scan file
metadata = self._get_metadata(audio_file)
# Build virtual path from template
virtual_path = self._format_path(metadata)
# Add to tree
self.directory_structure.addfile(
virtual_path.parent.parts,
virtual_path.name,
str(audio_file) # Store actual path instead of ID
)
```
---
## Recommended Migration Order
```
1. [ ] Fork and set up development environment
2. [ ] Add type hints throughout (helps catch issues)
3. [ ] Fix Python 3 syntax issues
4. [ ] Replace fuse-python with pyfuse3/llfuse
5. [ ] Add unit tests for FSNode and FileHandler
6. [ ] Refactor global state to instance variables
7. [ ] Implement lazy loading for audio data
8. [ ] Add MP3 support
9. [ ] Add integration tests
10. [ ] Optional: Create standalone mode
```
---
## Estimated Effort
| Phase | Effort | Risk |
|-------|--------|------|
| Phase 1 (Python 3) | 2-3 days | Low |
| Phase 2 (FUSE migration) | 3-5 days | Medium |
| Phase 3 (Architecture) | 3-5 days | Medium |
| Phase 4 (Testing) | 2-3 days | Low |
| Phase 5 (Standalone) | 3-5 days | Medium |
| **Total** | **13-21 days** | |
---
## Alternative: Rewrite from Scratch
Given the age of the codebase, a rewrite might be more efficient:
**Pros of Rewrite**:
- Clean architecture from start
- Modern async design
- Better memory management
- Easier to test
**Cons of Rewrite**:
- More initial effort
- Risk of missing edge cases
- Need to re-discover FLAC/ID3 intricacies
**Recommended Approach**: Start with Phase 1-2 to understand the code deeply, then decide whether to continue refactoring or rewrite.
docs/v1/rust-migration.md
-451
View File
@@ -1,451 +0,0 @@
# Rust Migration Analysis for beetfs
## Executive Summary
Migrating beetfs from Python to Rust is **strongly recommended** based on research findings. Expected improvements:
| Metric | Python (Current) | Rust (Expected) | Improvement |
|--------|------------------|-----------------|-------------|
| **Memory per file** | ~280 bytes overhead | ~60 bytes | **4-5x reduction** |
| **File open latency** | 200-500ms | 20-50ms | **10x faster** |
| **Read latency** | 5-10ms | 0.5-2ms | **5-10x faster** |
| **Concurrent opens** | ~1,000 (threading) | ~100,000+ (Tokio) | **100x more** |
| **GC pauses** | 50-2200ms | 0ms | **Eliminated** |
---
## 1. Rust FUSE Ecosystem
### Recommended: **fuser**
| Attribute | Value |
|-----------|-------|
| **Downloads** | 3.2M+ |
| **Maturity** | Production-ready |
| **Platforms** | Linux, macOS, FreeBSD |
| **Async** | Experimental (stable sync API) |
| **Used by** | AWS Mountpoint for S3 |
**API Example:**
```rust
use fuser::{Filesystem, Request, ReplyData};
impl Filesystem for BeetFS {
fn read(&self, _req: &Request, ino: u64, _fh: u64,
offset: i64, size: u32, _flags: i32,
_lock: Option<u64>, reply: ReplyData) {
let file = self.get_file(ino);
if offset < file.header_len {
// Return metadata from database (interpolated)
reply.data(&file.header[offset as usize..]);
} else {
// Return audio from original file (zero-copy via mmap)
let audio_offset = offset - file.header_len;
reply.data(&file.mmap[audio_offset as usize..]);
}
}
}
```
### Alternatives
| Library | Async | Maturity | Best For |
|---------|-------|----------|----------|
| **fuser** | Experimental | ⭐⭐⭐⭐⭐ | General purpose |
| **fuse3** | Native | ⭐⭐⭐⭐ | Async-heavy, Linux-only |
| **polyfuse** | Native | ⭐⭐⭐ | Custom control flow |
---
## 2. Rust Audio Metadata: **lofty**
Full feature parity with Python's mutagen:
| Feature | mutagen (Python) | lofty (Rust) |
|---------|------------------|--------------|
| FLAC Vorbis Comments | ✅ | ✅ |
| MP3 ID3v2 (all versions) | ✅ | ✅ |
| OGG Vorbis Comments | ✅ | ✅ |
| Opus metadata | ✅ | ✅ |
| In-memory manipulation | ✅ | ✅ |
| Header generation | ✅ | ✅ `dump_to()` |
| Picture/artwork | ✅ | ✅ |
**API Comparison:**
```python
# Python mutagen
audio = mutagen.File("song.flac")
audio['artist'] = 'New Artist'
audio['title'] = 'New Title'
audio.save()
```
```rust
// Rust lofty
let mut file = lofty::read_from_path("song.flac")?;
let tag = file.primary_tag_mut().unwrap();
tag.set_artist("New Artist".to_string());
tag.set_title("New Title".to_string());
tag.save_to_path("song.flac", WriteOptions::default())?;
```
**Header Generation (Critical for beetfs):**
```rust
// Generate FLAC header with modified tags WITHOUT writing to file
let mut buffer = Vec::new();
tag.dump_to(&mut buffer, WriteOptions::default())?;
// `buffer` contains serialized metadata header
```
---
## 3. Memory Benefits
### Python Object Overhead
| Python Type | Size | Notes |
|-------------|------|-------|
| Empty dict | 232 bytes | Base overhead |
| Dict entry | +184 bytes | Per key-value |
| Empty string | 49 bytes | Base overhead |
| Empty list | 56 bytes | Base overhead |
| Small int | 28 bytes | Even for `0` |
**Current beetfs FileHandler (Python):**
```
self.path → str → 49 + len(path) bytes
self.real_path → str → 49 + len(path) bytes
self.item → dict → 232 + entries
self.header → bytes → 33 + len(header)
self.music_data → bytes → 33 + len(audio) ← CRITICAL: full file!
self.inf → object → 100+ bytes
─────────────────────────────────────────
TOTAL: ~500 bytes + entire file in RAM
```
### Rust Struct Efficiency
```rust
struct FileHandler {
path: PathBuf, // 24 bytes (ptr+len+cap)
real_path: PathBuf, // 24 bytes
item_id: u64, // 8 bytes
header: Vec<u8>, // 24 bytes (ptr+len+cap) + header data
mmap: Mmap, // 24 bytes (NO file data in RAM!)
header_len: u64, // 8 bytes
audio_offset: u64, // 8 bytes
}
// TOTAL: ~120 bytes + header only (audio via mmap)
```
### Memory Comparison
| Scenario | Python | Rust | Savings |
|----------|--------|------|---------|
| 1 file (50MB) | ~50 MB | ~64 KB | **780x** |
| 10 files (50MB each) | ~500 MB | ~640 KB | **780x** |
| 100 files (50MB each) | ~5 GB | ~6.4 MB | **780x** |
| Library scan (1000 files) | **OOM** | ~64 MB | ∞ |
**Key insight**: Rust can use memory-mapped files (`mmap`) to serve audio data with zero copies, eliminating the need to load files into RAM.
---
## 4. Latency Benefits
### Python FUSE Bottlenecks
1. **Dict-to-struct conversion**: Every FUSE callback requires converting Python dicts to C structs
2. **GIL contention**: Single-threaded execution despite multi-core CPUs
3. **GC pauses**: Stop-the-world pauses of 50-2200ms under load
4. **Object allocation**: Creating Python objects for every I/O operation
### Rust FUSE Advantages
1. **Zero-cost abstractions**: No runtime overhead for type conversions
2. **No GIL**: True parallelism across all cores
3. **No GC**: Deterministic memory management, no pauses
4. **Stack allocation**: Small objects allocated on stack, not heap
### Benchmark Data
| Operation | Python FUSE | Rust FUSE | Improvement |
|-----------|-------------|-----------|-------------|
| File stat | 5-10ms | 0.5-1ms | **10x** |
| Small read | 5-10ms | 0.5-2ms | **5-10x** |
| Large read | 115 MB/s | 260+ MB/s | **2-3x** |
| Metadata lookup | 10ms | <1ms | **10x** |
### GC Pause Elimination
```
Python GC Pauses (measured):
├── P50: ~10ms
├── P95: ~50ms
├── P99: ~320ms
└── Max: ~2200ms (!)
Rust (no GC):
├── P50: ~0.5ms
├── P95: ~1ms
├── P99: ~2ms
└── Max: ~5ms (deterministic)
```
---
## 5. Concurrency Benefits
### Python Threading Limitations
```python
# Python (current beetfs)
server.multithreaded = 0 # Single-threaded!
# Even with threading enabled:
# - GIL prevents true parallelism
# - ~8MB per thread
# - OS limits: ~1000-2000 threads max
# - Context switch: 1-10μs (kernel)
```
### Rust Async (Tokio)
```rust
// Rust with Tokio
#[tokio::main]
async fn main() {
// Can handle 100K+ concurrent operations
// - ~2KB per task (4000x less than thread)
// - Work-stealing scheduler
// - Context switch: ~10ns (userspace)
}
```
| Metric | Python Threading | Rust Tokio |
|--------|------------------|------------|
| Memory per task | 8 MB | 2 KB |
| Max concurrent | ~1,000 | ~100,000+ |
| Context switch | 1-10μs | ~10ns |
| Parallelism | Blocked by GIL | True multi-core |
---
## 6. Zero-Copy I/O
### Python (Current)
```python
# Every read copies data through Python:
self.file_object.read() # syscall → kernel buffer
# kernel buffer → Python bytes object
# Python bytes → FUSE reply buffer
# = 2-3 copies per read
```
### Rust (Proposed)
```rust
// Memory-mapped file + zero-copy reply:
let mmap = unsafe { MmapOptions::new().map(&file)? };
fn read(&self, ..., reply: ReplyData) {
// Direct slice from mmap → FUSE kernel
reply.data(&self.mmap[offset..offset+size]);
// = 0 copies (kernel reads directly from mapped pages)
}
```
### I/O Comparison
| Scenario | Python | Rust | Benefit |
|----------|--------|------|---------|
| Serve 50MB file | 50MB copied to RAM | 0 bytes copied | **50MB saved** |
| 100 concurrent reads | 5GB buffers | ~0 (shared mmap) | **5GB saved** |
| Throughput | 115 MB/s | 260+ MB/s | **2.3x faster** |
---
## 7. Real-World Migration Results
### Case Studies
| Project | Metric | Python | Rust | Improvement |
|---------|--------|--------|------|-------------|
| API Service | Response time | 200ms | 8ms | **96% faster** |
| Data Pipeline | Processing | 3 hours | 4.5 min | **40x faster** |
| Web Backend | Memory | 1.2 GB | 180 MB | **85% less** |
| Trajectory Lib | Compute | baseline | 10x faster | **10x** |
### AWS Mountpoint for S3
- Built on **fuser** (Rust FUSE)
- Handles **terabits/sec** aggregate throughput
- Production-ready since 2024
- Validates Rust FUSE at scale
---
## 8. Migration Architecture
### Proposed Rust beetfs Structure
```
beetfs-rs/
├── Cargo.toml
├── src/
│ ├── main.rs # Entry point, mount logic
│ ├── lib.rs # Library root
│ ├── fs/
│ │ ├── mod.rs # FUSE filesystem impl
│ │ ├── tree.rs # Virtual directory tree (FSNode equivalent)
│ │ ├── file.rs # File handler with mmap
│ │ └── stat.rs # File attributes
│ ├── metadata/
│ │ ├── mod.rs # Metadata overlay logic
│ │ ├── flac.rs # FLAC header generation (using lofty)
│ │ ├── mp3.rs # MP3 ID3 header generation
│ │ └── db.rs # Database interface (SQLite or custom)
│ └── config.rs # Configuration (path templates, etc.)
└── tests/
├── fs_tests.rs
└── metadata_tests.rs
```
### Key Components
```rust
// Virtual directory tree (equivalent to FSNode)
pub struct VirtualTree {
root: Arc<RwLock<DirNode>>,
}
pub struct DirNode {
dirs: HashMap<OsString, Arc<RwLock<DirNode>>>,
files: HashMap<OsString, FileEntry>,
}
pub struct FileEntry {
inode: u64,
real_path: PathBuf,
metadata_id: i64, // Database reference
}
// File handler with memory-mapped audio
pub struct OpenFile {
header: Vec<u8>, // Generated header with DB metadata
header_len: usize,
mmap: Mmap, // Memory-mapped original file
audio_offset: usize, // Where audio starts in original
}
impl OpenFile {
pub fn read(&self, offset: usize, size: usize) -> &[u8] {
if offset < self.header_len {
// Return from generated header (DB metadata)
&self.header[offset..min(offset + size, self.header_len)]
} else {
// Return from mmap (original audio, zero-copy)
let audio_off = offset - self.header_len + self.audio_offset;
&self.mmap[audio_off..audio_off + size]
}
}
}
```
---
## 9. Migration Effort Estimate
### Timeline
| Phase | Duration | Deliverable |
|-------|----------|-------------|
| **1. Prototype** | 1-2 weeks | Basic FUSE mount, read-only |
| **2. Core features** | 2-3 weeks | Metadata overlay, FLAC support |
| **3. Full parity** | 2-3 weeks | MP3, write support, all fields |
| **4. Testing** | 1-2 weeks | Unit tests, integration tests |
| **5. Optimization** | 1-2 weeks | mmap, async, benchmarking |
**Total: 7-12 weeks**
### Skill Requirements
- Rust fundamentals (ownership, borrowing, lifetimes)
- FUSE protocol knowledge (from Python experience)
- Audio metadata formats (FLAC, ID3)
- Async Rust (Tokio) - optional for Phase 5
---
## 10. Risk Assessment
### Low Risk ✅
| Factor | Why Low Risk |
|--------|--------------|
| FUSE library | fuser is production-proven (AWS) |
| Metadata library | lofty has full mutagen parity |
| Core algorithm | Same logic, different language |
| File format support | FLAC/MP3/OGG all supported |
### Medium Risk ⚠️
| Factor | Mitigation |
|--------|------------|
| Learning curve | Existing Rust experience helps |
| Edge cases | Port Python tests to Rust |
| Async complexity | Start with sync API, add async later |
### Benefits vs Effort
```
Current Python Issues:
├── Memory: OOM on library scan → Fixed by mmap
├── Latency: 200-500ms file open → Fixed by zero-copy
├── GC pauses: 50-2200ms → Eliminated
├── Concurrency: single-threaded → Fixed by async
└── MP3 support: disabled → Implemented properly
Migration Effort: 7-12 weeks
Expected Lifetime: 5+ years
ROI: Highly positive
```
---
## 11. Recommendation
### ✅ **Proceed with Rust Migration**
**Justification:**
1. **10x memory reduction** via mmap (eliminates OOM)
2. **5-10x latency improvement** (eliminates blocking reads)
3. **GC pauses eliminated** (deterministic performance)
4. **100x concurrency** improvement (Tokio async)
5. **Production-proven** ecosystem (fuser + lofty)
6. **Reasonable effort** (7-12 weeks)
### Next Steps
1. **Set up Rust project** with fuser and lofty dependencies
2. **Port FSNode** to Rust VirtualTree
3. **Implement basic FUSE** operations (read, getattr, readdir)
4. **Add metadata overlay** with lofty for FLAC
5. **Add mmap** for zero-copy audio serving
6. **Benchmark** against Python implementation
7. **Add MP3/OGG** support
8. **Add async** with Tokio (optional)
### Dependencies
```toml
[dependencies]
fuser = "0.17"
lofty = "0.21"
memmap2 = "0.9"
tokio = { version = "1", features = ["full"], optional = true }
rusqlite = "0.31" # For beets DB compatibility
```
docs/v2/architecture.md
-1151
View File
File diff suppressed because it is too large Load Diff
docs/v2/development-plan.md
-1387
View File
File diff suppressed because it is too large Load Diff
docs/v2/features/metadata-enrichment-standalone.md
-579
View File
@@ -1,579 +0,0 @@
# Metadata Enrichment (Standalone Mode): Design Doc
**Authors:** Sisyphus
**Status:** Draft
**Last Updated:** 2026-05-18
**Reviewers:** —
**Approvers:** —
**Document Link:** `docs/v2/plans/metadata-enrichment-standalone.md`
**Prerequisites:** [architecture.md](../architecture.md), [week-12-external-metadata.md](week-12-external-metadata.md)
---
## 1. Abstract
When musicfs operates without the music-agregator orchestrator, it should
still be able to enrich file metadata (genres, label, artwork URL, album
type) by querying the metadata-agregator service directly. This document
describes a **built-in metadata provider** compiled into musicfs that
queries metadata-agregator's gRPC `SearchAlbums` endpoint using
artist + album names extracted from file tags. Enrichment is lazy and
non-blocking — file access always returns immediately using embedded
tags, while a background worker enriches metadata asynchronously.
This plan **supersedes** the week-12 plan's approach of embedding
MusicBrainz/Discogs/Last.fm HTTP clients directly into musicfs. Instead,
musicfs delegates all external metadata resolution to metadata-agregator,
which already handles provider APIs, rate limiting, and caching.
## 2. Background
### 2.1. Current State
musicfs extracts audio metadata via symphonia (FLAC, MP3, AAC, OGG,
Opus) and stores it in `AudioMeta`. This metadata is whatever the file
tags contain — typically title, artist, album, year, track number.
The existing plugin system (`musicfs-plugins`) defines a `MetadataPlugin`
trait for external metadata lookup, but:
- No plugins have been implemented yet.
- The plugin system only supports native `.so` and WASM plugins.
- A gRPC client to metadata-agregator would require bundling an async
runtime and tonic inside a `.so` — an awkward fit.
Meanwhile, metadata-agregator is a Go gRPC service that:
- Searches MusicBrainz by artist + album name (`SearchAlbums` RPC).
- Caches results in PostgreSQL.
- Returns rich metadata: genres, cover URL, label, release date, album
type, artist credits.
### 2.2. Pain Points
- musicfs files lack genres, artwork URLs, and label info unless the
original files were meticulously tagged.
- The week-12 plan proposed embedding 4 separate HTTP API clients
(MusicBrainz, Discogs, Last.fm, AcoustID) directly into musicfs,
duplicating what metadata-agregator already does.
- The `MetadataPlugin` trait is designed for `.so`/WASM plugins, which
is wrong for a core infrastructure gRPC client.
## 3. Goals & Non-Goals
### 3.1. Goals
- **G1:** Enrich file metadata with genres, label, album type, and cover
URL by querying metadata-agregator via gRPC.
- **G2:** Never block file access — enrichment happens in background.
- **G3:** Make the provider entirely optional — disabled by default,
musicfs works identically without it.
- **G4:** Respect enrichment source priority so orchestrator pushes
(from the full-system mode) are not overwritten.
### 3.2. Non-Goals
- **NG1:** Embedding MusicBrainz/Discogs/Last.fm HTTP clients directly
into musicfs (metadata-agregator handles this).
- **NG2:** Audio fingerprinting (AcoustID) — deferred to future work.
- **NG3:** Modifying the existing `MetadataPlugin` trait — the built-in
provider is separate from the plugin system.
- **NG4:** Bidirectional communication — musicfs only queries
metadata-agregator, never the reverse.
## 4. Proposed Design
### 4.1. High-Level Architecture
```plantuml
@startuml
!theme plain
skinparam componentStyle rectangle
package "musicfs" as mfs {
component "FUSE Layer\n(readdir/open/read)" as fuse
component "MetadataCache / DB" as db
component "OverlayReader\n(synthesize headers)" as overlay
component "EnrichmentQueue\n(bounded, async)" as queue
component "EnrichmentWorker\n(background)" as worker
}
component "metadata-agregator\nSearchAlbums(query, artist)" as meta
fuse -right-> db : lookup metadata
db -right-> overlay : serve with overlay
fuse -down-> queue : enriched_at NULL?\npush request
queue -down-> worker : dequeue
worker -down-> meta : gRPC:\nSearchAlbums(\n query=album,\n artist=artist)
meta -up-> worker : Album (genres,\nlabel, cover_url)
worker -up-> db : write enriched\nmetadata to overlay
note bottom of meta
metadata-agregator handles:
• MusicBrainz API
• rate limiting
• PostgreSQL cache
end note
note right of fuse
File access is never blocked.
Returns embedded tags immediately.
Enrichment happens async.
end note
@enduml
```
### 4.2. Enrichment Flow
```plantuml
@startuml
!theme plain
skinparam sequenceMessageAlign center
participant "Media Player" as mp
participant "FUSE Layer" as fuse
participant "MetadataCache\n(SQLite)" as db
participant "EnrichmentQueue" as queue
participant "EnrichmentWorker" as worker
participant "metadata-agregator" as meta
== File Access (non-blocking) ==
mp -> fuse : open("/Pink Floyd/The Wall/01 - In the Flesh.flac")
fuse -> db : lookup(virtual_path)
db --> fuse : AudioMeta(artist, album, title, ...)\nenriched_at = NULL
fuse -> queue : try_push(file_id, artist="Pink Floyd", album="The Wall")
note right of queue : non-blocking,\nbounded queue
fuse --> mp : return file handle\n(with embedded tags only)
== Background Enrichment (async) ==
queue -> worker : dequeue(file_id, artist, album)
worker -> worker : check enrichment_source\n(skip if 'orchestrator' or 'provider')
worker -> worker : dedup check:\nalready enriched same album?\n(reuse cached result)
worker -> meta : SearchAlbums(\n query="The Wall",\n artist="Pink Floyd",\n limit=1)
meta --> worker : Album(\n genres=["Progressive Rock", "Art Rock"],\n label="Harvest",\n cover_url="https://...",\n album_type="album")
worker -> db : update_metadata(\n file_id,\n genres, label, cover_url,\n enrichment_source='provider',\n enriched_at=now())
worker -> worker : publish EventBus::FileModified
note over mp : next access sees\nenriched metadata
@enduml
```
### 4.3. Detailed Design
#### 4.3.1. Configuration
Add `[metadata_provider]` section to `config.toml`:
```toml
[metadata_provider]
enabled = false # disabled by default
endpoint = "http://localhost:50051" # metadata-agregator gRPC
timeout_ms = 5000 # per-request timeout
retry_max = 3 # max retries on failure
retry_backoff_ms = 1000 # initial backoff between retries
queue_size = 256 # enrichment queue capacity
```
Config struct addition in `musicfs-core/src/config.rs`:
```rust
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct MetadataProviderConfig {
#[serde(default)]
pub enabled: bool,
#[serde(default = "default_provider_endpoint")]
pub endpoint: String,
#[serde(default = "default_provider_timeout_ms")]
pub timeout_ms: u64,
#[serde(default = "default_retry_max")]
pub retry_max: u32,
#[serde(default = "default_retry_backoff_ms")]
pub retry_backoff_ms: u64,
#[serde(default = "default_queue_size")]
pub queue_size: usize,
}
```
#### 4.3.2. Built-in Metadata Provider
New module in `musicfs-metadata` (not a plugin, compiled in):
```rust
// musicfs-metadata/src/provider.rs
pub struct MetadataAgregatorProvider {
client: MetadataServiceClient<Channel>,
config: MetadataProviderConfig,
}
impl MetadataAgregatorProvider {
pub async fn connect(config: &MetadataProviderConfig)
-> Result<Self>;
/// Query metadata-agregator by artist + album names.
/// Returns enriched metadata if a match is found.
pub async fn lookup(
&self,
artist: &str,
album: &str,
) -> Result<Option<EnrichedMetadata>>;
}
```
The `lookup` method calls `SearchAlbums(query=album, artist=artist,
limit=1)` on metadata-agregator. If a result is returned, it maps
the response to `EnrichedMetadata`:
```rust
pub struct EnrichedMetadata {
pub genres: Vec<String>,
pub label: Option<String>,
pub album_type: Option<String>,
pub cover_url: Option<String>,
pub release_date: Option<String>,
pub total_tracks: Option<u32>,
pub total_discs: Option<u32>,
}
```
#### 4.3.3. ExternalMetadata Extension
Extend the existing `ExternalMetadata` in `musicfs-plugins/src/traits.rs`
to carry richer data:
```rust
pub struct ExternalMetadata {
// existing fields...
pub title: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>, // kept for backward compat
pub year: Option<u32>,
pub track: Option<u32>,
pub disc: Option<u32>,
pub musicbrainz_id: Option<String>,
pub artwork_url: Option<String>,
// new fields
pub genres: Vec<String>,
pub label: Option<String>,
pub album_type: Option<String>,
pub cover_url: Option<String>,
}
```
#### 4.3.4. Database Schema Changes
Add columns to `file_metadata` table in
`musicfs-cache/src/schema.sql`:
```sql
ALTER TABLE file_metadata ADD COLUMN enrichment_source TEXT;
-- 'embedded' | 'provider' | 'orchestrator'
ALTER TABLE file_metadata ADD COLUMN enriched_at INTEGER;
-- unix timestamp, NULL = not enriched
ALTER TABLE file_metadata ADD COLUMN enrichment_attempts INTEGER DEFAULT 0;
-- number of failed enrichment attempts
ALTER TABLE file_metadata ADD COLUMN last_enrichment_error TEXT;
-- last error message, NULL if no error
ALTER TABLE file_metadata ADD COLUMN genres_json TEXT;
-- JSON array: '["Progressive Rock","Art Rock"]'
-- separate from existing `genre` (singular) for backward compat
ALTER TABLE file_metadata ADD COLUMN label TEXT;
ALTER TABLE file_metadata ADD COLUMN album_type TEXT;
ALTER TABLE file_metadata ADD COLUMN cover_url TEXT;
```
> **Note:** The existing `genre TEXT` column (singular) is preserved
> for backward compatibility. `genres_json` stores the full list.
> The singular `genre` field is set to the first genre in the array
> when enriched.
#### 4.3.5. Background Enrichment Queue + Worker
```rust
// musicfs-metadata/src/enrichment.rs
pub struct EnrichmentQueue {
tx: mpsc::Sender<EnrichmentRequest>,
/// Tracks in-flight (artist, album) pairs to prevent duplicate
/// API calls when multiple tracks from the same album are
/// accessed simultaneously.
in_flight: Arc<DashSet<(String, String)>>,
}
struct EnrichmentRequest {
file_id: FileId,
artist: String,
album: String,
}
pub struct EnrichmentWorker {
rx: mpsc::Receiver<EnrichmentRequest>,
provider: Arc<MetadataAgregatorProvider>,
db: Arc<Database>,
event_bus: Arc<EventBus>,
in_flight: Arc<DashSet<(String, String)>>,
config: MetadataProviderConfig,
}
```
##### Enqueue-time dedup
When `EnrichmentQueue::try_push()` is called, it checks the
`in_flight` `DashSet` before pushing. If `(artist, album)` is
already in the set, the request is dropped (the worker will enrich
all files with the same album in one pass). This prevents 12
simultaneous track opens from making 12 identical API calls.
If `try_push` fails because the queue is full, log at WARN level
and increment `enrichment_queue_drops_total` metric.
##### Worker loop (single-threaded, processes one at a time):
1. Dequeue `EnrichmentRequest`.
2. Check `enrichment_attempts` — skip if `>= retry_max`.
3. **Atomic conflict check**: write uses conditional SQL:
```sql
UPDATE file_metadata SET
genres_json = ?, label = ?, album_type = ?, cover_url = ?,
genre = ?, -- first genre for backward compat
enrichment_source = 'provider',
enriched_at = strftime('%s', 'now'),
enrichment_attempts = 0,
last_enrichment_error = NULL
WHERE file_id = ?
AND (enrichment_source IS NULL OR enrichment_source = 'embedded')
```
This prevents the TOCTOU race — if the orchestrator wrote between
dequeue and now, the `WHERE` clause prevents overwrite. The UPDATE
returns rows_affected=0, which the worker treats as "skip, already
enriched by higher-priority source".
4. Deduplicate by (artist, album) — if another file in the same album
was already enriched, reuse the cached `EnrichedMetadata` result
for all files with the same (artist, album) pair.
5. Call `provider.lookup(artist, album)`.
6. On success: execute atomic update (step 3) for all files with this
(artist, album). Publish `EventBus::FileModified` for each updated
file. Remove `(artist, album)` from `in_flight` set.
7. On failure: increment `enrichment_attempts`, set
`last_enrichment_error`. If `attempts < retry_max`, re-enqueue
with exponential backoff (`retry_backoff_ms * 2^attempts`).
If `attempts >= retry_max`, log at WARN and stop retrying.
Remove from `in_flight` set.
##### Shutdown behavior
Queue contents are lost on shutdown. This is acceptable — files will
be re-queued on next access since `enriched_at` is still NULL.
Enrichment is idempotent.
#### 4.3.6. FUSE Integration Point
In the FUSE `readdir` / `getattr` / `open` path
(`musicfs-fuse/src/ops.rs`), after loading `AudioMeta` from DB:
```rust
if metadata_provider.is_enabled()
&& file_meta.enriched_at.is_none()
&& file_meta.enrichment_attempts < config.retry_max
&& file_meta.audio.artist.is_some()
&& file_meta.audio.album.is_some()
{
if let Err(_) = enrichment_queue.try_push(EnrichmentRequest {
file_id: file_meta.id,
artist: file_meta.audio.artist.unwrap(),
album: file_meta.audio.album.unwrap(),
}) {
// Queue full — file will be retried on next access
tracing::warn!(
file_id = ?file_meta.id,
"enrichment queue full, dropping request"
);
metrics::ENRICHMENT_QUEUE_DROPS.inc();
}
// Non-blocking: returns immediately with embedded tags
}
```
The `enrichment_attempts < retry_max` check prevents files that have
permanently failed enrichment (e.g., metadata-agregator has no match)
from being re-queued on every access.
#### 4.3.7. Conflict Resolution
| Source | Priority | Writes When |
|--------|----------|-------------|
| `orchestrator` | Highest | Always overwrites (full-system mode push) |
| `provider` | Medium | Only if current source is NULL or `'embedded'` |
| `embedded` | Lowest | Implicit default from file tag parsing |
Conflict resolution is enforced **atomically at write time** using
conditional SQL (`WHERE enrichment_source IS NULL OR
enrichment_source = 'embedded'`), not at dequeue time. This prevents
the TOCTOU race where the orchestrator writes between the worker's
check and the worker's write.
#### 4.3.8. Proto Changes Required
The existing `UpdateMetadataRequest` in `musicfs.proto` must be
extended to carry the new enrichment fields:
```protobuf
// Add to UpdateMetadataRequest:
optional string label = 40;
optional string album_type = 41;
optional string cover_url = 42;
```
> **Note on genres:** metadata-agregator returns `repeated Genre`
> (objects with `id` + `name`). The provider extracts genre names
> and stores them as a JSON array in `genres_json`. The singular
> `genre` field in `UpdateMetadataRequest` (already exists at
> field 9) is set to the first/primary genre for backward compat.
#### 4.3.9. `cover_url` Usage
`cover_url` is stored in the metadata overlay but is **not used by
musicfs for artwork embedding or display** in this plan. It is
stored for consumption by external tools (e.g., media players that
query musicfs's gRPC `GetMetadata` and fetch artwork themselves).
Artwork download and caching is deferred to future work.
## 5. Cross-Cutting Concerns
### 5.1. Security & Privacy
- gRPC connection to metadata-agregator is plaintext (internal network).
TLS can be added via config if needed.
- No PII involved — only music metadata.
- No API keys stored in musicfs — metadata-agregator handles provider
auth.
### 5.2. Observability
New tracing spans and metrics:
| Metric | Type | Description |
|--------|------|-------------|
| `enrichment_queue_depth` | Gauge | Current queue size |
| `enrichment_queue_drops_total` | Counter | Requests dropped (queue full) |
| `enrichment_inflight_albums` | Gauge | In-flight (artist, album) dedup set size |
| `enrichment_lookups_total` | Counter | Total provider lookups |
| `enrichment_hits_total` | Counter | Successful matches |
| `enrichment_misses_total` | Counter | No match found |
| `enrichment_errors_total` | Counter | Provider errors |
| `enrichment_skipped_total` | Counter | Skipped (higher-priority source already wrote) |
| `enrichment_latency_ms` | Histogram | Lookup latency |
### 5.3. Scalability & Performance
- Queue is bounded (default 256) — backpressure via `try_push`.
- Album-level deduplication: 12 tracks in same album = 1 lookup.
- No impact on file read latency — enrichment is fully async.
- metadata-agregator caches in PostgreSQL, so repeated lookups are
cheap.
### 5.4. Testing Plan
| Test | Type | Validates |
|------|------|-----------|
| `test_provider_connect` | Unit | gRPC connection setup |
| `test_lookup_match` | Unit (mock) | SearchAlbums → EnrichedMetadata mapping |
| `test_lookup_no_match` | Unit (mock) | Graceful handling of empty results, increments attempts |
| `test_enrichment_queue_push` | Unit | Queue push + in_flight dedup |
| `test_enrichment_queue_full_drops` | Unit | try_push fails gracefully, logs, increments metric |
| `test_enrichment_worker_writes_db` | Integration | DB write after lookup |
| `test_enrichment_atomic_conflict` | Integration | Orchestrator writes between dequeue and worker write → worker does NOT overwrite |
| `test_enrichment_retry_backoff` | Unit | Failed attempts increment counter, exponential backoff |
| `test_enrichment_max_attempts_stop` | Unit | After retry_max failures, file not re-queued |
| `test_config_disabled` | Unit | No queue/worker when disabled |
| `test_album_dedup_simultaneous` | Integration | 12 tracks opened at once → 1 API call |
| `test_genre_backward_compat` | Unit | genres_json stored as array, genre set to first entry |
## 6. Alternatives Considered
### 6.1. Native .so Plugin
Rejected. Requires bundling a separate async runtime + tonic gRPC
stack inside a dynamically loaded library. ABI instability, duplicate
runtimes, and deployment complexity outweigh the "purity" of using the
plugin system.
### 6.2. Direct MusicBrainz/Discogs/Last.fm HTTP Clients (week-12 plan)
Rejected. metadata-agregator already handles these providers with rate
limiting, caching, and deduplication. Embedding HTTP clients in musicfs
would duplicate this work and couple musicfs to specific provider APIs.
### 6.3. WASM Plugin
Rejected. WASI networking is immature. gRPC over WASM adds unnecessary
latency and complexity.
### 6.4. On-Demand Blocking Lookup
Rejected. Blocking file access while waiting for a gRPC response would
cause latency spikes and kill media player UX. Background async is the
only acceptable approach.
## 7. Implementation Plan
### Phase 1: Foundation (Day 1)
- [ ] Add `MetadataProviderConfig` to config.rs
- [ ] Add DB schema columns: `enrichment_source`, `enriched_at`,
`enrichment_attempts`, `last_enrichment_error`, `genres_json`,
`label`, `album_type`, `cover_url`
- [ ] Add `label`, `album_type`, `cover_url` fields to
`UpdateMetadataRequest` in `musicfs.proto`
- [ ] Extend `ExternalMetadata` struct
- [ ] Update `config.example.toml`
### Phase 2: Provider + Worker (Day 1–2)
- [ ] Implement `MetadataAgregatorProvider` (gRPC client wrapper)
- [ ] Implement `EnrichmentQueue` with `DashSet` in-flight dedup
- [ ] Implement `EnrichmentWorker` with:
- Atomic conditional write (`WHERE enrichment_source IS NULL OR ...`)
- Retry tracking (`enrichment_attempts`, exponential backoff)
- Album-level result caching
- [ ] Add queue drop logging + metrics
- [ ] Wire into startup (musicfs-cli) — conditional on config
### Phase 3: Integration + Tests (Day 2)
- [ ] Wire enrichment trigger in FUSE getattr/readdir path
(with `enrichment_attempts < retry_max` guard)
- [ ] Write unit tests: atomic conflict, queue drops, retry backoff,
max attempts, genre backward compat
- [ ] Write integration test: 12-track simultaneous dedup
- [ ] Write integration test with in-memory DB + mock gRPC server
- [ ] Update architecture.md with metadata provider component
## 8. Glossary / References
| Term | Definition |
|------|------------|
| metadata-agregator | Go gRPC service that searches MusicBrainz and caches results in PostgreSQL |
| Enrichment | Adding genres, label, artwork URL to file metadata beyond what's in file tags |
| Overlay | musicfs mechanism for serving modified metadata without changing origin files |
| `AudioMeta` | Core metadata struct extracted from file tags by symphonia |
| `ExternalMetadata` | Metadata returned by external providers (plugin trait) |
| `enrichment_source` | Tracks who last wrote metadata: `embedded`, `provider`, or `orchestrator` |
- [metadata-agregator proto](../../../../metadata-agregator/proto/metadata/v1/metadata.proto)
- [musicfs-plugins traits](../../crates/musicfs-plugins/src/traits.rs)
- [musicfs-cache overlay](../../crates/musicfs-cache/src/overlay.rs)
- [architecture.md](../architecture.md)
docs/v2/features/metadata-overlay.md
-1010
View File
File diff suppressed because it is too large Load Diff
docs/v2/features/mkdir.md
-105
View File
@@ -1,105 +0,0 @@
**Date**: 2026-05-17
**Status**: Shipped
# Feature: Create Directory (mkdir)
## Overview
MusicFS supports creating directories in the virtual filesystem. This enables organizing files into custom folder structures beyond the auto-generated metadata-based layout.
## Behavior
### Basic Usage
```bash
mkdir "/mnt/music/New Artist"
mkdir "/mnt/music/New Artist/New Album"
```
- Creates empty directory at specified path
- Parent directory must exist
- Standard POSIX semantics
### Nested Directories
```bash
# This works (shell handles -p)
mkdir -p "/mnt/music/A/B/C"
# Equivalent to:
mkdir "/mnt/music/A"
mkdir "/mnt/music/A/B"
mkdir "/mnt/music/A/B/C"
```
The `-p` flag is handled by the shell, which makes multiple `mkdir` syscalls.
### Brace Expansion
```bash
# Shell expands this to multiple mkdir calls
mkdir "/mnt/music/Artist/{Album1,Album2,Album3}"
# Equivalent to:
mkdir "/mnt/music/Artist/Album1"
mkdir "/mnt/music/Artist/Album2"
mkdir "/mnt/music/Artist/Album3"
```
Brace expansion is shell functionality, not filesystem.
## Error Codes
| Condition | Error |
|-----------|-------|
| Parent doesn't exist | `ENOENT` |
| Path already exists | `EEXIST` |
## Persistence
**Empty directories persist across remounts.**
- User-created directories are stored in the `directories` table
- On mount, directories are restored from database
- Directories survive even when empty
## Use Cases
### Organizing Downloads
```bash
# Create structure
mkdir "/mnt/music/Unsorted"
mkdir "/mnt/music/Unsorted/2026"
# Move untagged files
mv "/mnt/music/Unknown Artist/Unknown Album/"*.flac "/mnt/music/Unsorted/2026/"
```
### Custom Collections
```bash
# Create playlist-like structure
mkdir "/mnt/music/_Playlists"
mkdir "/mnt/music/_Playlists/Road Trip"
# Move tracks (they'll still be in original location too - wait, no they won't)
# Note: mv moves, doesn't copy
```
## Implementation
| Component | File |
|-----------|------|
| Tree | `crates/musicfs-cache/src/tree.rs` |
| FUSE | `crates/musicfs-fuse/src/filesystem.rs` |
### Key Functions
- `VirtualTree::mkdir()` - Create directory node in tree
- `Filesystem::mkdir()` - FUSE operation handler
## Limitations
- **No permissions**: Mode/umask parameters are ignored (always 0755)
- **No ownership**: UID/GID set to mounting user
docs/v2/features/mv.md
-94
View File
@@ -1,94 +0,0 @@
**Date**: 2026-05-17
**Status**: Shipped
# Feature: Move/Rename (mv)
## Overview
MusicFS supports moving and renaming files and directories within the virtual filesystem. Moves are persisted to the SQLite database and survive remounts.
## Behavior
### File Rename
```bash
mv "/mnt/music/Artist/Album/old.flac" "/mnt/music/Artist/Album/new.flac"
```
- Renames file within same directory
- Updates `virtual_path` in database
- Original file on origin is unchanged
### File Move
```bash
mv "/mnt/music/Artist/Album/track.flac" "/mnt/music/Other Artist/Other Album/track.flac"
```
- Moves file to different directory
- **Requires target directory to exist** (use `mkdir` first)
- Returns `ENOENT` if target parent doesn't exist
### Directory Rename
```bash
mv "/mnt/music/Old Artist" "/mnt/music/New Artist"
```
- Renames directory and all descendants
- All files under the directory have their `virtual_path` updated in DB
- Single atomic operation
### Directory Move
```bash
mv "/mnt/music/Artist/Album" "/mnt/music/Other Artist/Album"
```
- Moves directory subtree to new parent
- **Requires target parent to exist**
- Returns `ENOENT` if target parent doesn't exist
## Error Codes
| Condition | Error |
|-----------|-------|
| Source doesn't exist | `ENOENT` |
| Target already exists | `EEXIST` |
| Target parent doesn't exist | `ENOENT` |
| Source is file but treated as dir | `EISDIR` |
| Source is dir but treated as file | `ENOTDIR` |
## Persistence
- File moves: `virtual_path` column updated in `files` table
- Directory moves: All matching `virtual_path` entries updated with new prefix
- User directories: Tracked in separate `directories` table
- Changes persist across unmount/remount cycles
On mount, the CLI:
1. Scans origin files
2. For each file, checks DB for stored `virtual_path` (by origin_id + real_path)
3. Uses stored path if found, otherwise generates from metadata
4. Restores user-created directories from `directories` table
## Limitations
- **Read-only content**: File contents cannot be modified, only paths
- **No cross-origin moves**: All files remain on their original origin
- **No overwrite**: Moving to existing path fails (no implicit delete)
## Implementation
| Component | File |
|-----------|------|
| Database | `crates/musicfs-cache/src/db.rs` |
| Tree | `crates/musicfs-cache/src/tree.rs` |
| FUSE | `crates/musicfs-fuse/src/filesystem.rs` |
### Key Functions
- `Database::update_virtual_path()` - Update single file path
- `Database::rename_directory()` - Bulk update paths with prefix
- `VirtualTree::rename_file()` - Move file node in tree
- `VirtualTree::rename_directory()` - Move directory subtree
docs/v2/features/rm.md
-166
View File
@@ -1,166 +0,0 @@
**Date**: 2026-05-17
**Status**: Shipped
# Feature: Remove (rm)
## Overview
MusicFS supports removing files and directories. Deleted files are moved to a virtual `/.trash/` directory and can be restored. The trash is browsable — users can manually move files out.
## Behavior
### Remove File
```bash
rm "/mnt/music/Artist/Album/track.flac"
```
- File moves to `/.trash/Artist/Album/track.flac`
- Original directory structure preserved in trash
- File still accessible via `/.trash/` path
- Database marks file as `trashed=1` with original path stored
### Remove Empty Directory
```bash
rmdir "/mnt/music/Empty Folder"
```
- Removes empty directory from tree
- Removes from `directories` table if user-created
- Fails with `ENOTEMPTY` if directory has children
### Remove Directory Recursively
```bash
rm -rf "/mnt/music/Artist"
```
- Shell handles recursion (depth-first unlink + rmdir)
- All files moved to `/.trash/Artist/...`
- Empty directories removed after files are trashed
## The `.trash/` Directory
Deleted files live in `/.trash/` with their original path structure:
```
/.trash/
├── Artist/
│ └── Album/
│ ├── track1.flac
│ └── track2.flac
└── Other Artist/
└── song.flac
```
### Browse Trash
```bash
ls "/.trash/"
ls "/.trash/Artist/Album/"
```
### Manual Restore
```bash
# Move file back manually - trashed flag is automatically cleared
mv "/.trash/Artist/Album/track.flac" "/Artist/Album/"
```
When moving a file out of `/.trash/`, the database `trashed` flag is automatically cleared.
## CLI Commands
All trash commands require either `--config` or `--cache-dir`:
```bash
musicfs trash -c config.toml <command>
musicfs trash --cache-dir ./dev/cache/musicfs <command>
```
### List Deleted Files
```bash
musicfs trash -c config.toml list
musicfs trash -c config.toml list --origin local-storage
musicfs trash -c config.toml list --since 7d
musicfs trash -c config.toml list --path "/Artist"
```
Output shows index, deletion time, and original path.
### Restore Files
```bash
# Restore single file or folder
musicfs trash -c config.toml restore "/Artist/Album/track.flac"
# Restore entire folder recursively
musicfs trash -c config.toml restore "/Artist"
# Restore everything
musicfs trash -c config.toml restore --all
```
CLI restore writes paths to a pending restore file and sends SIGHUP to the daemon.
The daemon processes pending restores and moves files back from `/.trash/`.
### Empty Trash
```bash
# Permanently delete all trashed files
musicfs trash -c config.toml empty
# Delete old items only
musicfs trash -c config.toml empty --older-than 30d
# Delete by path pattern
musicfs trash -c config.toml empty --pattern "/Artist"
```
**Warning:** Empty permanently removes files from MusicFS database. Origin files are unaffected.
## Error Codes
| Condition | Error |
|-----------|-------|
| Path doesn't exist | `ENOENT` |
| `rm` on directory (without `-r`) | `EISDIR` |
| `rmdir` on file | `ENOTDIR` |
| `rmdir` on non-empty directory | `ENOTEMPTY` |
| `rmdir` on `/.trash/` | `EPERM` |
## Database Schema
Files table extended with trash columns:
```sql
trashed INTEGER NOT NULL DEFAULT 0,
original_path TEXT,
trashed_at INTEGER
```
Partial index for efficient trash queries:
```sql
CREATE INDEX idx_files_trashed ON files(trashed) WHERE trashed = 1;
```
## How It Works
1. **Delete (`rm`)**: FUSE `unlink` moves file to `/.trash/`, marks `trashed=1` in DB
2. **Manual restore (`mv`)**: Moving out of `/.trash/` automatically clears `trashed` flag
3. **CLI restore**: Writes pending paths, sends SIGHUP to daemon, daemon processes restores
4. **Empty**: Deletes matching records from database
## Persistence
- Trashed files persist across remounts (stored in `/.trash/` subtree)
- Files marked with `trashed=1`, `original_path`, `trashed_at` in database
- PID file at `{cache_dir}/musicfs.pid` for CLI→daemon communication
## Limitations
- **No hard delete of remote files**: Origin content is never modified
- **Trash uses virtual space**: Files still in tree under `/.trash/` until emptied
- **CLI restore requires running daemon**: Manual `mv` works without daemon
docs/v2/mvp-performance-review.md
-239
View File
@@ -1,239 +0,0 @@
# MusicFS MVP Performance Review
**Date**: 2026-05-12
**Test Data**: Metallica - 72 Seasons (12 FLAC tracks, 625MB, 16-bit/44.1kHz)
**Origin**: Local filesystem (Docker volume)
**System**: Linux, NixOS
---
## Executive Summary
**Phase 1 MVP is functional** - the system mounts, browses, and reads files successfully. Audio playback works with valid FLAC headers served. However, there's a **critical gap** between the architecture specification and current implementation regarding content chunking.
---
## Benchmark Results
### Throughput Comparison
| Metric | Direct FS | MusicFS Cold | MusicFS Warm | Target (Spec) | Status |
|--------|-----------|--------------|--------------|---------------|--------|
| Single file read (64MB) | 0.022s (3 GB/s) | 0.035s (1.8 GB/s) | 0.020s (3.2 GB/s) | >500 MB/s | ✅ |
| Full album read (625MB) | 0.149s (4.2 GB/s) | 0.274s (2.3 GB/s) | 0.211s (3.0 GB/s) | >500 MB/s | ✅ |
### Metadata Operations
| Operation | Result | Target (Spec) | Status |
|-----------|--------|---------------|--------|
| Root listing | 0.006s | <10ms | ✅ |
| Full tree traversal (12 files) | 0.007s | <50ms | ✅ |
| stat() per operation | 0.003s | <1ms | ⚠️ |
| 4KB small reads (per op) | 0.006s | <1ms | ⚠️ |
| Random seek 1MB | 0.008-0.015s | <50ms | ✅ |
| Mount time | ~8ms | <500ms | ✅ |
### Cache Performance
| Metric | Value |
|--------|-------|
| Cache speedup (single file) | 1.75x |
| Cache speedup (full album) | 1.30x |
| Cache size | 25MB |
| Chunk count | 12 |
| Expected cache size | 625MB |
### FUSE Overhead
| Scenario | Overhead vs Direct |
|----------|-------------------|
| Single file cold cache | 59% slower |
| Single file warm cache | 9% faster* |
| Full album cold cache | 84% slower |
| Full album warm cache | 42% slower |
*Warm cache appears faster due to OS page cache effects on both paths.
---
## What's Working Well ✅
### 1. Mount Performance
- Mount completes in ~8ms (spec: <500ms) — **62x better than target**
- O(1) mount time achieved — no file scanning blocks mount
- Lazy loading working as designed per architecture section 4.3.1
### 2. Virtual Tree Organization
- Correct Artist/Album/Track hierarchy derived from metadata
- Example path: `/Metallica/72 Seasons/01. 72 Seasons.flac`
- Special character sanitization working (`/`, `\`, `:`, etc.)
### 3. File Reading
- Valid FLAC headers served (`fLaC` magic bytes verified)
- Sequential reads work correctly
- Random access (seek) functional
- Concurrent reads from multiple processes work
### 4. FUSE Integration
- Read-only enforcement (EROFS returned on write attempts)
- Proper inode assignment and file attributes
- AllowOther mount option working
- Clean unmount via fusermount3
### 5. Throughput
- Exceeds 500 MB/s target significantly (2-3 GB/s achieved)
- Parallel reads scale appropriately (4 files in 0.060s)
---
## Critical Issues 🔴
### Issue 1: Incomplete File Caching
**Symptom**: Cache is 25MB instead of expected 625MB (12 files × ~2MB each instead of full files)
**Root Cause**: In `fetcher.rs:74`:
```rust
let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await?;
```
And in `local.rs:96-98`:
```rust
let mut buffer = vec![0u8; size as usize];
let bytes_read = file.read(&mut buffer).await?;
buffer.truncate(bytes_read);
```
`tokio::fs::File::read()` reads **up to** buffer size but returns when the kernel buffer is exhausted (~2MB typical). Only first ~2MB of each file is being cached.
**Impact**:
- Subsequent reads beyond 2MB offset hit origin every time
- No cache benefit for majority of file content
- Cache eviction policy not being exercised
**Required Fix**: Use `read_to_end()` or loop until all bytes read:
```rust
let mut buffer = Vec::with_capacity(size as usize);
file.read_to_end(&mut buffer).await?;
```
### Issue 2: No CDC Chunking Implemented
**Architecture Spec** (Section 4.3.2):
> "All file content is stored as content-addressed chunks... Avg chunk: 64KB, Min: 16KB, Max: 256KB"
**Current Implementation**: Each file stored as ONE chunk (no FastCDC integration)
**Impact**:
- No content deduplication possible
- Delta sync impossible (FR-11.2 unmet)
- Cache efficiency severely reduced for similar files
---
## Architecture Gaps 🟡
| Spec Requirement | Current State | Gap |
|------------------|---------------|-----|
| CDC chunking (64KB avg) | No chunking | Missing FastCDC integration |
| Delta sync (>90% bandwidth reduction) | Not implemented | Requires CDC first |
| Deduplication (FR-20) | Not implemented | Requires CDC first |
| Search engine (tantivy) | Not implemented | Phase 3 scope |
| gRPC Control API | Not implemented | Phase 4 scope |
| Multi-origin federation | Single origin only | Phase 2 scope |
| Metadata persistence (SQLite) | In-memory HashMap | Missing persistence |
---
## Performance Analysis
### Why Warm Cache Appears Faster Than Direct FS
The warm cache shows 3.2 GB/s vs direct 3.0 GB/s because:
1. OS page cache is warm for both MusicFS chunks AND origin files
2. Both measurements are essentially hitting RAM, variance expected
3. MusicFS chunks may have slightly better cache locality
### stat() Latency Above Target
Current: 3ms per stat() vs target <1ms
Possible causes:
1. `RwLock<VirtualTree>` contention overhead
2. HashMap lookup plus FUSE context switch
3. Measurement includes full round-trip through FUSE
Mitigation options:
- Consider lock-free concurrent data structures
- Implement finer-grained locking
- Cache hot inodes in separate fast-path structure
---
## Recommendations
### Immediate Fixes (Before Phase 2)
1. **Fix file reading** — Use `read_to_end()` or implement proper streaming read loop
2. **Add CDC chunking** — Integrate FastCDC per architecture spec section 4.3.2
3. **Persist metadata** — Move from in-memory HashMap to SQLite as specified
### Phase 2 Priorities
1. Complete CDC chunking implementation (prerequisite for delta sync)
2. Add SQLite metadata persistence (FR-7.2)
3. Implement multi-origin support (FR-13)
### Testing Gaps to Address
1. No automated E2E tests for real FUSE operations
2. No stress testing with concurrent access patterns
3. No large library testing (target: 1M+ files per NFR-3.1)
4. No offline mode testing (origin unavailable scenarios)
---
## Test Environment Details
```
Origin Path: /home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/
Mount Point: /tmp/musicfs-benchmark/mount
Cache Dir: /tmp/musicfs-benchmark/cache
Binary: target/release/musicfs (via nix develop)
Files:
01. 72 Seasons.flac 64MB
02. Shadows Follow.flac 50MB
03. Screaming Suicide.flac 45MB
04. Sleepwalk My Life Away.flac 54MB
05. You Must Burn!.flac 57MB
06. Lux Æterna.flac 27MB
07. Crown Of Barbed Wire.flac 46MB
08. Chasing Light.flac 55MB
09. If Darkness Had A Son.flac 51MB
10. Too Far Gone_.flac 37MB
11. Room Of Mirrors.flac 45MB
12. Inamorata.flac 89MB
Total: 625MB, 12 tracks
```
---
## Conclusion
**The MVP demonstrates core functionality works** — mounting, browsing, and reading audio files through FUSE. Throughput performance exceeds targets significantly.
However, **the cache implementation is incomplete**:
- Only ~4% of file content is being cached (25MB/625MB)
- No CDC chunking means no deduplication or delta sync capability
- Architecture requirements FR-8.2, FR-11.2, FR-20 are unmet
**Recommendation**: Fix the file reading issue and add CDC chunking before proceeding to Phase 2. The architecture is sound; implementation needs to catch up to specification.
---
## References
- [Architecture Specification](architecture.md) — Section 4.3.2 (CAS), Section 4.3.5 (Read Flow)
- [Requirements Specification](requirements.md) — FR-8 (Content Cache), FR-11 (Delta Sync), FR-20 (CAS)
- [Week 4b Plan](plans/week-04b-origin-connector.md) — ContentFetcher implementation
docs/v2/plans/logging-comprehensive.md
-982
View File
@@ -1,982 +0,0 @@
# Comprehensive Logging Plan
**Goal**: Add production-grade logging with trace-level observability, file rotation, and systemd integration
**Effort**: ~10-12 hours
**Dependencies**: Existing libraries only (no custom code)
> **Review Status**: Reviewed by Oracle - all gaps addressed
---
## Libraries Used
| Need | Library | Status |
|------|---------|--------|
| Instrumentation | `tracing` | Already in workspace |
| Subscriber/filtering | `tracing-subscriber` | Already in workspace |
| File rotation | `tracing-appender` | Add to workspace |
| systemd journal | `tracing-journald` | Add to workspace |
| Compression | `logrotate` (Linux tool) | Config file only |
---
## Phase 1: Config & Dependencies (2 hours)
### 1.1 Add dependencies to workspace
```toml
# Cargo.toml [workspace.dependencies]
tracing-appender = "0.2"
tracing-journald = "0.3"
```
```toml
# crates/musicfs-cli/Cargo.toml
tracing-appender.workspace = true
tracing-journald.workspace = true
```
### 1.2 Add LoggingConfig to config.rs
```rust
// crates/musicfs-core/src/config.rs
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Config {
pub mount_point: PathBuf,
pub cache_dir: PathBuf,
pub origins: Vec<OriginConfig>,
#[serde(default)]
pub cache: CacheConfig,
#[serde(default)]
pub health: HealthConfig,
#[serde(default)]
pub logging: LoggingConfig, // NEW
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LoggingConfig {
#[serde(default = "default_log_dir")]
pub log_dir: PathBuf,
#[serde(default)]
pub json_output: bool,
#[serde(default = "default_true")]
pub journald: bool,
#[serde(default = "default_log_level")]
pub level: String,
}
impl Default for LoggingConfig {
fn default() -> Self {
Self {
log_dir: default_log_dir(),
json_output: false,
journald: true,
level: default_log_level(),
}
}
}
fn default_log_dir() -> PathBuf {
PathBuf::from("/var/log/musicfs")
}
fn default_log_level() -> String {
"musicfs=info,warn".to_string()
}
fn default_true() -> bool {
true
}
```
### 1.3 Expand init_logging() in main.rs
```rust
// crates/musicfs-cli/src/main.rs
use tracing_appender::non_blocking::WorkerGuard;
use tracing_subscriber::{fmt, prelude::*, EnvFilter};
fn init_logging(config: &LoggingConfig) -> Result<WorkerGuard> {
std::fs::create_dir_all(&config.log_dir)?;
// File layer with daily rotation
let file_appender = tracing_appender::rolling::daily(&config.log_dir, "musicfs.log");
let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
let file_layer = if config.json_output {
fmt::layer()
.json()
.with_writer(non_blocking)
.with_ansi(false)
.boxed()
} else {
fmt::layer()
.with_writer(non_blocking)
.with_ansi(false)
.boxed()
};
// Journald layer (Linux only)
#[cfg(target_os = "linux")]
let journald_layer = if config.journald {
tracing_journald::layer()
.ok()
.map(|l| l.with_syslog_identifier("musicfs".to_string()))
} else {
None
};
// Stderr layer for interactive use
let stderr_layer = fmt::layer()
.with_writer(std::io::stderr)
.compact();
// Filter from config or env
let filter = EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new(&config.level));
// Compose
let subscriber = tracing_subscriber::registry()
.with(filter)
.with(file_layer)
.with(stderr_layer);
#[cfg(target_os = "linux")]
let subscriber = subscriber.with(journald_layer);
subscriber.init();
tracing::info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting");
Ok(guard)
}
```
### 1.4 Add logrotate config
```bash
# dist/logrotate.d/musicfs
/var/log/musicfs/*.log {
daily
rotate 30
compress
delaycompress
missingok
notifempty
create 0640 musicfs musicfs
}
```
---
## Phase 2: Add tracing to musicfs-core (1 hour)
### 2.1 Add dependency
```toml
# crates/musicfs-core/Cargo.toml
[dependencies]
tracing.workspace = true # ADD THIS
```
### 2.2 Instrument core modules
| File | What to Add |
|------|-------------|
| `config.rs` | Log config file loading, parse errors |
| `credentials.rs` | Log credential loading (redacted values) |
| `events.rs` | Log event publishing with counts |
---
## Phase 3: Instrument Hot Paths (4 hours)
### Priority order by impact
| Crate | Files | What to Add |
|-------|-------|-------------|
| musicfs-fuse | `filesystem.rs` | `#[instrument]` on all FUSE ops, trace at decision points |
| musicfs-origins | `failover.rs`, `health.rs`, `router.rs` | Retry loops, state transitions, selection logic |
| musicfs-cache | `tree.rs`, `metadata.rs` | Tree mutations, cache hit/miss |
| musicfs-cas | `reader.rs`, `store.rs` | Chunk operations, dedup decisions |
| musicfs-sync | `delta.rs`, `watcher.rs` | Change detection, file events |
### Instrumentation patterns
```rust
// Function level - add to all public async functions
#[tracing::instrument(level = "debug", skip(self), fields(path = %path))]
pub async fn read(&self, path: &str) -> Result<Bytes> {
// ...
}
// Decision points - add trace! at match/if branches
match result {
Ok(data) => {
tracing::trace!(bytes = data.len(), "read success");
data
}
Err(e) => {
tracing::trace!(error = %e, "read failed");
return Err(e);
}
}
// State changes - use info! for important transitions
tracing::info!(old = ?old_status, new = ?new_status, origin = %id, "health changed");
// Cache operations
tracing::trace!(hit = true, fresh = true, "cache hit");
tracing::trace!(hit = false, "cache miss");
```
### FUSE operations (filesystem.rs) - highest priority
| Operation | Level | Fields |
|-----------|-------|--------|
| `lookup()` | debug | parent, name, result_ino |
| `getattr()` | debug | ino, file_type |
| `readdir()` | debug | ino, entry_count |
| `read()` | debug | ino, offset, size, bytes_read |
| `open()` | debug | ino, flags |
| `release()` | trace | ino |
### Origin operations - critical for debugging
| Function | Level | Fields |
|----------|-------|--------|
| `read_with_failover()` | debug | path, origins_tried, success |
| `read_with_retry()` | trace | origin, attempt, success |
| `check_health()` | debug | origin, old_status, new_status |
| `select_origin()` | trace | candidates, selected, reason |
---
## Phase 4: Update Production Files (1 hour)
### 4.1 Update systemd service
```ini
# dist/musicfs.service (add these lines)
Environment="RUST_LOG=musicfs=info,warn"
StandardOutput=journal
StandardError=journal
SyslogIdentifier=musicfs
RateLimitIntervalSec=30s
RateLimitBurst=1000
```
### 4.2 Example config.toml
```toml
# dist/config.example.toml
mount_point = "/mnt/music"
cache_dir = "/var/cache/musicfs"
[logging]
log_dir = "/var/log/musicfs"
json_output = true
journald = true
level = "musicfs=info,warn"
[cache]
metadata_cache_mb = 100
content_cache_gb = 10
[health]
check_interval_secs = 30
timeout_ms = 5000
[[origins]]
id = "local"
origin_type = "local"
priority = 1
path = "/srv/music"
```
---
## Detailed Log Locations by Level
### ERROR Level (25+ locations) - Unrecoverable Failures
| File | Line | Log Message |
|------|------|-------------|
| `musicfs-grpc/src/webhook.rs` | 43 | `error!("Failed to initialize webhook HTTP client: {error}")` |
| `musicfs-grpc/src/webhook.rs` | 133 | `error!("Invalid HMAC secret key for webhook signature: {error}")` |
| `musicfs-plugins/src/manager.rs` | 272 | `error!("Plugin manager initialization failed: {error}")` |
| `musicfs-plugins/src/wasm.rs` | 142,183 | `error!("WASM plugin host initialization failed: {error}")` |
| `musicfs-search/src/index.rs` | 211,217 | `error!("Search index corrupted: failed to deserialize at position {pos}")` |
| `musicfs-cas/src/store.rs` | 105 | `error!("CAS chunk not found: {hash} - possible data loss")` |
| `musicfs-cas/src/store.rs` | 124-131 | `error!("CAS integrity check failed: expected {expected}, got {actual}")` |
| `musicfs-fuse/src/filesystem.rs` | 103 | `error!("Failed to mount filesystem at {mountpoint}: {error}")` |
| `musicfs-origins/src/failover.rs` | 76 | `error!("No origins available for path {path}")` |
| `musicfs-origins/src/failover.rs` | 125,186 | `error!("Max retries ({max_attempts}) exceeded for origin {origin_id}")` |
| `musicfs-origins/src/nfs.rs` | 63 | `error!("NFS stale file handle after {max_retries} retries for {path}")` |
| `musicfs-cas/src/reader.rs` | 75 | `error!("File manifest not found for file_id {file_id}")` |
| `musicfs-cas/src/fetcher.rs` | 60,68 | `error!("File/Origin not found for file_id {file_id}")` |
| `musicfs-search/src/indexer.rs` | 44,56 | `error!("Search indexer/commit failed: {error}")` |
| `musicfs-sync/src/watcher.rs` | 36,59,63 | `error!("Watcher failed for origin {origin_id}: {error}")` |
### WARN Level (50+ locations) - Recoverable Issues
| Category | File | Line | Log Message |
|----------|------|------|-------------|
| **Retry Logic** | `failover.rs` | 90 | `warn!("Origin {origin_id} failed: {error}, trying next (attempt {n}/{total})")` |
| **Retry Logic** | `failover.rs` | 111-118 | `warn!("Retrying origin {origin_id} after {delay:?} (attempt {n}/{max})")` |
| **Retry Logic** | `nfs.rs` | 47-52 | `warn!("NFS stale handle for {path} (attempt {n}/{max}), retrying")` |
| **Retry Logic** | `smb.rs` | 45 | `warn!("SMB connection lost (ENOTCONN), retrying (attempt {n}/{max})")` |
| **Retry Logic** | `webhook.rs` | 94-108 | `warn!("Webhook delivery failed to {url} (attempt {n}/{max}): {error}")` |
| **Fallback** | `failover.rs` | 70-73 | `warn!("No healthy origins for {path}, using fallback {origin_id}")` |
| **Timeout** | `smb.rs` | 107-109 | `warn!("SMB health check timed out after 5s for {origin_id}")` |
| **Timeout** | `nfs.rs` | 104-106 | `warn!("NFS health check timed out after 5s for {origin_id}")` |
| **Timeout** | `prefetch.rs` | 91 | `warn!("Prefetch event receive timed out after 1s")` |
| **Health** | `health.rs` | 209 | `warn!("Origin {origin_id} is degraded (failures: {count})")` |
| **Health** | `health.rs` | 217-220 | `warn!("Origin {origin_id} is now unhealthy after {n} consecutive failures")` |
| **Remote FS** | `smb.rs` | 118 | `warn!("SMB watch using inotify on {share_path} - may be unreliable")` |
| **Remote FS** | `nfs.rs` | 115 | `warn!("NFS watch using inotify on {mount_point} - may be unreliable")` |
| **Plugin** | `manager.rs` | 152 | `warn!("Failed to load plugin from {path}: {error}")` |
| **Plugin** | `manager.rs` | 193-194 | `warn!("Failed to unload plugin {plugin_id}: {error}")` |
| **Prefetch** | `prefetch.rs` | 97 | `warn!("Failed to record access pattern for {file_id}: {error}")` |
| **Prefetch** | `prefetch.rs` | 159-161 | `warn!("Prefetch skipped: concurrency limit reached ({max})")` |
| **Search** | `indexer.rs` | 49 | `warn!("Search indexer event receive error: {error}")` |
| **Search** | `indexer.rs` | 82 | `warn!("No metadata found for file {path}, skipping indexing")` |
| **Collections** | `collections.rs` | 146,180 | `warn!("Failed to save/delete collection {name}: {error}")` |
### INFO Level (35+ locations) - Lifecycle & Major Operations
| Category | File | Line | Log Message |
|----------|------|------|-------------|
| **Lifecycle** | `main.rs` | 118 | `info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting")` |
| **Lifecycle** | `filesystem.rs` | 94 | `info!("Mounting MusicFS at {:?}", mountpoint)` |
| **Lifecycle** | `filesystem.rs` | 154 | `info!("MusicFS initialized")` |
| **Lifecycle** | `filesystem.rs` | 159 | `info!("MusicFS destroyed")` |
| **Origin** | `registry.rs` | 28 | `info!("Registering origin {} with priority {}", id, priority)` |
| **Origin** | `registry.rs` | 36 | `info!("Unregistering origin {}", id)` |
| **Origin** | `watcher.rs` | 65 | `info!("Watching origin {} at {:?}", origin_id, path)` |
| **Config** | `main.rs` | 127 | `info!("Cache directory: {:?}", cache_dir)` |
| **Config** | `main.rs` | 141 | `info!("CAS store initialized")` |
| **Config** | `store.rs` | 51 | `info!("CAS store opened: {} chunks, {} bytes", count, size)` (ADD) |
| **Sync** | `main.rs` | 150,152 | `info!("Scanning music files...")` / `info!("Found {} music files", count)` |
| **Sync** | `delta.rs` | 104 | `info!("Delta complete: {} added, {} removed, {} modified", a, r, m)` |
| **Sync** | `delta.rs` | 63 | `info!("Sync started for origin {}", origin_id)` (ADD) |
| **Index** | `main.rs` | 160 | `info!("Virtual tree built")` |
| **Index** | `indexer.rs` | 62 | `info!("Indexer stopping")` |
| **Index** | `indexer.rs` | 114 | `info!("Indexed {} files", count)` |
| **Index** | `index.rs` | 170 | `info!("Search index committed")` |
| **Health** | `health.rs` | 202 | `info!("Origin {} is now healthy", id)` |
| **Health** | `health.rs` | 150 | `info!("Health monitor started with interval {:?}", interval)` (ADD) |
| **Plugin** | `manager.rs` | 127 | `info!("Initializing plugin system")` |
| **Plugin** | `manager.rs` | 150 | `info!("Loaded plugin '{}' with id {:?}", name, id)` |
| **Plugin** | `manager.rs` | 256 | `info!("Shutting down plugin system")` |
| **Cache** | `prefetch.rs` | 123 | `info!("Prefetch engine stopped")` |
| **Cache** | `prefetch.rs` | 174 | `info!("Prefetched {:?}: {} chunks, {} bytes", file_id, chunks, bytes)` |
| **Cache** | `eviction.rs` | 51 | `info!("Evicted {} bytes from cache", bytes)` |
| **Cache** | `prefetch.rs` | 73 | `info!("Prefetch engine started (lookahead: {}, max_concurrent: {})")` (ADD) |
### DEBUG Level (60+ locations) - Operation Details
| Category | File | Line | Log Message |
|----------|------|------|-------------|
| **FUSE lookup** | `filesystem.rs` | 162,195,200 | Entry + result/miss |
| **FUSE getattr** | `filesystem.rs` | 203,230,233 | Entry + result/miss |
| **FUSE readdir** | `filesystem.rs` | 237,263,303 | Entry + result/miss |
| **FUSE read** | `filesystem.rs` | 325,338,362,364 | Entry + file_id + result/error |
| **Local origin** | `local.rs` | 51,68 | readdir entry + result |
| **Local origin** | `local.rs` | 88-91,112 | read entry + result |
| **SMB origin** | `smb.rs` | 86,93 | readdir/read entry + result |
| **NFS origin** | `nfs.rs` | 81,89 | readdir/read entry + result |
| **Failover** | `failover.rs` | 66,82,87 | Entry + trying origin + success |
| **Tree lookup** | `tree.rs` | 124,132 | Entry + result |
| **Metadata cache** | `metadata.rs` | 36,40 | lookup + is_fresh entry/result |
| **CAS store** | `store.rs` | 70,101 | put/get entry |
| **File reader** | `reader.rs` | 66,86 | manifest cache + read entry |
| **Search** | `ops/search.rs` | 107,141,182 | readdir_query + readlink + execute_query |
| **Search index** | `index.rs` | 98,174 | index_file + search entry |
| **Fetcher** | `fetcher.rs` | 54,61,121 | fetch_file entry + meta + ensure_cached |
**Key DEBUG fields**: `ino`, `parent`, `name`, `offset`, `size`, `bytes_read`, `origin_id`, `path`, `file_id`, `query`, `results_count`, `latency_ms`
### TRACE Level (100+ locations) - Fine-Grained Flow
| Category | File | Lines | What to Log |
|----------|------|-------|-------------|
| **Manifest cache** | `reader.rs` | 67-74 | Cache hit/miss decision |
| **Chunk iteration** | `reader.rs` | 107-127 | Each chunk: skip/read boundaries |
| **CAS dedup** | `store.rs` | 74-77 | Dedup hit decision |
| **CAS integrity** | `store.rs` | 121-134 | Verification result |
| **Tree lookup** | `tree.rs` | 118-129 | Path→inode + child lookup |
| **Tree parent** | `tree.rs` | 148-153 | Parent resolution path |
| **Prefetch event** | `prefetch.rs` | 91-120 | Event type match arms |
| **Prefetch semaphore** | `prefetch.rs` | 150-164 | In-flight check + acquire |
| **Delta scan** | `delta.rs` | 79-102 | Each file: cached/modified/unchanged/removed |
| **Delta entries** | `delta.rs` | 128-146 | Each entry: dir/audio/skip |
| **CDC chunking** | `cdc.rs` | 84-93 | Each chunk: offset/length/hash |
| **Failover origin** | `failover.rs` | 68-93 | Each origin attempt result |
| **Failover retry** | `failover.rs` | 107-122 | Each retry: attempt/success/delay |
| **Router select** | `router.rs` | 79-108 | Each candidate + selection reason |
| **FUSE node→attr** | `filesystem.rs` | 109-145 | Directory vs file conversion |
| **FUSE lookup** | `filesystem.rs` | 192-200 | Found/not found |
| **FUSE readdir** | `filesystem.rs` | 274-291 | Each child entry |
| **FUSE read** | `filesystem.rs` | 340-367 | file_id resolution + result |
| **Metadata tag** | `parser.rs` | 86-100 | Each tag extraction |
| **Health transition** | `health.rs` | 199-237 | State transition details |
| **Latency recording** | `router.rs` | 23-42 | Stats update per sample |
**Key TRACE patterns**:
- Every `match` arm: `trace!("match arm: {variant}")`
- Every `if/else`: `trace!("branch: {condition}={value}")`
- Every loop iteration: `trace!("iteration {i}/{total}: ...")`
- Every cache lookup: `trace!("cache lookup key={key}, hit={hit}")`
---
## gRPC Handler Instrumentation (ADDED - Oracle Review)
**Gap identified**: 8/10 gRPC handlers had no logging.
### server.rs - All Handlers
| Handler | Line | Level | Log Message |
|---------|------|-------|-------------|
| `get_status()` | 209 | DEBUG | `debug!("gRPC get_status called")` |
| `get_cache_stats()` | 241 | DEBUG | `debug!("gRPC get_cache_stats called")` |
| `clear_cache()` | 278 | INFO | `info!("gRPC clear_cache: clearing {tier}")` |
| `prefetch()` | 296 | DEBUG | `debug!(file_count = paths.len(), "gRPC prefetch started")` |
| `list_origins()` | 322 | DEBUG | `debug!("gRPC list_origins called")` |
| `get_origin_health()` | 329 | DEBUG | `debug!(origin_id = %id, "gRPC get_origin_health")` |
| `rescan_origin()` | 337 | INFO | `info!(origin_id = %id, "gRPC rescan_origin started")` |
| `subscribe_events()` | 376 | INFO | `info!("gRPC subscribe_events: client connected")` |
| `shutdown()` | 402 | INFO | `info!(graceful = graceful, "gRPC shutdown requested")` |
### search_service.rs
| Handler | Line | Level | Log Message |
|---------|------|-------|-------------|
| `search()` | entry | DEBUG | `debug!(query = %q, limit = limit, "gRPC search")` |
| `search()` | result | DEBUG | `debug!(results = results.len(), "gRPC search completed")` |
### Pattern: Use `#[instrument]` on all handlers
```rust
#[tracing::instrument(level = "debug", skip(self, request), fields(method = "get_status"))]
async fn get_status(&self, request: Request<()>) -> Result<Response<StatusResponse>, Status> {
// ...
}
```
---
## Async Task Spawn Instrumentation (ADDED - Oracle Review)
**Gap identified**: 14 `tokio::spawn` sites need correlation IDs and span propagation.
### Spawn Sites Requiring Instrumentation
| File | Line | Task | Instrumentation |
|------|------|------|-----------------|
| `server.rs` | 305 | prefetch stream | `spawn(async { ... }.instrument(info_span!("prefetch_stream")))` |
| `server.rs` | 354 | rescan stream | `spawn(async { ... }.instrument(info_span!("rescan_stream", origin_id = %id)))` |
| `server.rs` | 384 | subscribe events | `spawn(async { ... }.instrument(info_span!("event_subscriber")))` |
| `search_service.rs` | spawn | search task | `spawn(async { ... }.instrument(debug_span!("search_task", query = %q)))` |
| `indexer.rs` | spawn | indexer loop | `spawn(async { ... }.instrument(info_span!("indexer")))` |
| `prefetch.rs` | 87 | prefetch engine | `spawn(async { ... }.instrument(info_span!("prefetch_engine")))` |
| `prefetch.rs` | 169 | prefetch file | `spawn(async { ... }.instrument(debug_span!("prefetch_file", file_id = ?id)))` |
| `health.rs` | 154 | health monitor | `spawn(async { ... }.instrument(info_span!("health_monitor")))` |
| `watcher.rs` | 34 | file watcher | `spawn(async { ... }.instrument(info_span!("file_watcher", origin_id = %id)))` |
| `artwork.rs` | spawn | image decode | `spawn_blocking(|| { ... })` - add span before spawn |
### Pattern: Span Propagation
```rust
use tracing::Instrument;
// BEFORE (loses context)
tokio::spawn(async move {
do_work().await;
});
// AFTER (preserves correlation)
let span = tracing::info_span!("task_name", task_id = %id);
tokio::spawn(async move {
do_work().await;
}.instrument(span));
```
### Add to init_logging() for request IDs
```rust
// Generate request ID for correlation
use tracing::Span;
use uuid::Uuid;
fn with_request_id<F, R>(f: F) -> R
where F: FnOnce() -> R {
let request_id = Uuid::new_v4();
let span = tracing::info_span!("request", request_id = %request_id);
span.in_scope(f)
}
```
---
## Database Operation Logging (ADDED - Oracle Review)
**Gap identified**: Zero logging for rusqlite operations in db.rs, collections.rs, patterns.rs, artwork.rs.
### db.rs - Core Database
| Function | Line | Level | Log Message |
|----------|------|-------|-------------|
| `open()` | entry | INFO | `info!(path = ?path, "Opening metadata database")` |
| `open()` | success | INFO | `info!(file_count = count, "Database opened")` |
| `upsert_file()` | entry | DEBUG | `debug!(file_id = ?id, path = %path, "Upserting file")` |
| `upsert_file()` | error | ERROR | `error!(file_id = ?id, error = %e, "Failed to upsert file")` |
| `get_file_by_id()` | miss | TRACE | `trace!(file_id = ?id, "File not found in db")` |
| `delete_file()` | entry | DEBUG | `debug!(file_id = ?id, "Deleting file from db")` |
| `list_files_by_origin()` | result | DEBUG | `debug!(origin_id = %id, count = files.len(), "Listed files")` |
### collections.rs
| Function | Line | Level | Log Message |
|----------|------|-------|-------------|
| `create()` | entry | INFO | `info!(name = %name, "Creating collection")` |
| `save()` | error | WARN | `warn!(name = %name, error = %e, "Failed to save collection")` |
| `delete()` | entry | INFO | `info!(name = %name, "Deleting collection")` |
| `list()` | result | DEBUG | `debug!(count = collections.len(), "Listed collections")` |
### patterns.rs - Access Patterns
| Function | Line | Level | Log Message |
|----------|------|-------|-------------|
| `record_access()` | entry | TRACE | `trace!(file_id = ?id, "Recording access pattern")` |
| `predict_next()` | result | DEBUG | `debug!(predictions = preds.len(), "Predicted next files")` |
### artwork.rs
| Function | Line | Level | Log Message |
|----------|------|-------|-------------|
| `store()` | entry | DEBUG | `debug!(file_id = ?id, size_bytes = data.len(), "Storing artwork")` |
| `get()` | hit/miss | TRACE | `trace!(file_id = ?id, found = found, "Artwork lookup")` |
### Pattern: Database Error Wrapper
```rust
// Add to musicfs-cache/src/db.rs
fn log_db_result<T>(op: &str, result: Result<T, rusqlite::Error>) -> Result<T, Error> {
match result {
Ok(v) => {
tracing::trace!(op = op, "db operation succeeded");
Ok(v)
}
Err(e) => {
tracing::error!(op = op, error = %e, "db operation failed");
Err(Error::Database(e.to_string()))
}
}
}
```
---
## Channel Operation Logging (ADDED - Oracle Review)
**Gap identified**: No logging for channel capacity, close, or broadcast lag.
### Channel Locations
| File | Type | Log Points |
|------|------|------------|
| `events.rs` | broadcast | Lag warning when receiver falls behind |
| `watcher.rs` | mpsc | Channel close on watcher shutdown |
| `server.rs` | mpsc | gRPC stream channel capacity |
| `indexer.rs` | mpsc | Event queue depth |
| `health.rs` | mpsc | Health check channel |
### Patterns
```rust
// Broadcast lag detection (events.rs)
match rx.recv().await {
Ok(event) => { /* handle */ }
Err(broadcast::error::RecvError::Lagged(n)) => {
tracing::warn!(skipped = n, "Event subscriber lagged, skipped events");
}
Err(broadcast::error::RecvError::Closed) => {
tracing::debug!("Event channel closed");
break;
}
}
// Channel capacity warning (before send)
if tx.capacity() < 10 {
tracing::warn!(remaining = tx.capacity(), "Channel near capacity");
}
// Channel close
impl Drop for EventBus {
fn drop(&mut self) {
tracing::debug!("Event bus shutting down");
}
}
```
---
## Drop Implementation Logging (ADDED - Oracle Review)
**Gap identified**: No logging in Drop impls for cleanup verification.
| File | Type | Log Message |
|------|------|-------------|
| `manager.rs:276` | `PluginManager` | `debug!("PluginManager dropping, unloading {} plugins", self.plugins.len())` |
| `watcher.rs:157` | `WatchHandle` | `trace!(origin_id = %self.origin_id, "WatchHandle dropped")` |
| `prefetch.rs` | `PrefetchEngine` | `debug!("PrefetchEngine dropping, {} in-flight", self.in_flight.len())` |
| `server.rs` | gRPC server | `info!("gRPC server shutting down")` |
### Pattern
```rust
impl Drop for PluginManager {
fn drop(&mut self) {
tracing::debug!(
plugin_count = self.plugins.len(),
"PluginManager dropping"
);
// existing cleanup...
}
}
```
---
## Credential Loading (ADDED - Oracle Review)
**Gap identified**: No logging in credentials.rs::load().
| Function | Level | Log Message |
|----------|-------|-------------|
| `load()` entry | DEBUG | `debug!(origin_id = %origin_id, "Loading credentials")` |
| `load()` cache hit | TRACE | `trace!(origin_id = %origin_id, "Credential cache hit")` |
| `load()` success | INFO | `info!(origin_id = %origin_id, cred_type = %cred.type_name(), "Credential loaded")` |
| `load()` not found | DEBUG | `debug!(origin_id = %origin_id, "No credential found")` |
| `load()` error | WARN | `warn!(origin_id = %origin_id, error = %e, "Credential load failed")` |
**SECURITY**: Never log credential values. The existing Debug impl with redaction is correct.
---
## Security Considerations (ADDED - Oracle Review)
### Never Log These
| Data | Location | Mitigation |
|------|----------|------------|
| `WebhookConfig.secret` | webhook.rs | Add `#[serde(skip_serializing)]`, use custom Debug |
| Credential values | credentials.rs | Already redacted in Debug impl ✓ |
| Full file paths with usernames | everywhere | Sanitize `/home/{user}/` → `~/` |
| API keys/tokens | config.rs | Mark sensitive fields |
### Sanitization Helper
```rust
// Add to musicfs-core/src/lib.rs
pub fn sanitize_path(path: &Path) -> String {
if let Ok(home) = std::env::var("HOME") {
path.to_string_lossy()
.replace(&home, "~")
.to_string()
} else {
path.to_string_lossy().to_string()
}
}
// Usage
debug!(path = %sanitize_path(&path), "Reading file");
```
### WebhookConfig Fix
```rust
// webhook.rs - add custom Debug
#[derive(Clone, Serialize, Deserialize)]
pub struct WebhookConfig {
pub url: String,
#[serde(skip_serializing)]
pub secret: Option<String>, // Never serialize
// ...
}
impl std::fmt::Debug for WebhookConfig {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("WebhookConfig")
.field("url", &self.url)
.field("secret", &self.secret.as_ref().map(|_| "[REDACTED]"))
.finish()
}
}
```
---
## Performance Considerations (ADDED - Oracle Review)
### Hot Path Warnings
| Path | Risk | Mitigation |
|------|------|------------|
| `reader.rs` chunk loop | 100s of TRACE logs per seek | Log summary only: `trace!(chunks_read = n, "Read complete")` |
| `store.rs` put/get | 1000s during sync | Keep at DEBUG, not TRACE |
| `delta.rs` file scan | Log per file during full scan | Use TRACE, batch summaries at DEBUG |
| `parser.rs` tag extraction | Many TRACE per file | Sample: log every 100th file |
### Trace Sampling Config
```rust
// Add to LoggingConfig
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct LoggingConfig {
// ... existing fields ...
/// Sample rate for TRACE logs in hot paths (0.0-1.0, default 1.0)
#[serde(default = "default_sample_rate")]
pub trace_sample_rate: f32,
}
fn default_sample_rate() -> f32 { 1.0 }
// Usage in hot paths
if rand::random::<f32>() < config.trace_sample_rate {
trace!(...);
}
```
### Rate-Limited Warnings
```rust
// For repeating warnings during outages (failover.rs)
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
static LAST_FAILOVER_WARN: AtomicU64 = AtomicU64::new(0);
fn warn_rate_limited(origin_id: &str, error: &str) {
let now = Instant::now().elapsed().as_secs();
let last = LAST_FAILOVER_WARN.load(Ordering::Relaxed);
if now - last >= 60 { // Max once per minute
LAST_FAILOVER_WARN.store(now, Ordering::Relaxed);
warn!(origin_id = %origin_id, error = %error, "Origin failover");
}
}
```
---
## Standardized Field Names (ADDED - Oracle Review)
Use these consistently across all log statements:
| Field | Type | Usage |
|-------|------|-------|
| `origin_id` | String | Origin identifier (not `origin`) |
| `file_id` | FileId | File identifier |
| `path` | String | Virtual or real path (sanitized) |
| `size_bytes` | u64 | Size in bytes (not `size`, `bytes`, `len`) |
| `offset` | u64 | Read offset |
| `duration_ms` | u64 | Operation duration in milliseconds |
| `count` | usize | Generic count |
| `attempt` | u32 | Retry attempt number |
| `max_attempts` | u32 | Maximum retry attempts |
| `error` | impl Display | Error message (not `err`, `e`) |
| `request_id` | Uuid | Correlation ID for requests |
---
## Instrumentation Patterns (ADDED - Oracle Review)
### Use `#[instrument(err)]` for Automatic Error Logging
```rust
// BEFORE: Manual error logging
pub async fn read(&self, path: &Path) -> Result<Bytes> {
match self.inner_read(path).await {
Ok(data) => Ok(data),
Err(e) => {
error!(path = ?path, error = %e, "Read failed");
Err(e)
}
}
}
// AFTER: Automatic with #[instrument]
#[tracing::instrument(level = "debug", skip(self), err)]
pub async fn read(&self, path: &Path) -> Result<Bytes> {
self.inner_read(path).await
}
```
### Span Events vs Regular Logs
```rust
// Regular log - standalone event
info!("Operation completed");
// Span event - attached to current span context
tracing::Span::current().record("result", "success");
// Prefer span events for operation outcomes
#[instrument(fields(result))]
async fn operation() -> Result<()> {
// ... work ...
Span::current().record("result", "success");
Ok(())
}
```
---
## Fixes: Incorrect Line References (ADDED - Oracle Review)
| File | Issue | Fix |
|------|-------|-----|
| `webhook.rs:43` | Uses `expect()` (panics) | Replace with `?` + error log |
| `webhook.rs:133` | Uses `expect()` (panics) | Replace with `?` + error log |
```rust
// webhook.rs - BEFORE
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.build()
.expect("Failed to create HTTP client");
// webhook.rs - AFTER
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.build()
.map_err(|e| {
error!(error = %e, "Failed to create webhook HTTP client");
WebhookError::ClientInit(e.to_string())
})?;
```
---
## Log Levels Guide
| Level | Use Case | Example |
|-------|----------|---------|
| `ERROR` | Unrecoverable failures | Mount failed, DB corruption |
| `WARN` | Recoverable issues | Origin timeout, retry needed |
| `INFO` | Lifecycle events | Service start/stop, health change |
| `DEBUG` | Operation details | Function entry, request params |
| `TRACE` | Fine-grained flow | Match arms, cache hit/miss |
---
## Testing Checklist
### Basic Functionality
- [ ] Log files created in configured directory
- [ ] Daily rotation creates new files at midnight
- [ ] JSON output parseable by `jq`
- [ ] `journalctl -t musicfs` shows logs
- [ ] `RUST_LOG=musicfs=trace` enables trace output
- [ ] WorkerGuard kept alive (logs flush on shutdown)
- [ ] Logrotate compresses old files
### Correlation & Context (NEW)
- [ ] Request IDs propagate through async tasks
- [ ] Spawned task logs include parent span context
- [ ] gRPC handler logs show method name in span
### Security (NEW)
- [ ] WebhookConfig.secret never appears in logs
- [ ] Credential values never appear in logs
- [ ] File paths with `/home/{user}` show as `~/`
### Performance (NEW)
- [ ] TRACE sampling respects `trace_sample_rate` config
- [ ] Hot path chunk loops log summary, not per-chunk
- [ ] Origin failover warnings are rate-limited (1/minute)
- [ ] Database operations log without blocking
### Database & Channels (NEW)
- [ ] Database open logs file count
- [ ] Channel capacity warnings appear when queue fills
- [ ] Broadcast lag warnings appear when subscriber falls behind
- [ ] Drop implementations log cleanup
---
## Summary
| Phase | Effort | Deliverables |
|-------|--------|--------------|
| 1. Config & Dependencies | 2h | LoggingConfig, init_logging(), logrotate, trace sampling |
| 2. Core instrumentation | 1h | tracing in musicfs-core, credentials, sanitization |
| 3. Hot path instrumentation | 4h | #[instrument] + trace! across 5 crates |
| 4. gRPC & async tasks | 2h | Handler instrumentation, spawn correlation |
| 5. Database & channels | 2h | rusqlite logging, channel capacity/close |
| 6. Production files | 1h | Updated systemd, example config |
| **Total** | **12h** | Full observability |
---
## Files to Modify
### Phase 1: Config & Dependencies
| File | Changes |
|------|---------|
| `Cargo.toml` (workspace) | Add tracing-appender, tracing-journald |
| `crates/musicfs-cli/Cargo.toml` | Add dependencies |
| `crates/musicfs-core/Cargo.toml` | Add tracing |
| `crates/musicfs-core/src/config.rs` | Add LoggingConfig with trace_sample_rate |
| `crates/musicfs-cli/src/main.rs` | Expand init_logging(), request ID helper |
| `crates/musicfs-core/src/lib.rs` | Add sanitize_path() helper |
### Phase 2: Core Instrumentation
| File | Changes |
|------|---------|
| `crates/musicfs-core/src/credentials.rs` | Add load() logging (redacted) |
| `crates/musicfs-core/src/events.rs` | Add broadcast lag detection |
### Phase 3: Hot Path Instrumentation
| File | Changes |
|------|---------|
| `crates/musicfs-fuse/src/filesystem.rs` | Add #[instrument], trace! |
| `crates/musicfs-origins/src/failover.rs` | Add #[instrument], trace!, rate-limited warn |
| `crates/musicfs-origins/src/health.rs` | Add state transition logging |
| `crates/musicfs-origins/src/router.rs` | Add selection logging |
| `crates/musicfs-cache/src/tree.rs` | Add mutation logging |
| `crates/musicfs-cache/src/metadata.rs` | Add hit/miss logging |
| `crates/musicfs-cas/src/reader.rs` | Add chunk assembly logging (summary, not per-chunk) |
| `crates/musicfs-cas/src/store.rs` | Add dedup logging |
| `crates/musicfs-sync/src/delta.rs` | Add change detection logging |
### Phase 4: gRPC & Async Tasks (NEW)
| File | Changes |
|------|---------|
| `crates/musicfs-grpc/src/server.rs` | Add #[instrument] to all 10 handlers, spawn correlation |
| `crates/musicfs-grpc/src/search_service.rs` | Add #[instrument], spawn instrumentation |
| `crates/musicfs-grpc/src/webhook.rs` | Fix expect() → error!, custom Debug for secret |
| `crates/musicfs-cache/src/prefetch.rs` | Add spawn instrumentation, Drop logging |
| `crates/musicfs-search/src/indexer.rs` | Add spawn instrumentation |
| `crates/musicfs-sync/src/watcher.rs` | Add spawn instrumentation, Drop logging |
| `crates/musicfs-plugins/src/manager.rs` | Add Drop logging |
### Phase 5: Database & Channels (NEW)
| File | Changes |
|------|---------|
| `crates/musicfs-cache/src/db.rs` | Add log_db_result() helper, open/upsert/query logging |
| `crates/musicfs-search/src/collections.rs` | Add CRUD operation logging |
| `crates/musicfs-cache/src/patterns.rs` | Add access pattern logging |
| `crates/musicfs-cache/src/artwork.rs` | Add store/get logging |
### Phase 6: Production Files
| File | Changes |
|------|---------|
| `dist/musicfs.service` | Add logging directives |
| `dist/logrotate.d/musicfs` | New file |
| `dist/config.example.toml` | Add logging section with trace_sample_rate |
docs/v2/plans/persistent-state-impl.md
-796
View File
@@ -1,796 +0,0 @@
# Persistent State: Implementation Plan
**Authors:** AI-assisted
**Status:** Draft
**Last Updated:** 2026-05-13
**Reviewers:** TBD
**Approvers:** TBD
**Prerequisites:** [persistent-state.md](persistent-state.md) (research), [phase-a-stop-dying.md](phase-a-stop-dying.md) (signal handling + shutdown)
**Estimated Effort:** ~8 days
---
[TOC]
---
## 1. Abstract
Wire up the existing SQLite persistence layer into the mount path so that subsequent mounts load from database instead of rescanning origins. This transforms mount time from O(N × origin_latency) to O(N × SQLite_read) — roughly 1000x faster for remote origins.
**Storage decision: SQLite (Option A).** Rationale:
- `Database` struct with full CRUD already exists in `musicfs-cache/src/db.rs`
- Schema with `chunk_manifest BLOB` column already exists in `schema.sql`
- `ChunkManifest::from_db()` and `chunks_to_bytes()` already exist but are never called
- Row-to-`FileMeta` mapping already exists in `get_file_by_virtual_path()`
- WAL mode crash safety already configured
- 2-4 second bulk load for 1M rows is acceptable (target is <5s, not <500ms — the <500ms target is for the mount syscall itself, which returns immediately with lazy tree loading)
No new storage engine. No new dependencies. Wire existing code.
---
## 2. Background
### 2.1 Current State
`run_mount()` in `main.rs`:
1. Opens CAS store ✅
2. Creates origin connection ✅
3. `scan_music_files()` — walks entire origin, parses every file with symphonia ❌ **BOTTLENECK**
4. Builds VirtualTree from scan results (in-memory only) ❌ **LOST ON RESTART**
5. Registers every file in ContentFetcher (in-memory only) ❌ **LOST ON RESTART**
6. Mounts FUSE ✅
### 2.2 What Exists But Is Not Wired
| Component | Exists | Wired Into Mount? |
|-----------|--------|--------------------|
| `Database::open()` + schema + WAL | ✅ | ❌ |
| `Database::upsert_file()` | ✅ | ❌ |
| `Database::get_file_by_virtual_path()` (returns `FileMeta`) | ✅ | ❌ |
| `schema.sql` with `chunk_manifest BLOB` column | ✅ | ❌ |
| `ChunkManifest::chunks_to_bytes()` (serialize) | ✅ | ❌ |
| `ChunkManifest::from_db()` (deserialize) | ✅ | ❌ |
| `TreeBuilder::add_file(&FileMeta)` | ✅ | ✅ (from scan, not from DB) |
| `ContentFetcher::register_file(FileMeta)` | ✅ | ✅ (from scan, not from DB) |
| `PatternStore::new(db_path)` (loads from SQLite on open) | ✅ | ❌ |
| `CollectionStore::new(db_path)` | ✅ | ❌ |
| `SearchIndex::open(path)` (opens tantivy from disk) | ✅ | ❌ |
### 2.3 What's Missing
| Component | Needs Building |
|-----------|----------------|
| `Database::list_all_files()` → `Vec<FileMeta>` | New method (SQL exists, just needs `SELECT *`) |
| `Database::update_manifest(FileId, &[u8])` | New method (column exists) |
| `Database::get_manifest(FileId)` → `Option<Vec<u8>>` | New method |
| `Database::list_all_manifests()` → `Vec<(FileId, ChunkManifest)>` | New method |
| Background delta sync task | New (compare DB state vs origin) |
| First-mount detection | New (check `file_count() > 0`) |
---
## 3. Goals & Non-Goals
### 3.1 Goals
- Subsequent mount loads tree from SQLite, not origin scan
- Chunk manifests persist to SQLite, loaded on mount (no re-download)
- tantivy index, PatternStore, CollectionStore opened on mount
- Background delta sync reconciles DB vs origin after mount
- First mount (empty DB) falls back to current full-scan behavior
- Mount time for 10K files: <1 second (subsequent mount)
- All existing tests pass, no regressions
### 3.2 Non-Goals
- Achieving <500ms mount for 1M+ files (requires lazy tree loading — future work)
- LRU eviction persistence (separate task, low urgency)
- Changing the storage engine (SQLite is the decision)
- Config file parsing changes (origin config stays in TOML, not DB)
- Schema migrations for existing data (fresh DB on first mount)
---
## 4. Proposed Design
### 4.1 Implementation Order
```
4.2 Database: list_all_files() + manifest CRUD (foundation)
↓
4.3 Mount path: load tree + fetcher from DB (core change)
↓
4.4 Persist manifests after fetch (write path)
↓
4.5 Open tantivy + PatternStore + CollectionStore (quick wiring)
↓
4.6 Background delta sync (post-mount reconciliation)
↓
4.7 First-mount detection + fallback (edge case)
↓
4.8 Shutdown: WAL checkpoint + flush (cleanup)
```
### 4.2 Database: New Methods
**File**: `musicfs-cache/src/db.rs`
#### list_all_files()
Bulk load all files from DB. Reuses the existing row-to-FileMeta mapping from `get_file_by_virtual_path()`.
```rust
pub fn list_all_files(&self) -> Result<Vec<FileMeta>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
r#"SELECT id, origin_id, real_path, virtual_path,
title, artist, album, album_artist, genre,
year, track, disc,
duration_ms, bitrate, sample_rate, format,
origin_mtime, origin_size, content_hash
FROM files
ORDER BY virtual_path"#
).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?;
let files = stmt.query_map([], |row| {
// Same mapping as get_file_by_virtual_path
Ok(Self::row_to_file_meta(row))
})
.map_err(|e| Error::Database(format!("query failed: {}", e)))?
.filter_map(|r| r.ok())
.collect();
Ok(files)
}
```
Extract the row mapping into a shared `row_to_file_meta(row)` helper to avoid duplication with `get_file_by_virtual_path()`.
#### Manifest CRUD
```rust
pub fn update_manifest(&self, file_id: FileId, manifest_blob: &[u8]) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute(
"UPDATE files SET chunk_manifest = ?1 WHERE id = ?2",
params![manifest_blob, file_id.0],
).map_err(|e| Error::Database(format!("update manifest failed: {}", e)))?;
Ok(())
}
pub fn get_manifest(&self, file_id: FileId) -> Result<Option<Vec<u8>>> {
let conn = self.conn.lock().unwrap();
conn.query_row(
"SELECT chunk_manifest FROM files WHERE id = ?1",
params![file_id.0],
|row| row.get(0),
)
.optional()
.map_err(|e| Error::Database(format!("get manifest failed: {}", e)))
}
pub fn list_all_manifests(&self) -> Result<Vec<(FileId, u64, i64, Vec<u8>)>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT id, origin_size, origin_mtime, chunk_manifest FROM files WHERE chunk_manifest IS NOT NULL"
).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?;
let manifests = stmt.query_map([], |row| {
Ok((
FileId(row.get(0)?),
row.get::<_, i64>(1)? as u64,
row.get::<_, i64>(2)?,
row.get::<_, Vec<u8>>(3)?,
))
})
.map_err(|e| Error::Database(format!("query failed: {}", e)))?
.filter_map(|r| r.ok())
.collect();
Ok(manifests)
}
```
#### WAL Checkpoint
```rust
pub fn checkpoint(&self) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE)")
.map_err(|e| Error::Database(format!("WAL checkpoint failed: {}", e)))?;
info!("SQLite WAL checkpoint completed");
Ok(())
}
```
#### Tests
```rust
#[test]
fn test_list_all_files() {
let db = Database::open_memory().unwrap();
// Insert 3 files
// list_all_files() returns 3
// Verify FileMeta fields match what was inserted
}
#[test]
fn test_manifest_roundtrip() {
let db = Database::open_memory().unwrap();
// Insert file, update_manifest with blob, get_manifest returns same blob
}
#[test]
fn test_list_all_manifests_skips_null() {
let db = Database::open_memory().unwrap();
// Insert 3 files, only 1 with manifest
// list_all_manifests() returns 1
}
```
---
### 4.3 Mount Path: Load From DB
**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()`
The key change: replace `scan_music_files()` with DB load when data exists.
```rust
fn run_mount(mountpoint: PathBuf, origin_path: Option<PathBuf>, cache_dir: Option<PathBuf>) -> Result<()> {
let origin_path = origin_path.context("--origin is required")?;
let runtime = tokio::runtime::Runtime::new()?;
let handle = runtime.handle().clone();
let (tree, reader, db) = runtime.block_on(async {
let cache_dir = resolve_cache_dir(cache_dir);
std::fs::create_dir_all(&cache_dir)?;
std::fs::create_dir_all(&mountpoint)?;
// Open CAS store
let store = Arc::new(CasStore::open(CasConfig {
chunks_dir: cache_dir.join("chunks"),
..Default::default()
}).await?);
// Open database
let db_path = cache_dir.join("metadata.db");
let db = Arc::new(Database::open_with_integrity_check(&db_path)
.or_else(|_| Database::open(&db_path))?); // Fallback to normal open if integrity check fails
let fetcher = Arc::new(ContentFetcher::new(store.clone()));
let origin_id = OriginId::from("local");
let origin = Arc::new(LocalOrigin::new(origin_id.clone(), origin_path.clone()));
fetcher.register_origin(origin);
// Decide: load from DB or full scan
let file_count = db.file_count().unwrap_or(0);
let files = if file_count > 0 {
// SUBSEQUENT MOUNT — load from DB
info!(file_count, "Loading metadata from database");
let start = Instant::now();
let files = db.list_all_files()?;
info!(elapsed_ms = start.elapsed().as_millis() as u64, "Database load complete");
files
} else {
// FIRST MOUNT — full origin scan
info!("First mount: scanning origin");
let files = scan_music_files(&origin_path, &origin_id).await?;
info!(file_count = files.len(), "Scan complete, persisting to database");
// Persist to DB for next mount
for file in &files {
if let Some(ref audio) = file.audio {
db.upsert_file(
&file.real_path.origin_id,
&file.real_path.path,
&file.virtual_path,
audio,
file.mtime,
file.size,
)?;
}
}
info!("Metadata persisted to database");
files
};
// Build tree + register files (same as before, but from DB or scan)
let mut builder = TreeBuilder::new();
for file in &files {
builder.add_file(file);
fetcher.register_file(file.clone());
}
let tree = Arc::new(RwLock::new(builder.build()));
// Load manifests from DB
let reader = Arc::new(FileReader::with_fetcher(store, fetcher));
let manifest_count = load_manifests_from_db(&db, &reader)?;
if manifest_count > 0 {
info!(manifest_count, "Loaded chunk manifests from database");
}
Ok::<_, anyhow::Error>((tree, reader, db))
})?;
// Open search index
let search_dir = cache_dir.join("search.idx");
let _search_index = SearchIndex::open_with_recovery(&search_dir)
.context("Failed to open search index")?;
// Open pattern store
let patterns_path = cache_dir.join("patterns.db");
let _pattern_store = PatternStore::new(&patterns_path, 30)
.context("Failed to open pattern store")?;
// ... mount, signal handler, shutdown (same as current) ...
// On shutdown: checkpoint WAL
db.checkpoint().unwrap_or_else(|e| warn!("WAL checkpoint failed: {}", e));
}
```
Helper function:
```rust
fn load_manifests_from_db(db: &Database, reader: &FileReader) -> Result<usize> {
let manifests = db.list_all_manifests()?;
let mut count = 0;
for (file_id, total_size, mtime, blob) in manifests {
if let Some(manifest) = ChunkManifest::from_db(file_id, total_size, mtime, &blob) {
reader.register_manifest(manifest);
count += 1;
}
}
Ok(count)
}
```
---
### 4.4 Persist Manifests After Fetch
**File**: `musicfs-cas/src/fetcher.rs`
After `fetch_file()` downloads and chunks a file, persist the manifest to SQLite.
The fetcher currently doesn't have access to the Database. Two options:
1. Pass `Arc<Database>` to ContentFetcher (adds dependency musicfs-cas → musicfs-cache)
2. Emit an event with the manifest, have the caller persist it
**Approach**: Option 2 — use the existing EventBus. Add a new event variant:
**File**: `musicfs-core/src/events.rs`
```rust
pub enum Event {
// ... existing variants
ManifestCached {
file_id: FileId,
manifest_blob: Vec<u8>,
},
}
```
**File**: `musicfs-cas/src/fetcher.rs` — emit event after fetch:
```rust
pub async fn fetch_file(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
// ... existing fetch + chunk logic ...
// Emit manifest for persistence
if let Some(bus) = &self.event_bus {
bus.publish(Event::ManifestCached {
file_id,
manifest_blob: manifest.chunks_to_bytes(),
});
}
Ok(manifest)
}
```
**File**: `musicfs-cli/src/main.rs` — subscribe to ManifestCached events:
```rust
// Spawn manifest persistence listener
let db_for_manifests = db.clone();
let mut manifest_rx = event_bus.subscribe();
tokio::spawn(async move {
while let Ok(event) = manifest_rx.recv().await {
if let Event::ManifestCached { file_id, manifest_blob } = event {
if let Err(e) = db_for_manifests.update_manifest(file_id, &manifest_blob) {
warn!(file_id = ?file_id, error = %e, "Failed to persist manifest");
}
}
}
});
```
---
### 4.5 Open tantivy + PatternStore + CollectionStore
These already have `open()` methods that load from disk. Just call them in the mount path.
**File**: `musicfs-cli/src/main.rs`
```rust
// After tree is built, before FUSE mount
// Search index
let search_dir = cache_dir.join("search.idx");
let search_index = Arc::new(
SearchIndex::open_with_recovery(&search_dir)
.unwrap_or_else(|e| {
warn!("Search index failed, creating fresh: {}", e);
SearchIndex::open(&search_dir).expect("Failed to create search index")
})
);
// Pattern store (already persists to SQLite, loads sequence_counts on open)
let patterns_path = cache_dir.join("patterns.db");
let pattern_store = Arc::new(
PatternStore::new(&patterns_path, 30)
.unwrap_or_else(|e| {
warn!("Pattern store failed: {}", e);
PatternStore::new(&patterns_path, 30).expect("Failed to create pattern store")
})
);
// Collection store
let collections_path = cache_dir.join("collections.db");
let collection_store = Arc::new(
CollectionStore::new(&collections_path)
.unwrap_or_else(|e| {
warn!("Collection store failed: {}", e);
CollectionStore::new(&collections_path).expect("Failed to create collection store")
})
);
```
For tantivy: if this is a first mount, index all files after scan:
```rust
if file_count == 0 {
// First mount — index all files
info!("First mount: building search index");
let indexer = Indexer::new(search_index.clone(), event_bus.clone(), /* metadata_lookup */);
indexer.index_batch(&files)?;
}
```
---
### 4.6 Background Delta Sync
After mount completes, spawn a background task that compares DB state against origin and reconciles differences.
**File**: `musicfs-sync/src/delta.rs` or new `musicfs-cli/src/sync.rs`
```rust
pub async fn background_delta_sync(
origin: Arc<dyn Origin>,
origin_id: OriginId,
db: Arc<Database>,
tree: Arc<RwLock<VirtualTree>>,
fetcher: Arc<ContentFetcher>,
event_bus: Arc<EventBus>,
) -> Result<SyncSummary> {
info!("Starting background delta sync");
let start = Instant::now();
let mut added = 0u64;
let mut modified = 0u64;
let mut removed = 0u64;
let mut unchanged = 0u64;
// Get all files currently in DB
let db_files: HashMap<PathBuf, FileMeta> = db.list_all_files()?
.into_iter()
.map(|f| (f.real_path.path.clone(), f))
.collect();
// Walk origin
let origin_files = scan_origin_recursive(&origin, Path::new("/")).await?;
// Compare
for (path, origin_stat) in &origin_files {
match db_files.get(path) {
Some(db_file) if db_file.mtime == origin_stat.mtime && db_file.size == origin_stat.size => {
unchanged += 1;
}
Some(db_file) => {
// Modified — re-parse metadata, update DB, update tree
modified += 1;
// ... update logic ...
}
None => {
// New file — parse metadata, add to DB + tree
added += 1;
// ... add logic ...
}
}
}
// Find removed files (in DB but not on origin)
let origin_paths: HashSet<_> = origin_files.keys().collect();
for (path, db_file) in &db_files {
if !origin_paths.contains(path) {
removed += 1;
db.delete_file(db_file.id)?;
tree.write().remove_file(&db_file.virtual_path);
}
}
let elapsed = start.elapsed();
info!(
added, modified, removed, unchanged,
elapsed_ms = elapsed.as_millis() as u64,
"Delta sync complete"
);
Ok(SyncSummary { added, modified, removed, unchanged })
}
```
Spawn in `run_mount()` after FUSE mount:
```rust
// Background delta sync (non-blocking)
let sync_db = db.clone();
let sync_tree = tree.clone();
let sync_fetcher = fetcher.clone();
let sync_origin = origin.clone();
let sync_origin_id = origin_id.clone();
let sync_bus = event_bus.clone();
tokio::spawn(async move {
if let Err(e) = background_delta_sync(
sync_origin, sync_origin_id, sync_db, sync_tree, sync_fetcher, sync_bus,
).await {
warn!("Delta sync failed: {}", e);
}
});
```
---
### 4.7 First-Mount Detection
Simple: check `db.file_count()`:
```rust
let file_count = db.file_count().unwrap_or(0);
if file_count > 0 {
// Load from DB
} else {
// Full scan + persist
}
```
This is already shown in Section 4.3. No separate implementation step.
---
### 4.8 Shutdown: WAL Checkpoint + Flush
**File**: `musicfs-cli/src/main.rs` — in the shutdown sequence (after signal, before dropping session):
```rust
info!("Beginning ordered shutdown");
shutdown_token.cancel();
tokio::time::sleep(Duration::from_millis(500)).await;
// Flush persistence
if let Err(e) = db.checkpoint() {
warn!("SQLite WAL checkpoint failed: {}", e);
}
info!("Background tasks stopped, state flushed");
```
---
## 5. Cross-Cutting Concerns
### 5.1 Security & Privacy
- No new attack surface — SQLite file has same permissions as cache directory
- Metadata in DB is the same as what's already in the FUSE virtual tree (not new data)
- `chunk_manifest` BLOB is binary chunk hashes — not sensitive
### 5.2 Observability
- Mount time logged: "Loading metadata from database" with elapsed_ms
- First-mount detected and logged: "First mount: scanning origin"
- Delta sync summary logged: added/modified/removed/unchanged counts + elapsed
- WAL checkpoint logged on shutdown
- Manifest persistence failures logged at WARN (non-fatal)
### 5.3 Scalability
| Library Size | First Mount (scan) | Subsequent Mount (DB load) |
|---|---|---|
| 1K files | ~1-2s | <100ms |
| 10K files | ~10-20s | ~200ms |
| 100K files | ~2-5 min | ~1-2s |
| 1M files | ~20-60 min | ~2-4s |
Delta sync runs in background — mount returns immediately, user sees stale-but-functional data while sync catches up.
### 5.4 Testing
```rust
// Test: subsequent mount loads from DB
#[tokio::test]
async fn test_mount_loads_from_db() {
let dir = TempDir::new().unwrap();
let db = Database::open(dir.path().join("test.db")).unwrap();
// Insert files
for i in 0..100 {
db.upsert_file(/* ... */).unwrap();
}
// Load all
let files = db.list_all_files().unwrap();
assert_eq!(files.len(), 100);
// Build tree from DB files (same as mount path)
let mut builder = TreeBuilder::new();
for f in &files { builder.add_file(f); }
let tree = builder.build();
assert_eq!(tree.file_count(), 100);
}
// Test: manifest roundtrip through DB
#[tokio::test]
async fn test_manifest_persists_and_loads() {
let dir = TempDir::new().unwrap();
let db = Database::open(dir.path().join("test.db")).unwrap();
let id = db.upsert_file(/* ... */).unwrap();
let manifest = ChunkManifest { /* ... */ };
let blob = manifest.chunks_to_bytes();
db.update_manifest(id, &blob).unwrap();
let loaded = db.get_manifest(id).unwrap().unwrap();
let restored = ChunkManifest::from_db(id, 1000, 0, &loaded).unwrap();
assert_eq!(restored.chunks.len(), manifest.chunks.len());
}
// Test: first mount detects empty DB
#[tokio::test]
async fn test_first_mount_detection() {
let dir = TempDir::new().unwrap();
let db = Database::open(dir.path().join("test.db")).unwrap();
assert_eq!(db.file_count().unwrap(), 0); // First mount
}
// Test: delta sync detects changes
#[tokio::test]
async fn test_delta_sync_detects_added_file() {
// DB has files A, B
// Origin has files A, B, C
// Delta sync should detect C as added
}
// Test: delta sync detects removed file
#[tokio::test]
async fn test_delta_sync_detects_removed_file() {
// DB has files A, B, C
// Origin has files A, B
// Delta sync should detect C as removed
}
// Test: shutdown checkpoints WAL
#[tokio::test]
async fn test_shutdown_checkpoints_wal() {
let dir = TempDir::new().unwrap();
let db_path = dir.path().join("test.db");
let db = Database::open(&db_path).unwrap();
db.upsert_file(/* ... */).unwrap();
// WAL file should exist
let wal_path = db_path.with_extension("db-wal");
// After checkpoint, WAL should be truncated
db.checkpoint().unwrap();
}
```
---
## 6. Alternatives Considered
### 6.1 sled for Tree Storage (Option B)
sled is faster for bulk key-value reads (~1-2s for 1M entries vs SQLite's ~2-4s). Rejected because:
- SQLite code already exists (schema, CRUD, row mapping)
- sled would require new serialization layer (bincode/msgpack for FileMeta)
- Two persistence engines is more complex
- SQLite's 2-4s is acceptable for the target
### 6.2 Flat File Snapshot (Option C)
Fastest possible bulk load (<1s via mmap). Rejected because:
- No incremental updates — every change rewrites the entire file
- At 1M files (~500MB), delta sync triggers a 500MB write for each changed file
- No concurrent access safety
- No crash recovery for partial writes
### 6.3 Lazy Tree Loading
Instead of loading all files into memory on mount, load only the root directories and fetch deeper levels on demand from SQLite. This would achieve true O(1) mount. Deferred because:
- Requires significant refactoring of VirtualTree (currently all-in-memory)
- SQLite 2-4s load is good enough for production
- Can be added later as optimization without changing the persistence layer
### 6.4 Separate Manifest Store
Instead of storing manifests in the `files.chunk_manifest` column, use a separate sled tree or SQLite table. Rejected because the column already exists and the schema already supports it.
---
## 7. Implementation Plan
### 7.1 Task Sequence
| Day | Task | Deliverable |
|-----|------|-------------|
| 1 | Database methods: `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`. Extract `row_to_file_meta()` helper. | New DB methods + tests |
| 2 | Rewrite `run_mount()`: DB load path vs scan path. First-mount detection. | Core mount change |
| 3 | Persist manifests: `ManifestCached` event + listener in main.rs. Load manifests on mount via `load_manifests_from_db()`. | Manifest persistence |
| 4 | Wire tantivy + PatternStore + CollectionStore into mount path. First-mount indexing. | Search/patterns on mount |
| 5 | Background delta sync: compare DB vs origin, update differences. | Delta sync task |
| 6 | Shutdown: WAL checkpoint. Upsert files to DB during first-mount scan. | Clean shutdown |
| 7 | Integration testing: full mount→read→restart→mount cycle. Verify tree + manifests survive restart. | E2E validation |
| 8 | Buffer for issues found during integration. | — |
### 7.2 Verification Checklist
- [ ] `cargo check` — zero errors
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass
- [ ] Manual test: first mount (empty cache dir) — scans origin, creates DB
- [ ] Manual test: second mount (DB exists) — loads from DB, no origin scan
- [ ] Manual test: add file to origin, restart — delta sync discovers it
- [ ] Manual test: `kill -9` daemon, restart — DB loads, manifests intact
- [ ] Mount time for 10K test files: <1 second on subsequent mount
- [ ] `ls -la ~/.cache/musicfs/metadata.db` exists after first mount
---
## 8. Files Changed
| File | Change |
|------|--------|
| `musicfs-cache/src/db.rs` | `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`, `row_to_file_meta()` refactor |
| `musicfs-core/src/events.rs` | Add `ManifestCached` event variant |
| `musicfs-cli/src/main.rs` | Rewrite `run_mount()`: DB load vs scan, open tantivy/patterns/collections, manifest listener, delta sync spawn, shutdown checkpoint |
| `musicfs-cli/Cargo.toml` | Add `musicfs-search`, `musicfs-cache` dependencies (for PatternStore, CollectionStore, SearchIndex) |
| `musicfs-cas/src/fetcher.rs` | Emit `ManifestCached` event after `fetch_file()` |
| `musicfs-sync/src/delta.rs` | New `background_delta_sync()` function (or new file) |
| `musicfs-test-utils/tests/resilience.rs` | New tests: mount-from-DB, manifest roundtrip, delta sync, first-mount detection |
---
## 9. Glossary / References
| Term | Definition |
|------|------------|
| **First mount** | Initial mount with empty database — triggers full origin scan |
| **Subsequent mount** | Mount with existing database — loads from SQLite |
| **Delta sync** | Background task that compares DB state against origin after mount |
| **Stale data window** | Time between mount and delta sync completion when data may be outdated |
| **WAL checkpoint** | SQLite operation that flushes write-ahead log to main database file |
| Document | Path |
|----------|------|
| Persistent state research | [persistent-state.md](persistent-state.md) |
| Phase A (signals, shutdown) | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
| Phase B (crash recovery) | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) |
| Architecture | [architecture.md](../architecture.md) |
docs/v2/plans/persistent-state.md
-353
View File
@@ -1,353 +0,0 @@
# MusicFS Persistent State Plan
**Date**: 2026-05-13
**Status**: Research Complete — Design Decision Needed
**Prerequisites**: [architecture.md](../architecture.md), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
**Related Requirements**: G1 (O(1) mount time), NFR-1.7 (<500ms mount), FR-7.1 (cache persists across restarts)
---
## 1. Problem Statement
Every mount is a full cold start. The `run_mount()` function in `main.rs` does not use any persistent storage — it walks the entire origin filesystem, parses metadata from every audio file, and builds all runtime state from scratch.
The architecture designed persistence infrastructure (SQLite schema, `chunk_manifest` column, `ChunkManifest::from_db()`, `chunks_to_bytes()`) but **none of it is wired into the mount path**. The mount flow doesn't even open the database.
### Mount Time by Library Size (Current)
| Library Size | Estimated Mount Time | Target (NFR-1.7) |
|---|---|---|
| 1K files | ~1-2s | <500ms |
| 10K files | ~10-20s | <500ms |
| 100K files | ~2-5 minutes | <500ms |
| 1M files | ~20-60 minutes | <500ms |
| 10M files (stretch) | hours | <500ms |
---
## 2. In-Memory State Inventory
### 2.1 State That Must Survive Restart
These are the large, expensive-to-rebuild data structures. Losing them forces a full origin rescan.
#### VirtualTree (~300-400MB at 1M files)
**Location**: `musicfs-cache/src/tree.rs`
**Contents**:
- `nodes: HashMap<Inode, VirtualNode>` — every directory and file node
- `path_to_inode: HashMap<VirtualPath, Inode>` — reverse path lookup
- `next_inode: AtomicU64` — inode counter
**Currently rebuilt from**: Full recursive origin scan + metadata parse of every audio file. This is the single most expensive operation on mount — it touches every file on origin, runs symphonia metadata extraction, and builds the entire tree structure.
**What's needed**: Load from persistent storage on mount. Rebuild only on first-ever mount or if storage is corrupt.
---
#### ContentFetcher.file_meta (~200MB at 1M files)
**Location**: `musicfs-cas/src/fetcher.rs`
**Contents**:
- `file_meta: RwLock<HashMap<FileId, FileMeta>>` — full metadata for every file
- Each `FileMeta` contains: id, virtual_path, real_path (origin_id + path), size, mtime, content_hash, audio metadata
**Currently rebuilt from**: Same origin scan that builds the tree. Every file is registered via `fetcher.register_file(meta)`.
**What's needed**: This is essentially a duplicate of the tree data in a different shape. If the tree is loaded from storage, this map should be populated from the same source.
---
#### FileReader.manifests (~100MB at 1M files)
**Location**: `musicfs-cas/src/reader.rs`
**Contents**:
- `manifests: RwLock<HashMap<FileId, ChunkManifest>>` — maps FileId to list of chunk hashes + offsets
- Each `ChunkManifest` contains: file_id, total_size, mtime, chunks (Vec<ChunkRef> with hash + offset + size)
**Currently rebuilt from**: Re-fetched from origin on first `read()` after restart. The fetcher downloads the entire file, chunks it via CDC, stores chunks in CAS (dedup catches existing ones), and builds the manifest. This means every file is re-downloaded once after restart even though the chunks are already on disk.
**What's needed**: Persist manifests to storage after fetch. Load on mount. This is the difference between "restart = re-download everything" and "restart = instant reads from cache."
**Existing dead code**: SQLite `files` table has `chunk_manifest BLOB` column. `ChunkManifest::chunks_to_bytes()` and `ChunkManifest::from_db()` exist but are never called.
---
#### LruEviction access times (~50MB at 100K chunks)
**Location**: `musicfs-cache/src/eviction.rs`
**Contents**:
- `access_times: RwLock<BTreeMap<Instant, ChunkHash>>` — ordered by access time
- `hash_to_time: RwLock<HashMap<ChunkHash, Instant>>` — reverse lookup
**Currently rebuilt from**: Nothing. After restart, all chunks have equal eviction priority. The album you're currently listening to is just as likely to be evicted as something you played 6 months ago.
**What's needed**: Persist last-access timestamps. On mount, load and reconstruct the LRU order so hot data stays cached.
---
### 2.2 State That Survives But Is Ignored on Mount
These persist on disk but `run_mount()` never opens them.
| Component | Persisted To | Loaded on Mount? | Effect |
|---|---|---|---|
| SQLite metadata (files table) | `metadata.db` | ❌ | All metadata re-scanned from origin |
| tantivy search index | `search.idx/` | ❌ | Index rebuilt from scratch (or not at all) |
| PatternStore (access patterns) | SQLite (separate DB) | ❌ | Predictions reset to zero |
| CollectionStore (smart collections) | SQLite (same as patterns) | ❌ | Collections unavailable until opened |
### 2.3 State That Correctly Does Not Need Persistence
| Component | Why Transient Is Fine |
|---|---|
| OriginRegistry (origin connections) | Reconstructed from config on startup |
| Router (priorities, latency stats) | Priorities from config; latency stats warm up within seconds |
| HealthMonitor (health state) | All origins start as Unknown, converge within one check cycle (~30s) |
| EventBus (in-flight events) | Transient by nature |
| PrefetchEngine.in_flight | Transient work queue |
| PluginManager | Re-loaded from config + plugin directories |
| MusicFs.query_inodes | Transient search session state |
| CasStore.current_size | Recalculated on open (though currently broken — see resilience doc 3.10) |
---
## 3. Storage Decision
### 3.1 Requirements for Persistent State
1. **Bulk sequential read on mount** — load ~1M records into in-memory structures as fast as possible
2. **Incremental updates at runtime** — delta sync adds/removes/modifies individual files
3. **Crash safety** — no corruption on unclean shutdown (SIGKILL, power loss)
4. **Manifest storage** — binary blobs (msgpack-encoded chunk lists), variable size (100 bytes to 10KB per file)
5. **LRU timestamps** — simple key-value (ChunkHash → last_access_timestamp)
6. **Already in project** — minimize new dependencies
### 3.2 Options
#### Option A: SQLite (Current Architecture Choice)
**Already in project**: `rusqlite` dependency, `schema.sql` with `files` table, `Database` struct with full CRUD, `chunk_manifest BLOB` column ready.
| Metric | Performance |
|---|---|
| Bulk load 1M rows | ~2-4 seconds (WAL mode, indexed) |
| Single row upsert | ~50μs |
| Crash safety | WAL mode — excellent |
| Manifest blobs | Native BLOB support, no size limit |
**Pros**: Already built (schema, code, tests exist). Well-understood crash semantics. Single file backup. SQL queries for debugging. The `chunk_manifest` column and `from_db()`/`to_bytes()` methods are already written.
**Cons**: Not the fastest for pure key-value workloads. WAL checkpoint can cause brief write pauses. Single-writer limitation (Mutex around connection).
**Effort to wire up**: ~5-7 days (mostly connecting existing code, not writing new code)
---
#### Option B: sled (Already in Project for CAS Index)
**Already in project**: Used for CAS chunk hash → location mapping.
| Metric | Performance |
|---|---|
| Bulk load 1M entries | ~1-2 seconds (LSM, sequential reads) |
| Single entry upsert | ~10-20μs |
| Crash safety | Built-in WAL — good |
| Manifest blobs | Native byte value support |
**Pros**: Faster than SQLite for pure key-value. Already a dependency. Good for LRU timestamps (simple k/v).
**Cons**: No SQL — querying for debugging is harder. No schema migration story. Limited tooling. Has known issues with large datasets (memory usage during compaction). Two persistence engines = two things to maintain.
**Effort**: ~7-9 days (new serialization layer, no existing code to reuse)
---
#### Option C: Flat File (bincode/msgpack dump)
| Metric | Performance |
|---|---|
| Bulk load 1M entries | <1 second (mmap, zero-parse with bincode) |
| Single entry upsert | N/A — full rewrite required |
| Crash safety | Must write atomically (tmp + rename) |
| Manifest blobs | Part of serialized struct |
**Pros**: Fastest possible bulk load. Simplest implementation.
**Cons**: No incremental updates — every change requires serializing and rewriting the entire file. At 1M files (~500MB serialized), a single file modification triggers a 500MB write. No concurrent access. No recovery from partial corruption.
**Effort**: ~3-4 days but creates ongoing maintenance burden for delta updates
---
#### Option D: Hybrid (SQLite for metadata + sled for hot-path data)
Use SQLite for structured metadata (files, collections, patterns — already built) and sled for hot-path key-value data (manifests, LRU timestamps — performance-critical).
**Pros**: Each store optimized for its access pattern. SQLite for queryable metadata, sled for fast blob lookup.
**Cons**: Two persistence engines to coordinate. Consistency between them on crash. More complex startup/shutdown.
---
### 3.3 Recommendation
**Pending your decision.** The tradeoffs are:
- **Simplest path**: Option A (SQLite) — most code already exists, just needs wiring
- **Fastest hot-path**: Option D (Hybrid) — but more complexity
- **Fastest bulk load**: Option C (Flat file) — but no incremental updates
The choice depends on what you value most. SQLite at 1M files loads in ~2-4 seconds — is that acceptable vs the <500ms target? If not, a flat file or sled for the tree data with SQLite for everything else might be needed.
---
## 4. What Needs to Change
Regardless of storage choice, these are the code changes needed:
### 4.1 Mount Path (musicfs-cli/src/main.rs)
Current `run_mount()` flow:
```
1. Open CAS store → O(1)
2. Create origin connection → O(1)
3. scan_music_files() — FULL ORIGIN WALK → O(N × origin_latency) ← BOTTLENECK
4. Build tree from scan results → O(N)
5. Register files in fetcher → O(N)
6. Mount FUSE → O(1)
```
Required flow:
```
1. Open CAS store → O(1)
2. Open persistent state store → O(1)
3. IF store has data:
Load tree from store → O(N × local_read) ← ~1000x faster
Load manifests from store → O(N × local_read)
Load LRU access times from store → O(chunks)
ELSE (first mount):
Full origin scan (current behavior) → O(N × origin_latency)
Persist results to store → O(N × local_write)
4. Open tantivy search index → O(1)
5. Open PatternStore → O(1)
6. Create origin connections → O(1)
7. Mount FUSE → O(1)
8. Background: delta sync (origin vs store) → incremental, non-blocking
```
### 4.2 Runtime Persistence (Write Path)
These operations must persist state changes as they happen, not just on shutdown:
| Event | What to Persist | When |
|---|---|---|
| File discovered during sync | FileMeta → store | Immediately (in batch if scanning) |
| File removed during sync | Delete from store | Immediately |
| File metadata changed | Update FileMeta in store | Immediately |
| File content fetched (cache miss) | ChunkManifest → store | After fetch completes |
| Chunk accessed | Update LRU timestamp | Batched (every 10s or 100 accesses) |
| Search index updated | tantivy handles its own persistence | On commit (every 5s) |
| Access pattern recorded | PatternStore handles its own persistence | Already persisted per-access |
### 4.3 Files That Need Changes
| File | Change |
|---|---|
| `musicfs-cli/src/main.rs` | Rewrite `run_mount()` to load from store; add background delta sync |
| `musicfs-cache/src/db.rs` | Add `list_all_files()` bulk load; add manifest read/write methods (if SQLite) |
| `musicfs-cache/src/tree.rs` | Add `TreeBuilder::from_file_metas(iter)` — build tree from stored records |
| `musicfs-cas/src/reader.rs` | Load manifests from store on startup; persist after fetch |
| `musicfs-cas/src/fetcher.rs` | After `fetch_file()`, persist manifest to store |
| `musicfs-cache/src/eviction.rs` | Persist access times; load on startup |
| `musicfs-search/src/indexer.rs` | On mount, check what's already indexed vs what's in store — skip known files |
| `musicfs-sync/src/delta.rs` | Background delta sync: compare store state vs origin, sync differences |
### 4.4 Shutdown Persistence
On graceful shutdown (after signal handling from resilience plan Phase A is implemented):
| Step | What |
|---|---|
| 1 | Flush any batched LRU timestamp updates |
| 2 | Commit tantivy index writer |
| 3 | WAL checkpoint SQLite (if SQLite): `PRAGMA wal_checkpoint(TRUNCATE)` |
| 4 | Flush sled (if sled): `sled::Db::flush()` |
| 5 | Close all database connections |
On crash (no graceful shutdown):
- SQLite WAL mode: automatic recovery on next open (no data loss for committed transactions)
- sled: automatic recovery via internal WAL
- tantivy: up to 5 seconds of uncommitted documents lost, but recoverable from store
- LRU timestamps: batched updates may lose last batch (10s window) — acceptable
---
## 5. Background Delta Sync
After mounting from persistent state, the data may be stale (origin changed while daemon was stopped). A background sync reconciles:
```
1. Walk origin (or use watcher for inotify-capable origins)
2. For each file on origin:
a. Compare mtime + size against stored record
b. If unchanged → skip
c. If modified → re-parse metadata, update store, update tree, invalidate manifest
d. If new → parse metadata, add to store + tree
3. For each file in store not found on origin:
a. Remove from store + tree
4. Update search index for changed files
5. Log summary: "Delta sync complete: N added, M modified, K removed, T unchanged"
```
This runs in the background AFTER mount completes. Users see the filesystem immediately (from stored state), and it converges to current reality within minutes.
### 5.1 Stale Data Window
Between mount and delta sync completion, users may see:
- Files that were deleted on origin (will get ENOENT or EIO on read — origin returns not found)
- Files with old metadata (wrong track name, etc.)
- Missing files that were added to origin (won't appear until sync discovers them)
This is acceptable — it's the same behavior as any cached filesystem (NFS, CIFS). The key insight: **stale data for 30 seconds is infinitely better than no data for 5 minutes.**
---
## 6. First Mount vs Subsequent Mount
| | First Mount (empty store) | Subsequent Mount (store has data) |
|---|---|---|
| **Tree source** | Origin scan + metadata parse | Load from store |
| **Manifests** | None (populated on first read) | Loaded from store |
| **Search index** | Built during/after scan | Opened from disk |
| **LRU data** | Empty (cold cache) | Loaded from store |
| **Mount time** | O(N × origin_latency) — same as today | O(N × local_read) — target <5s for 1M files |
| **Accuracy** | 100% current | Stale until delta sync completes |
| **Detection** | Store file doesn't exist or is empty | Store file exists with data |
---
## 7. Estimated Effort
| Task | Effort | Depends On |
|---|---|---|
| Rewrite `run_mount()` with store loading + fallback | 2 days | Storage decision |
| Persist chunk manifests after fetch | 1 day | Storage decision |
| Load manifests on mount + register in FileReader | 0.5 day | Above |
| Open tantivy on mount, skip known files | 1 day | — |
| Open PatternStore + CollectionStore on mount | 0.5 day | — |
| Background delta sync | 1.5 days | — |
| Persist LRU access times + load on mount | 1 day | Storage decision |
| First-mount detection + fallback to full scan | 0.5 day | — |
| **Total** | **~8 days** | |
---
## 8. Open Decision
**Which storage engine for the persistent state?**
The answer drives the implementation of every task above. See Section 3 for tradeoffs.
docs/v2/plans/phase-a-stop-dying.md
-569
View File
@@ -1,569 +0,0 @@
# Phase A: Stop Dying — Implementation Plan
**Authors:** AI-assisted
**Status:** Draft
**Last Updated:** 2026-05-13
**Reviewers:** TBD
**Approvers:** TBD
**Prerequisites:** [resilience-fault-tolerance.md](resilience-fault-tolerance.md), [resilience-testing.md](resilience-testing.md)
**Estimated Effort:** ~5 days
---
[TOC]
---
## 1. Abstract
Implement the 6 most critical resilience fixes (issues 2.1, 2.2, 2.7, 2.9, 2.10, 3.7 from the [resilience audit](resilience-fault-tolerance.md)) that prevent MusicFS from dying on common operational events: signals, panics, lock poisoning, and systemd lifecycle.
Issues 2.3 (shutdown orchestration), 2.4 (cache integrity), 2.5 (sync recovery), 2.6 (task supervisor), 2.8 (disk space) are deferred to Phase B — they depend on Phase A infrastructure or on the [persistent state](persistent-state.md) work.
**Development flow** (TDD, per-issue):
1. Create stubs so the codebase compiles
2. Write RED tests that express the expected behavior
3. Implement the fix
4. Verify tests turn GREEN
5. Run full test suite — no regressions
---
## 2. Background
MusicFS currently dies on:
- Any signal (SIGTERM, SIGINT) — instant death, no cleanup
- Any panic in a writer thread — RwLock poisons, all FUSE ops crash
- systemd lifecycle — `Type=notify` but no `sd_notify`, ExecStop is a stub
- Crash leaves stale FUSE mount — users must manually `fusermount -u`
The [resilience test crate](../../musicfs/crates/musicfs-test-utils/) and RED tests are already in place. This plan implements the fixes to turn them GREEN.
---
## 3. Goals & Non-Goals
### 3.1 Goals
- Signal handler catches SIGTERM/SIGINT and initiates clean exit
- Panics are logged with full context before process terminates
- RwLock poisoning cannot cascade to kill FUSE operations
- systemd integration works (`sd_notify READY=1`, `ExecStopPost`)
- Stale FUSE mounts are detected and cleaned on startup
- All existing 162 tests continue to pass
- All Phase A RED tests turn GREEN
### 3.2 Non-Goals
- Graceful shutdown orchestration with ordered teardown (Phase B — needs CancellationToken plumbing through all components)
- Task supervisor for background task restart (Phase B)
- Cache integrity checks on startup (Phase B — needs persistent state)
- Disk space monitoring (Phase B)
- Interrupted sync recovery (Phase B — needs persistent state)
---
## 4. Proposed Design
### 4.1 Implementation Order
Dependencies determine the order. Each issue is independent except where noted.
```
4.2 RwLock poison fix (no deps, instant win, unblocks safety)
↓
4.3 Panic hook (no deps, complements RwLock fix)
↓
4.4 systemd ExecStopPost (no deps, config-only change)
↓
4.5 sd_notify integration (no deps, new crate dependency)
↓
4.6 Signal handling (depends on: FUSE mount change to spawn_mount2)
↓
4.7 Stale mount detection (depends on: signal handling for clean test)
```
### 4.2 Issue 2.9: RwLock Poison Fix
**Approach**: Replace `std::sync::RwLock` with `parking_lot::RwLock` in all production paths. `parking_lot` never poisons — a panic in a writer releases the lock and subsequent readers see the pre-panic state.
**Why parking_lot over poison recovery**: The codebase already uses `parking_lot` in `prefetch.rs` and `index.rs`. Using it everywhere is consistent. The alternative (`.unwrap_or_else(|p| p.into_inner())`) is verbose and error-prone — one missed call re-introduces the bug.
#### Step 1: Stubs (compile)
None needed — `parking_lot::RwLock` is a drop-in replacement (same API, no `PoisonError`).
#### Step 2: RED tests
Already exist in `tests/resilience.rs`:
- `test_poisoned_tree_lock_returns_eio_not_panic` — currently passes (demonstrates the problem)
- `test_parking_lot_rwlock_survives_panic` — currently passes (proves the fix works)
Additional test to add: verify FUSE filesystem survives a writer panic on the tree lock.
#### Step 3: Implementation
**Files to change:**
| File | Change |
|------|--------|
| `musicfs-fuse/src/filesystem.rs` | `use std::sync::RwLock` → `use parking_lot::RwLock`; remove all `.unwrap()` on lock calls (parking_lot returns guard directly, not `Result`) |
| `musicfs-cas/src/reader.rs` | Same change for `manifests: RwLock<HashMap<...>>` |
| `musicfs-cas/src/fetcher.rs` | Same change for `origins` and `file_meta` locks |
| `musicfs-origins/src/registry.rs` | Same change for `origins` and `watch_handles` locks |
| `musicfs-cache/src/eviction.rs` | Same change for `access_times` and `hash_to_time` locks |
| `musicfs-core/src/metrics.rs` | Same change for histogram locks |
| `musicfs-cache/src/tree.rs` | Same change for `last_refresh` lock |
**Pattern**: In each file:
```rust
// BEFORE
use std::sync::RwLock;
let guard = self.tree.read().unwrap();
// AFTER
use parking_lot::RwLock;
let guard = self.tree.read(); // No unwrap needed
```
For the `MusicFs` struct in `filesystem.rs`, the `tree` field is `Arc<RwLock<VirtualTree>>` — this is passed in from `main.rs`. Change `main.rs` to use `parking_lot::RwLock` there too.
#### Step 4: Verify
```bash
cargo test # All 162+ tests pass
cargo test -p musicfs-test-utils # Resilience tests pass
cargo check # No warnings
```
---
### 4.3 Issue 2.2: Panic Hook
**Approach**: Install a custom panic hook at daemon startup that logs the panic with `tracing::error!` before the default behavior (abort/unwind). This ensures panics are captured in log files and journald.
#### Step 1: Stubs
Add to `musicfs-core/src/lib.rs`:
```rust
pub fn install_panic_hook() {
// stub — will be implemented
}
```
#### Step 2: RED tests
Write in `tests/resilience.rs`:
```rust
#[test]
fn test_panic_hook_logs_to_tracing() {
// Install hook with a test tracing subscriber
// Trigger panic via catch_unwind
// Verify error! log contains panic message + thread name
}
```
#### Step 3: Implementation
**File**: `musicfs-core/src/lib.rs` (or new `musicfs-core/src/panic.rs`)
```rust
pub fn install_panic_hook() {
let default_hook = std::panic::take_hook();
std::panic::set_hook(Box::new(move |info| {
let thread = std::thread::current();
let thread_name = thread.name().unwrap_or("<unnamed>");
let message = if let Some(s) = info.payload().downcast_ref::<&str>() {
s.to_string()
} else if let Some(s) = info.payload().downcast_ref::<String>() {
s.clone()
} else {
"unknown panic".to_string()
};
let location = info.location().map(|l| format!("{}:{}:{}", l.file(), l.line(), l.column()))
.unwrap_or_else(|| "unknown location".to_string());
tracing::error!(
thread = thread_name,
location = %location,
"PANIC: {}",
message
);
default_hook(info);
}));
}
```
**Call site**: `musicfs-cli/src/main.rs`, at the very top of `main()`:
```rust
fn main() -> Result<()> {
musicfs_core::install_panic_hook();
let cli = Cli::parse();
// ...
}
```
#### Step 4: Verify
```bash
cargo test -p musicfs-core # Panic hook unit tests
cargo test -p musicfs-test-utils # Resilience tests
```
---
### 4.4 Issue 3.7 + 2.7: systemd Service Fix + FUSE Cleanup
**Approach**: Fix the systemd service file and add stale mount detection on startup.
#### Step 1: No stubs needed (config change)
#### Step 2: RED tests
Already exists: `test_systemd_service_has_execstoppost` — currently fails because service file lacks `ExecStopPost`.
Add test for stale mount detection:
```rust
#[test]
fn test_stale_mount_check_function_exists() {
// Verify the function signature exists
// (actual mount test needs privileged environment)
}
```
#### Step 3: Implementation
**File**: `dist/musicfs.service`
```diff
ExecStop=/usr/bin/musicfs shutdown
+ExecStopPost=/usr/bin/fusermount -uz %h/music || true
Restart=on-failure
```
Note: `fusermount -uz` is "lazy unmount" — always succeeds even if mount is busy. The `|| true` prevents systemd from treating cleanup failure as a service failure.
**File**: `musicfs-cli/src/main.rs` — add stale mount check before mounting:
```rust
fn check_stale_mount(mountpoint: &Path) -> Result<()> {
// Check /proc/mounts for existing mount at this path
if let Ok(mounts) = std::fs::read_to_string("/proc/mounts") {
for line in mounts.lines() {
if line.contains(&mountpoint.to_string_lossy().as_ref()) && line.contains("fuse") {
warn!("Stale FUSE mount detected at {:?}, attempting cleanup", mountpoint);
let status = std::process::Command::new("fusermount")
.args(["-uz", &mountpoint.to_string_lossy()])
.status();
match status {
Ok(s) if s.success() => info!("Stale mount cleaned up"),
Ok(s) => warn!("fusermount exited with: {}", s),
Err(e) => warn!("Failed to run fusermount: {}", e),
}
}
}
}
Ok(())
}
```
Also fix the `test_systemd_service_has_execstoppost` test path — currently points to wrong location.
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils -- test_systemd # Service file test
```
---
### 4.5 Issue 2.10: sd_notify Integration
**Approach**: Add `sd-notify` crate, call `READY=1` after mount, `STOPPING` on shutdown.
#### Step 1: Stubs
Add dependency to `musicfs-cli/Cargo.toml`:
```toml
sd-notify = "0.4"
```
#### Step 2: RED tests
Write test that mocks the notify socket:
```rust
#[test]
fn test_sd_notify_ready_sent() {
// Create Unix datagram socket at $NOTIFY_SOCKET
// Call sd_notify::notify(READY=1)
// Verify message received on socket
}
```
#### Step 3: Implementation
**File**: `musicfs-cli/src/main.rs`
After `fs.mount()` succeeds (or more precisely, after `spawn_mount2` — see 4.6):
```rust
// Notify systemd we're ready
if let Err(e) = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]) {
debug!("sd_notify not available (not running under systemd): {}", e);
}
```
On shutdown path:
```rust
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]);
```
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils -- test_sd_notify
cargo build -p musicfs-cli # Verify it compiles with new dep
```
---
### 4.6 Issue 2.1: Signal Handling
**Approach**: Switch from `fuser::mount2` (blocking) to `fuser::spawn_mount2` (background), then listen for signals on the main thread.
This is the most complex change in Phase A. It restructures the daemon's main loop.
#### Step 1: Stubs
Change `MusicFs::mount()` signature to return a session handle:
```rust
// BEFORE
pub fn mount(self, mountpoint: &Path) -> Result<()> {
fuser::mount2(self, mountpoint, &options)?;
Ok(())
}
// AFTER (stub — returns BackgroundSession)
pub fn spawn_mount(self, mountpoint: &Path) -> Result<fuser::BackgroundSession> {
let session = fuser::spawn_mount2(self, mountpoint, &options)?;
Ok(session)
}
```
Keep old `mount()` temporarily for compatibility.
#### Step 2: RED tests
Write in `tests/resilience.rs`:
```rust
#[tokio::test]
async fn test_sigterm_triggers_shutdown() {
// Spawn daemon as child process
// Wait for mount
// Send SIGTERM
// Verify clean exit within 10s
// Verify mountpoint is unmounted
}
```
This test requires the signal handler to exist. It will be RED until implementation.
#### Step 3: Implementation
**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()`:
```rust
fn run_mount(mountpoint: PathBuf, origin_path: Option<PathBuf>, cache_dir: Option<PathBuf>) -> Result<()> {
let origin_path = origin_path.context("--origin is required")?;
let runtime = tokio::runtime::Runtime::new()?;
let handle = runtime.handle().clone();
let (tree, reader) = runtime.block_on(async {
// ... existing setup code (unchanged) ...
Ok::<_, anyhow::Error>((tree, reader))
})?;
// Check for stale mount before mounting
check_stale_mount(&mountpoint)?;
let fs = MusicFs::with_reader(tree, reader, handle.clone());
info!("Mounting filesystem at {:?}", mountpoint);
// spawn_mount2 returns immediately — FUSE runs in background
let session = fs.spawn_mount(&mountpoint)
.context("Failed to mount filesystem")?;
// Notify systemd
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]);
info!("MusicFS ready, PID {}", std::process::id());
// Block on signal
runtime.block_on(async {
let mut sigterm = tokio::signal::unix::signal(
tokio::signal::unix::SignalKind::terminate()
)?;
let mut sigint = tokio::signal::unix::signal(
tokio::signal::unix::SignalKind::interrupt()
)?;
tokio::select! {
_ = sigterm.recv() => {
info!("Received SIGTERM, shutting down");
}
_ = sigint.recv() => {
info!("Received SIGINT, shutting down");
}
}
Ok::<_, anyhow::Error>(())
})?;
// Shutdown sequence
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]);
info!("Unmounting filesystem");
drop(session); // BackgroundSession::drop() calls unmount
info!("Shutdown complete");
Ok(())
}
```
**File**: `musicfs-fuse/src/filesystem.rs` — add `spawn_mount()`:
```rust
pub fn spawn_mount(self, mountpoint: &Path) -> Result<fuser::BackgroundSession> {
info!("Mounting MusicFS at {:?}", mountpoint);
let options = vec![
fuser::MountOption::RO,
fuser::MountOption::FSName("musicfs".to_string()),
fuser::MountOption::AutoUnmount,
fuser::MountOption::AllowOther,
];
let session = fuser::spawn_mount2(self, mountpoint, &options)
.map_err(musicfs_core::Error::Io)?;
Ok(session)
}
```
#### Step 4: Verify
```bash
cargo build -p musicfs-cli
cargo test -p musicfs-test-utils -- test_sigterm # Process-level test
cargo test # No regressions
```
---
## 5. Cross-Cutting Concerns
### 5.1 Security & Privacy
- No new attack surface — changes are internal lifecycle management
- Panic hook does NOT log sensitive data (only panic message, thread name, location)
- `sd_notify` uses existing systemd socket — no new IPC
### 5.2 Observability
- Panic hook ensures all panics are captured in logs/journald
- Signal handling logs which signal triggered shutdown
- sd_notify gives systemd accurate service state
- Stale mount detection logs cleanup attempts
### 5.3 Testing
All changes follow the TDD flow:
1. Stubs compile
2. RED tests document expected behavior
3. Implementation turns tests GREEN
4. Full suite passes (no regressions)
---
## 6. Alternatives Considered
### 6.1 Poison Recovery Instead of parking_lot
**Alternative**: Keep `std::sync::RwLock`, add `.unwrap_or_else(|p| p.into_inner())` to every lock call.
**Rejected**: 30+ call sites to change, easy to miss one, and the pattern is verbose. `parking_lot` is already a dependency and is strictly better for this use case (faster, no poison, correct API).
### 6.2 Keep mount2 (blocking) with Signal Thread
**Alternative**: Keep `fuser::mount2`, spawn a separate thread for signal handling, use a channel to communicate shutdown.
**Rejected**: `mount2` consumes `self` and blocks — there's no clean way to interrupt it from another thread. `spawn_mount2` is the canonical solution from the `fuser` crate.
### 6.3 Defer sd_notify Until Full Shutdown Orchestration
**Alternative**: Implement sd_notify only after CancellationToken + graceful shutdown are in place.
**Rejected**: sd_notify `READY=1` is critical now — without it, `Type=notify` in the service file means systemd will timeout and kill the daemon on every start. The shutdown `STOPPING` notification is a bonus but not required for Phase A.
---
## 7. Implementation Plan
### 7.1 Task Sequence
| Day | Task | Issue | Effort | Test Approach |
|-----|------|-------|--------|---------------|
| 1 (morning) | RwLock → parking_lot migration | 2.9 | 2h | Existing GREEN test validates; verify no `.unwrap()` on locks |
| 1 (afternoon) | Panic hook | 2.2 | 2h | New test: panic → verify tracing output |
| 2 (morning) | systemd ExecStopPost + stale mount check | 3.7 + 2.7 | 2h | Existing RED test → GREEN; new stale mount test |
| 2 (afternoon) | sd_notify integration | 2.10 | 2h | New test: mock socket → verify READY=1 |
| 3 | Signal handling (spawn_mount2 + signal loop) | 2.1 | 4h | Fork daemon → send SIGTERM → verify exit |
| 4 | Integration + regression testing | — | 4h | Full `cargo test`, manual FUSE mount test |
| 5 | Buffer for issues found during integration | — | 4h | — |
### 7.2 Verification Checklist
After all tasks complete:
- [ ] `cargo check` — zero errors, zero warnings
- [ ] `cargo test` — all 162+ existing tests pass
- [ ] `cargo test -p musicfs-test-utils` — all resilience tests pass
- [ ] `cargo clippy` — no new warnings
- [ ] `grep -r '\.read()\.unwrap()\|\.write()\.unwrap()' crates/` — zero hits in production code (test code is OK)
- [ ] `dist/musicfs.service` contains `ExecStopPost`
- [ ] Manual test: `musicfs mount`, then `kill -TERM <pid>`, verify clean exit + mount gone
- [ ] Manual test: `kill -9 <pid>`, then `musicfs mount` again — no "already mounted" error
---
## 8. Files Changed
| File | Change | Issue |
|------|--------|-------|
| `musicfs-fuse/src/filesystem.rs` | `std::sync::RwLock` → `parking_lot::RwLock`; add `spawn_mount()` | 2.9, 2.1 |
| `musicfs-cas/src/reader.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-cas/src/fetcher.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-origins/src/registry.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-cache/src/eviction.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-cache/src/tree.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-core/src/metrics.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
| `musicfs-core/src/lib.rs` | Add `install_panic_hook()` | 2.2 |
| `musicfs-cli/src/main.rs` | Panic hook, signal handler, spawn_mount2, sd_notify, stale mount check | 2.1, 2.2, 2.7, 2.10 |
| `musicfs-cli/Cargo.toml` | Add `sd-notify`, `tokio-util` deps | 2.10, 2.1 |
| `dist/musicfs.service` | Add `ExecStopPost`, fix `ExecStop` | 3.7 |
| `tests/resilience.rs` | Update/add tests for signal, panic hook, sd_notify | all |
---
## 9. Glossary / References
| Term | Definition |
|------|------------|
| **parking_lot** | Fast, poison-free lock implementation. Already a project dependency. |
| **spawn_mount2** | `fuser` API that mounts FUSE in a background thread, returning a `BackgroundSession` handle |
| **sd_notify** | systemd notification protocol. `READY=1` signals service started, `STOPPING` signals shutdown. |
| **BackgroundSession** | Handle returned by `spawn_mount2`. Dropping it unmounts the filesystem. |
| Document | Path |
|----------|------|
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
| Resilience testing | [resilience-testing.md](resilience-testing.md) |
| Architecture | [architecture.md](../architecture.md) |
docs/v2/plans/phase-b-crash-recovery.md
-830
View File
@@ -1,830 +0,0 @@
# Phase B: Crash Recovery — Implementation Plan
**Authors:** AI-assisted
**Status:** Draft
**Last Updated:** 2026-05-13
**Reviewers:** TBD
**Approvers:** TBD
**Prerequisites:** [phase-a-stop-dying.md](phase-a-stop-dying.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
**Estimated Effort:** ~5 days
---
[TOC]
---
## 1. Abstract
Phase A made the daemon survive signals and panics. Phase B makes it **recover from crashes** — startup integrity checks for all storage layers (SQLite, tantivy, sled), graceful shutdown with ordered teardown of background tasks, disk space pre-checks, and a task supervisor that restarts dead background tasks.
This covers issues 2.3, 2.4, 2.6, and 2.8 from the [resilience audit](resilience-fault-tolerance.md), deferred from Phase A.
Issue 2.5 (interrupted sync recovery) is deferred to after [persistent state](persistent-state.md) is wired up — checkpoint/resume requires the DB to be in the mount path.
**RED tests to turn GREEN** (from current `resilience.rs`):
- `test_sqlite_integrity_check_detects_corruption` — currently `todo!()`
- `test_tantivy_corruption_triggers_rebuild` — currently `todo!()`
- `test_sled_corruption_triggers_repair` — currently `todo!()`
- `test_cas_put_handles_enospc` — currently fails (no size pre-check)
- `test_tantivy_survives_uncommitted_crash` — currently `todo!()`
**New tests to write:**
- Shutdown orchestration: CancellationToken propagation, ordered teardown, tantivy flush
- Task supervisor: panic detection, restart with backoff, status reporting
---
## 2. Background
### 2.1 What Phase A Delivered
- Signal handling via `spawn_mount2` + tokio signal loop ✅
- Panic hook logging via `tracing::error!` ✅
- RwLock → `parking_lot` (no more poison cascade) ✅
- sd_notify READY/STOPPING ✅
- ExecStopPost + stale mount detection ✅
### 2.2 What's Still Broken After Phase A
The daemon now **stops cleanly** on signals but:
1. **Shutdown is unordered** — `drop(session)` unmounts FUSE, but background tasks (health monitor, indexer, watcher, prefetcher) are killed mid-operation by runtime drop. No tantivy flush, no SQLite checkpoint.
2. **No startup integrity checks** — if the daemon was `kill -9`'d (or OOM-killed, power loss), SQLite/tantivy/sled may have partial writes. Currently these propagate as runtime errors or silent corruption.
3. **Background tasks are fire-and-forget** — health monitor, watcher, indexer, prefetcher use `tokio::spawn` with no `JoinHandle` stored. If a task panics, it's silently dead.
4. **CAS accepts oversized writes** — `put()` doesn't check `max_size` before writing. Cache grows unbounded.
---
## 3. Goals & Non-Goals
### 3.1 Goals
- Graceful shutdown flushes tantivy, checkpoints SQLite WAL, stops background tasks in order
- Corrupted SQLite detected on open via `PRAGMA integrity_check`
- Corrupted tantivy index detected and rebuilt from scratch
- Corrupted sled index detected and repaired
- CAS rejects writes that would exceed `max_size`
- Background tasks are supervised — panics detected, critical tasks restarted
- All 5 RED tests turn GREEN
- All new tests for shutdown + supervisor are GREEN
### 3.2 Non-Goals
- Interrupted sync recovery (2.5) — depends on persistent state work
- Disk space monitoring daemon (periodic `statvfs`) — Phase C
- Connection pooling, config reload, watchdog — Phase C/D
- Passthrough mode when cache dies — Phase F
---
## 4. Proposed Design
### 4.1 Implementation Order
```
4.2 CAS size pre-check (no deps, simplest fix)
↓
4.3 SQLite integrity check (no deps)
↓
4.4 tantivy corruption recovery (no deps)
↓
4.5 sled corruption recovery (no deps)
↓
4.6 Graceful shutdown orchestration (depends on: Phase A signal handler)
↓
4.7 Task supervisor (depends on: 4.6 CancellationToken)
```
### 4.2 Issue 2.8: CAS Size Pre-Check
**Problem**: `CasStore::put()` writes data without checking if it would exceed `max_size`. The existing test `test_cas_put_handles_enospc` creates a store with `max_size: 100` and writes 1000 bytes — currently succeeds when it should fail.
#### Step 1: Stubs — none needed
#### Step 2: RED test — already exists
```rust
// Currently FAILS — this is what we need to fix
#[tokio::test]
async fn test_cas_put_handles_enospc() {
let store = CasStore::open(CasConfig { max_size: 100, ... }).await.unwrap();
let large_data = vec![0u8; 1000];
let result = store.put(&large_data).await;
assert!(result.is_err());
}
```
#### Step 3: Implementation
**File**: `musicfs-cas/src/store.rs` — add size check at top of `put()`:
```rust
pub async fn put(&self, data: &[u8]) -> Result<ChunkHash, CasError> {
let hash = ChunkHash::from_bytes(data);
let path = self.chunk_path(&hash);
if path.exists() {
trace!(hash = %hash, size_bytes = data.len(), "dedup hit");
return Ok(hash);
}
// NEW: Pre-check size limit
if self.config.max_size > 0 {
let new_size = self.current_size.load(Ordering::SeqCst) + data.len() as u64;
if new_size > self.config.max_size {
warn!(
current_size = self.current_size.load(Ordering::SeqCst),
chunk_size = data.len(),
max_size = self.config.max_size,
"CAS store full, rejecting write"
);
return Err(CasError::StoreFull {
current: self.current_size.load(Ordering::SeqCst),
max: self.config.max_size,
});
}
}
// ... rest of put() unchanged
}
```
Also add new error variant:
```rust
pub enum CasError {
// ... existing variants
#[error("Store full: {current} / {max} bytes")]
StoreFull { current: u64, max: u64 },
}
```
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_cas_put_handles_enospc
```
---
### 4.3 Issue 2.4 (part 1): SQLite Integrity Check
**Problem**: `Database::open()` runs schema but no integrity check. After crash, corrupt pages serve bad data silently.
#### Step 1: Stubs
Add to `musicfs-cache/src/db.rs`:
```rust
pub fn open_with_integrity_check(path: &Path) -> Result<Self> {
todo!()
}
```
#### Step 2: RED test — already exists as `todo!()`
Replace the `todo!()` with a real test:
```rust
#[tokio::test]
async fn test_sqlite_integrity_check_detects_corruption() {
let dir = TempDir::new().unwrap();
let db_path = dir.path().join("test.db");
// Create valid DB with data
{
let db = Database::open(&db_path).unwrap();
db.upsert_file(
&OriginId::from("test"),
Path::new("/test.flac"),
&VirtualPath::new("/Test.flac"),
&AudioMeta::default(),
UNIX_EPOCH,
1000,
).unwrap();
}
// Corrupt the file
let mut data = std::fs::read(&db_path).unwrap();
let mid = data.len() / 2;
data[mid..mid+100].fill(0xFF);
std::fs::write(&db_path, &data).unwrap();
// open_with_integrity_check should detect corruption
let result = Database::open_with_integrity_check(&db_path);
assert!(result.is_err());
}
```
#### Step 3: Implementation
**File**: `musicfs-cache/src/db.rs`
```rust
pub fn open_with_integrity_check(path: &Path) -> Result<Self> {
debug!(?path, "Opening database with integrity check");
let conn = Connection::open(path)
.map_err(|e| Error::Database(format!("open failed: {}", e)))?;
// Quick integrity check — verifies page-level consistency
let integrity: String = conn
.query_row("PRAGMA integrity_check(1)", [], |row| row.get(0))
.map_err(|e| Error::Database(format!("integrity check failed: {}", e)))?;
if integrity != "ok" {
warn!(path = ?path, result = %integrity, "Database integrity check failed");
return Err(Error::DatabaseCorrupted(format!(
"integrity check failed: {}", integrity
)));
}
conn.execute_batch(SCHEMA)
.map_err(|e| Error::Database(format!("schema init failed: {}", e)))?;
let db = Self { conn: Arc::new(Mutex::new(conn)) };
let count = db.file_count().unwrap_or(0);
info!(path = ?path, file_count = count, "Database opened (integrity verified)");
Ok(db)
}
```
Also add the error variant to `musicfs-core/src/error.rs`:
```rust
pub enum Error {
// ... existing
#[error("Database corrupted: {0}")]
DatabaseCorrupted(String),
}
```
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_sqlite_integrity
```
---
### 4.4 Issue 2.4 (part 2): tantivy Corruption Recovery
**Problem**: If tantivy `meta.json` or segment files are corrupted, `Index::open_in_dir()` panics or returns an error. No recovery path — daemon crashes.
#### Step 1: Stubs
Add to `musicfs-search/src/index.rs`:
```rust
pub fn open_with_recovery(index_path: &Path) -> Result<Self, SearchError> {
todo!()
}
```
#### Step 2: RED test — replace `todo!()` with real test
```rust
#[tokio::test]
async fn test_tantivy_corruption_triggers_rebuild() {
let dir = TempDir::new().unwrap();
let index_path = dir.path().join("search_idx");
// Create valid index with data
{
let index = SearchIndex::open(&index_path).unwrap();
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
index.commit().unwrap();
}
// Corrupt meta.json
std::fs::write(index_path.join("meta.json"), b"corrupted").unwrap();
// open_with_recovery should detect corruption and rebuild empty
let index = SearchIndex::open_with_recovery(&index_path).unwrap();
let results = index.search("a", 10).unwrap();
assert_eq!(results.len(), 0); // Rebuilt empty but functional
}
```
Also replace the tantivy crash test `todo!()`:
```rust
#[test]
fn test_tantivy_survives_uncommitted_crash() {
let dir = TempDir::new().unwrap();
let index_path = dir.path().join("search_idx");
{
let index = SearchIndex::open(&index_path).unwrap();
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
index.commit().unwrap();
// Write without commit, then "crash" (drop without commit)
index.index_file(&make_file_meta(2, "/b.flac", 1000)).unwrap();
// mem::forget would leak, just drop naturally
}
let index = SearchIndex::open(&index_path).unwrap();
let results = index.search("a", 10).unwrap();
assert_eq!(results.len(), 1); // Committed doc survives
}
```
#### Step 3: Implementation
**File**: `musicfs-search/src/index.rs`
```rust
pub fn open_with_recovery(index_path: &Path) -> Result<Self, SearchError> {
match Self::open(index_path) {
Ok(index) => {
// Verify index is functional with a simple search
match index.reader.searcher().num_docs() {
docs => {
info!(docs, "Search index opened successfully");
Ok(index)
}
}
}
Err(e) => {
warn!(
error = %e,
path = ?index_path,
"Search index corrupted, rebuilding from scratch"
);
// Delete corrupted index
if index_path.exists() {
std::fs::remove_dir_all(index_path)
.map_err(|e| SearchError::Io(e))?;
}
// Create fresh index
Self::open(index_path)
}
}
}
```
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_tantivy
```
---
### 4.5 Issue 3.5: sled Corruption Recovery
**Problem**: `sled::open()` on a corrupted DB returns `sled::Error::Corruption` which propagates as `CasError::Sled` and crashes the daemon on startup.
#### Step 1: Stubs — none needed, modify existing `open()`
#### Step 2: RED test — replace `todo!()`
```rust
#[tokio::test]
async fn test_sled_corruption_triggers_repair() {
let dir = TempDir::new().unwrap();
let chunks_dir = dir.path().join("chunks");
let config = CasConfig { chunks_dir: chunks_dir.clone(), max_size: 10_000_000, shard_levels: 2 };
// Create valid store with data
{
let store = CasStore::open(config.clone()).await.unwrap();
store.put(b"test data").await.unwrap();
}
// Corrupt sled index files
let sled_dir = chunks_dir.join("index.sled");
if sled_dir.exists() {
for entry in std::fs::read_dir(&sled_dir).unwrap() {
let entry = entry.unwrap();
if entry.metadata().unwrap().is_file() {
std::fs::write(entry.path(), b"corrupted").unwrap();
}
}
}
// Re-open should recover (repair or recreate)
let result = CasStore::open(config).await;
assert!(result.is_ok(), "sled should recover from corruption");
}
```
#### Step 3: Implementation
**File**: `musicfs-cas/src/store.rs` — modify `open()`:
```rust
pub async fn open(config: CasConfig) -> Result<Self, CasError> {
fs::create_dir_all(&config.chunks_dir).await?;
let index_path = config.chunks_dir.join("index.sled");
let index = match sled::open(&index_path) {
Ok(db) => db,
Err(e) => {
warn!(error = %e, path = ?index_path, "sled index corrupted, attempting recovery");
// Try repair
match sled::Config::new().path(&index_path).repair(true).open() {
Ok(db) => {
info!("sled index repaired successfully");
db
}
Err(repair_err) => {
warn!(error = %repair_err, "sled repair failed, recreating index");
// Delete and recreate
if index_path.exists() {
std::fs::remove_dir_all(&index_path)
.map_err(|e| CasError::Io(e))?;
}
sled::open(&index_path)?
}
}
}
};
let current_size = Self::calculate_size(&config.chunks_dir).await;
Ok(Self {
config,
index,
current_size: AtomicU64::new(current_size),
})
}
```
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_sled_corruption
```
---
### 4.6 Issue 2.3: Graceful Shutdown Orchestration
**Problem**: On signal, `drop(session)` unmounts FUSE, then `drop(runtime)` kills all tokio tasks abruptly. No tantivy flush, no SQLite WAL checkpoint, no ordered task shutdown.
**Approach**: `CancellationToken` from `tokio_util` propagated to all background tasks. Signal triggers token cancellation, then ordered shutdown.
#### Step 1: Add dependency
```toml
# musicfs-cli/Cargo.toml
tokio-util = { version = "0.7", features = ["rt"] }
```
#### Step 2: Tests
```rust
#[tokio::test]
async fn test_shutdown_cancels_background_tasks() {
let token = CancellationToken::new();
let stopped = Arc::new(AtomicBool::new(false));
let stopped_clone = stopped.clone();
let token_clone = token.clone();
tokio::spawn(async move {
token_clone.cancelled().await;
stopped_clone.store(true, Ordering::SeqCst);
});
assert!(!stopped.load(Ordering::SeqCst));
token.cancel();
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(stopped.load(Ordering::SeqCst));
}
#[tokio::test]
async fn test_shutdown_flushes_tantivy() {
let dir = TempDir::new().unwrap();
let index = SearchIndex::open(dir.path().join("idx")).unwrap();
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
// Graceful shutdown should commit
index.commit().unwrap();
let index2 = SearchIndex::open(dir.path().join("idx")).unwrap();
assert_eq!(index2.search("a", 10).unwrap().len(), 1);
}
```
#### Step 3: Implementation
**File**: `musicfs-cli/src/main.rs` — restructure the signal loop:
The current code:
```rust
// Wait for signal
runtime.block_on(async { ... signal select ... })?;
// Drop session, exit
```
Change to:
```rust
let shutdown_token = CancellationToken::new();
// TODO: Pass token to health monitor, watcher, indexer, prefetcher
// (requires their start() methods to accept CancellationToken)
// For now, we just use it for the shutdown sequence
runtime.block_on(async {
// ... signal select ...
// Ordered shutdown
info!("Beginning ordered shutdown");
shutdown_token.cancel();
// Wait briefly for tasks to notice cancellation
tokio::time::sleep(Duration::from_millis(500)).await;
// Flush search index if available
// (requires SearchIndex to be accessible — currently not wired in main.rs)
info!("Background tasks stopped");
})?;
```
**Note**: Full CancellationToken propagation through health monitor, watcher, indexer, and prefetcher `start()` methods requires changing their signatures. The current `mpsc::channel<()>` stop mechanism in each task should be replaced with or supplemented by the token. This can be done incrementally — start by adding the token to `run_mount()`, then wire it into each task as they're touched.
For this phase, the minimum viable change is:
1. Create the token in `run_mount()`
2. Cancel it on signal
3. Add a brief sleep for tasks to notice
4. The existing `drop(session)` and runtime drop handle cleanup
Full per-task CancellationToken wiring is tracked as follow-up work.
---
### 4.7 Issue 2.6: Task Supervisor
**Problem**: 13 `tokio::spawn()` calls with no `JoinHandle` stored. Dead tasks go unnoticed.
**Approach**: New `TaskSupervisor` struct in `musicfs-core` that stores handles, checks liveness, and restarts critical tasks.
#### Step 1: Stubs
**File**: `musicfs-core/src/supervisor.rs` (new file)
```rust
pub struct TaskSupervisor { ... }
pub enum TaskStatus {
Running,
Failed { error: String, at: Instant },
Restarting { attempt: u32 },
Stopped,
}
impl TaskSupervisor {
pub fn new() -> Self;
pub fn spawn_supervised(&self, name: &str, future: impl Future) -> ();
pub fn spawn_critical(&self, name: &str, factory: impl Fn() -> impl Future) -> ();
pub fn task_status(&self, name: &str) -> TaskStatus;
pub fn check_all(&self) -> Vec<(String, TaskStatus)>;
}
```
#### Step 2: Tests
```rust
#[tokio::test]
async fn test_supervisor_detects_task_completion() {
let supervisor = TaskSupervisor::new();
supervisor.spawn_supervised("fast", async { /* returns immediately */ });
tokio::time::sleep(Duration::from_millis(50)).await;
// Task completed normally — should be Stopped, not Failed
}
#[tokio::test]
async fn test_supervisor_detects_panic() {
let supervisor = TaskSupervisor::new();
supervisor.spawn_supervised("panicker", async {
panic!("boom");
});
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(matches!(supervisor.task_status("panicker"), TaskStatus::Failed { .. }));
}
#[tokio::test]
async fn test_supervisor_restarts_critical_task() {
let count = Arc::new(AtomicU32::new(0));
let c = count.clone();
let supervisor = TaskSupervisor::new();
supervisor.spawn_critical("restartable", move || {
let c = c.clone();
async move {
let n = c.fetch_add(1, Ordering::SeqCst);
if n == 0 { panic!("first run fails"); }
// Second run: stay alive
loop { tokio::time::sleep(Duration::from_secs(60)).await; }
}
});
tokio::time::sleep(Duration::from_secs(2)).await;
assert_eq!(count.load(Ordering::SeqCst), 2);
assert!(matches!(supervisor.task_status("restartable"), TaskStatus::Running));
}
```
#### Step 3: Implementation
**File**: `musicfs-core/src/supervisor.rs`
```rust
use parking_lot::RwLock;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::task::JoinHandle;
use tracing::{error, info, warn};
pub struct TaskSupervisor {
tasks: Arc<RwLock<HashMap<String, TaskEntry>>>,
}
struct TaskEntry {
handle: JoinHandle<()>,
status: TaskStatus,
restart_count: u32,
last_restart: Option<Instant>,
}
#[derive(Debug, Clone)]
pub enum TaskStatus {
Running,
Failed { error: String, at: Instant },
Restarting { attempt: u32 },
Stopped,
}
impl TaskSupervisor {
pub fn new() -> Self {
Self {
tasks: Arc::new(RwLock::new(HashMap::new())),
}
}
pub fn spawn_supervised<F>(&self, name: &str, future: F)
where
F: std::future::Future<Output = ()> + Send + 'static,
{
let tasks = self.tasks.clone();
let name_owned = name.to_string();
let handle = tokio::spawn(async move {
future.await;
});
// Monitor the handle
let tasks_monitor = self.tasks.clone();
let name_monitor = name.to_string();
let monitor_handle = handle;
self.tasks.write().insert(
name_owned,
TaskEntry {
handle: monitor_handle,
status: TaskStatus::Running,
restart_count: 0,
last_restart: None,
},
);
}
pub fn task_status(&self, name: &str) -> TaskStatus {
let mut tasks = self.tasks.write();
if let Some(entry) = tasks.get_mut(name) {
if entry.handle.is_finished() {
entry.status = TaskStatus::Failed {
error: "Task exited".into(),
at: Instant::now(),
};
}
entry.status.clone()
} else {
TaskStatus::Stopped
}
}
}
```
**Note**: The full `spawn_critical` with automatic restart requires a task factory (`Fn() -> Future`) pattern. The supervisor spawns a monitor task that awaits the `JoinHandle`, and on failure, calls the factory again with exponential backoff (1s→5s→30s, max 5 restarts). This is the most complex piece — the detailed implementation is in the test code above.
---
## 5. Cross-Cutting Concerns
### 5.1 Security & Privacy
- `PRAGMA integrity_check` is read-only — no risk to data
- sled repair may lose recently-written entries — acceptable for a cache
- tantivy rebuild deletes index entirely — no sensitive data exposure (metadata only)
### 5.2 Observability
- SQLite integrity check result logged at INFO (ok) or WARN (failed)
- sled repair attempts logged at WARN
- tantivy rebuild logged at WARN with file count before/after
- CAS `StoreFull` error logged at WARN with current/max sizes
- Task supervisor logs all state transitions (started, failed, restarting, stopped)
### 5.3 Testing
| Test | Status Before | Status After | Issue |
|------|---------------|--------------|-------|
| `test_cas_put_handles_enospc` | ❌ FAILED | ✅ GREEN | 2.8 |
| `test_sqlite_integrity_check_detects_corruption` | ❌ todo!() | ✅ GREEN | 2.4 |
| `test_tantivy_corruption_triggers_rebuild` | ❌ todo!() | ✅ GREEN | 2.4 |
| `test_tantivy_survives_uncommitted_crash` | ❌ todo!() | ✅ GREEN | 5.2 |
| `test_sled_corruption_triggers_repair` | ❌ todo!() | ✅ GREEN | 3.5 |
| `test_shutdown_cancels_background_tasks` | NEW | ✅ GREEN | 2.3 |
| `test_shutdown_flushes_tantivy` | NEW | ✅ GREEN | 2.3 |
| `test_supervisor_detects_panic` | NEW | ✅ GREEN | 2.6 |
| `test_supervisor_restarts_critical_task` | NEW | ✅ GREEN | 2.6 |
---
## 6. Alternatives Considered
### 6.1 Full `PRAGMA integrity_check` vs Quick Check
`PRAGMA integrity_check` scans every page — slow for large DBs (seconds for 1M rows). `PRAGMA integrity_check(1)` stops after the first error — fast enough for startup. We use the quick variant.
### 6.2 tantivy Repair vs Rebuild
tantivy has no built-in repair. If `meta.json` is corrupt or segments are missing, the only option is delete + recreate. This is acceptable because the search index can be rebuilt from SQLite metadata (once persistent state is wired up). For now, rebuild produces an empty index.
### 6.3 sled Repair vs Recreate
sled has `Config::repair(true)` which attempts to recover. If repair fails, we delete and recreate. After recreation, the index is empty but chunk files still exist on disk — a future reconciliation pass can rebuild the index from chunk files (Phase F).
### 6.4 Custom Supervisor vs `tokio-graceful` Crate
`tokio-graceful` provides shutdown coordination but not task restart. Our needs are specific (restart with backoff, status reporting, critical vs non-critical distinction). A custom `TaskSupervisor` is simpler and avoids a dependency for ~100 lines of code.
---
## 7. Implementation Plan
### 7.1 Task Sequence
| Day | Task | Issue | Effort | Test |
|-----|------|-------|--------|------|
| 1 (morning) | CAS size pre-check + `StoreFull` error variant | 2.8 | 1h | `test_cas_put_handles_enospc` → GREEN |
| 1 (afternoon) | SQLite `open_with_integrity_check` + `DatabaseCorrupted` error | 2.4 | 2h | `test_sqlite_integrity_check` → GREEN |
| 2 (morning) | tantivy `open_with_recovery` (detect + delete + recreate) | 2.4 | 2h | `test_tantivy_corruption` + `test_tantivy_survives_uncommitted_crash` → GREEN |
| 2 (afternoon) | sled recovery in `CasStore::open` (repair + fallback recreate) | 3.5 | 2h | `test_sled_corruption` → GREEN |
| 3 | Graceful shutdown with CancellationToken | 2.3 | 4h | `test_shutdown_cancels_background_tasks`, `test_shutdown_flushes_tantivy` → GREEN |
| 4 | Task supervisor implementation | 2.6 | 4h | `test_supervisor_detects_panic`, `test_supervisor_restarts` → GREEN |
| 5 | Integration + regression testing | — | 4h | Full `cargo test`, verify no regressions |
### 7.2 Verification Checklist
After all tasks:
- [ ] `cargo check` — zero errors, zero warnings
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all tests pass (exclude pre-existing grpc issue)
- [ ] `cargo test -p musicfs-test-utils --test resilience` — 5 previously-RED tests now GREEN
- [ ] `cargo clippy` — no new warnings
- [ ] Remaining RED tests are only for Phases C-F (health timeout, parallel checks, fd exhaustion, chunk auto-repair, passthrough mode)
---
## 8. Files Changed
| File | Change | Issue |
|------|--------|-------|
| `musicfs-cas/src/store.rs` | Size pre-check in `put()`, `StoreFull` error, sled recovery in `open()` | 2.8, 3.5 |
| `musicfs-cache/src/db.rs` | `open_with_integrity_check()` with `PRAGMA integrity_check(1)` | 2.4 |
| `musicfs-core/src/error.rs` | Add `DatabaseCorrupted(String)` variant | 2.4 |
| `musicfs-search/src/index.rs` | `open_with_recovery()` — detect, delete, recreate | 2.4 |
| `musicfs-core/src/supervisor.rs` | NEW — `TaskSupervisor`, `TaskStatus`, spawn/monitor/restart | 2.6 |
| `musicfs-core/src/lib.rs` | Re-export supervisor module | 2.6 |
| `musicfs-cli/src/main.rs` | CancellationToken creation, ordered shutdown sequence | 2.3 |
| `musicfs-cli/Cargo.toml` | Add `tokio-util` dependency | 2.3 |
| `musicfs-test-utils/tests/resilience.rs` | Replace `todo!()` stubs with real tests, add supervisor tests | all |
---
## 9. Glossary / References
| Term | Definition |
|------|------------|
| **CancellationToken** | `tokio_util::sync::CancellationToken` — cooperative cancellation signal for async tasks |
| **PRAGMA integrity_check** | SQLite command that verifies page-level data consistency |
| **sled repair** | sled's built-in recovery mode that attempts to reconstruct a corrupted database |
| **TaskSupervisor** | New struct that monitors `JoinHandle`s and restarts failed tasks with backoff |
| **StoreFull** | New `CasError` variant returned when a write would exceed `max_size` |
| Document | Path |
|----------|------|
| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
| Resilience testing | [resilience-testing.md](resilience-testing.md) |
| Persistent state | [persistent-state.md](persistent-state.md) |
docs/v2/plans/phase-c-hardening.md
-598
View File
@@ -1,598 +0,0 @@
# Phase C: Production Hardening — Implementation Plan
**Authors:** AI-assisted
**Status:** Draft
**Last Updated:** 2026-05-13
**Reviewers:** TBD
**Approvers:** TBD
**Prerequisites:** [phase-b-crash-recovery.md](phase-b-crash-recovery.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
**Estimated Effort:** ~4 days
---
[TOC]
---
## 1. Abstract
Phase C merges the practical items from Phases C and D of the resilience audit into a single implementation pass. It fixes the remaining 6 RED tests and addresses production-critical issues: health check hangs that block all origin monitoring, unbounded FUSE reads that can freeze the filesystem, broken CAS size accounting that disables eviction, and concurrent mount protection.
**Deferred items** (depend on unimplemented features or low urgency): interrupted sync recovery (needs persistent state), SIGHUP config reload, connection pooling (S3/SFTP are stubs), event bus backpressure, FUSE session recovery, offline mode state machine, DNS failure handling, stale-data awareness.
**RED tests to turn GREEN:**
- `test_local_origin_health_check_has_timeout` (D1)
- `test_health_checks_run_in_parallel` (D2)
- `test_fd_exhaustion_handling` (E — 5.3)
- `test_corrupt_chunk_auto_refetched` (F — 6.4)
- `test_missing_chunk_triggers_origin_fetch` (F — 6.4)
- `test_passthrough_mode_when_cache_disk_dead` (F — 6.6)
---
## 2. Background
After Phase A+B, the daemon survives signals, recovers from storage corruption on startup, supervises background tasks, and rejects oversized CAS writes. But:
1. **Health checks hang on dead origins** — `check_one()` calls `origin.health().await` with no timeout. A dead NAS (local origin pointing to network mount) blocks health monitoring for ALL origins because checks run sequentially.
2. **FUSE reads have no timeout** — `reader.read()` in the FUSE `read()` callback has no timeout. A slow or hung origin blocks the FUSE thread indefinitely.
3. **CAS size tracking is broken** — `calculate_size()` only scans top-level of `chunks_dir`, missing all chunks in shard subdirectories (`aa/bb/<hash>`). `current_size` is always ~0, eviction never triggers.
4. **Corrupt chunks return EIO** — when `verify_integrity()` detects a bad chunk, it returns `CasError::IntegrityError`. The reader propagates this as EIO to FUSE. It should auto-re-fetch from origin instead.
5. **No concurrent mount protection** — two `musicfs mount` commands can run simultaneously, corrupting SQLite and sled.
6. **fd exhaustion is unhandled** — no graceful behavior when file descriptors run out.
---
## 3. Goals & Non-Goals
### 3.1 Goals
- Health checks complete within 5 seconds regardless of origin responsiveness
- Health checks run in parallel (3 origins checked in ~5s, not ~15s)
- FUSE reads timeout after 30 seconds (returns EIO, doesn't hang)
- CAS size accounting is correct (recursive shard scan)
- Corrupt/missing chunks are auto-re-fetched from origin transparently
- PID file prevents concurrent mounts
- fd exhaustion produces clean errors, not panics
- All 6 remaining RED tests turn GREEN
### 3.2 Non-Goals
- Interrupted sync recovery (C1) — blocked on persistent state
- systemd watchdog (C3) — useful but not critical yet
- SIGHUP config reload (C4) — nice-to-have
- Connection pooling (C5) — S3/SFTP origins are stubs
- Event bus backpressure (C8) — low urgency
- FUSE session recovery (C10) — complex edge case
- Offline mode state machine (D3) — needs broader design
- DNS failure handling (D5) — depends on C5
- Stale-data awareness (D6) — low severity for music FS
---
## 4. Proposed Design
### 4.1 Implementation Order
```
4.2 Health check timeout + parallel checks (2 RED tests, independent)
↓
4.3 Fix CAS calculate_size() (independent, unblocks eviction)
↓
4.4 FUSE read timeout (independent)
↓
4.5 CAS chunk auto-re-fetch on corruption (2 RED tests)
↓
4.6 PID file / flock (independent)
↓
4.7 fd exhaustion handling (1 RED test)
```
### 4.2 Issues D1+D2: Health Check Timeout + Parallel Checks
**Problem**: `check_one()` awaits `origin.health()` with no timeout. `check_all()` iterates sequentially. One hung origin blocks everything.
#### Step 1: No stubs needed
#### Step 2: RED tests already exist
`test_local_origin_health_check_has_timeout` — FaultyOrigin with `TimeoutMs(5000)`, asserts check completes in <2s.
`test_health_checks_run_in_parallel` — 3 origins each with `TimeoutMs(200)`, asserts `check_all()` completes in <350ms (parallel), not ~600ms (sequential).
#### Step 3: Implementation
**File**: `musicfs-origins/src/health.rs`
Wrap `origin.health()` in `check_one()` with timeout:
```rust
async fn check_one(&self, id: &OriginId, origin: &Arc<dyn Origin>) {
let start = Instant::now();
let health_timeout = Duration::from_secs(5);
let status = match tokio::time::timeout(health_timeout, origin.health()).await {
Ok(status) => status,
Err(_) => {
warn!(origin_id = %id, timeout_ms = health_timeout.as_millis() as u64,
"Health check timed out");
HealthStatus::Unhealthy
}
};
let latency_ms = start.elapsed().as_millis() as u64;
// ... rest unchanged
}
```
Change `check_all()` to use `futures::future::join_all`:
```rust
pub async fn check_all(&self) {
let origins: Vec<_> = self.origins.iter()
.map(|e| (e.key().clone(), e.value().clone()))
.collect();
let checks: Vec<_> = origins.iter()
.map(|(id, origin)| self.check_one(id, origin))
.collect();
futures::future::join_all(checks).await;
}
```
Add `futures` to `musicfs-origins/Cargo.toml` (or use `tokio::join!` macro if count is small/known).
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_local_origin_health_check
cargo test -p musicfs-test-utils --test resilience -- test_health_checks_run_in_parallel
```
---
### 4.3 Issue C6: Fix CAS calculate_size()
**Problem**: `calculate_size()` only scans direct children of `chunks_dir`. Chunks live in shard subdirectories (`chunks/aa/bb/<hash>`). Size is always ~0, eviction never triggers.
#### Step 1: No stubs needed
#### Step 2: Test
```rust
#[tokio::test]
async fn test_cas_size_tracking_is_correct() {
let dir = TempDir::new().unwrap();
let config = CasConfig { chunks_dir: dir.path().join("chunks"), max_size: 10_000_000, shard_levels: 2 };
let store = CasStore::open(config).await.unwrap();
let data = vec![0u8; 1000];
store.put(&data).await.unwrap();
// Size should reflect the chunk we just wrote (~1000 bytes)
assert!(store.current_size() >= 1000, "current_size should track chunk data, got {}", store.current_size());
}
```
#### Step 3: Implementation
**File**: `musicfs-cas/src/store.rs` — make `calculate_size` recursive:
```rust
async fn calculate_size(dir: &Path) -> u64 {
Self::calculate_size_recursive(dir).await
}
#[async recursion::async_recursion]
async fn calculate_size_recursive(dir: &Path) -> u64 {
let mut size = 0u64;
if let Ok(mut entries) = fs::read_dir(dir).await {
while let Ok(Some(entry)) = entries.next_entry().await {
if let Ok(meta) = entry.metadata().await {
if meta.is_file() {
size += meta.len();
} else if meta.is_dir() {
// Skip sled index directory
let name = entry.file_name();
if name != "index.sled" {
size += Self::calculate_size_recursive(&entry.path()).await;
}
}
}
}
}
size
}
```
Alternative without `async_recursion` (use `Box::pin`):
```rust
fn calculate_size_recursive(dir: &Path) -> Pin<Box<dyn Future<Output = u64> + Send + '_>> {
Box::pin(async move {
let mut size = 0u64;
if let Ok(mut entries) = fs::read_dir(dir).await {
while let Ok(Some(entry)) = entries.next_entry().await {
if let Ok(meta) = entry.metadata().await {
if meta.is_file() {
size += meta.len();
} else if meta.is_dir() {
let name = entry.file_name();
if name != "index.sled" {
size += Self::calculate_size_recursive(&entry.path()).await;
}
}
}
}
}
size
})
}
```
---
### 4.4 Issue C7: FUSE Read Timeout
**Problem**: FUSE `read()` calls `handle.block_on(reader.read(...))` with no timeout. A slow origin blocks the entire FUSE thread.
#### Step 1: No stubs needed
#### Step 2: Test
```rust
#[tokio::test]
async fn test_fuse_read_timeout_returns_eio() {
// Uses FaultyOrigin with TimeoutMs(60_000) — simulates hung read
// FUSE read should timeout at 30s and return EIO, not hang forever
// (This test validates the timeout wrapper, not actual FUSE mount)
}
```
#### Step 3: Implementation
**File**: `musicfs-fuse/src/filesystem.rs` — wrap the read with timeout:
```rust
fn read(&mut self, _req: &Request, ino: u64, _fh: u64, offset: i64, size: u32, _flags: i32, _lock_owner: Option<u64>, reply: ReplyData) {
// ... file_id lookup unchanged ...
let reader = reader.clone();
let handle = self.runtime_handle.clone();
let result = std::thread::scope(|_| {
handle.block_on(async {
tokio::time::timeout(
Duration::from_secs(30),
reader.read(file_id, offset as u64, size),
).await
})
});
match result {
Ok(Ok(data)) => {
trace!(ino, bytes_read = data.len(), "read successful");
reply.data(&data);
}
Ok(Err(e)) => {
warn!(ino, error = %e, "read failed");
reply.error(libc::EIO);
}
Err(_timeout) => {
warn!(ino, offset, size, "read timed out after 30s");
reply.error(libc::EIO);
}
}
}
```
---
### 4.5 Issues 6.4: CAS Chunk Auto-Re-Fetch on Corruption/Missing
**Problem**: When `store.get()` finds a corrupt or missing chunk, it returns an error. The reader propagates this as EIO to FUSE. It should try to re-fetch the chunk from the origin instead.
#### Step 1: No stubs needed — modify `FileReader::read()`
#### Step 2: RED tests already exist
`test_corrupt_chunk_auto_refetched` — corrupts chunk file on disk, expects read to succeed (re-fetched from origin).
`test_missing_chunk_triggers_origin_fetch` — deletes chunk file, expects read to succeed.
Both currently fail because the reader doesn't attempt re-fetch on chunk errors.
#### Step 3: Implementation
**File**: `musicfs-cas/src/reader.rs` — add retry-with-refetch in the chunk read loop:
```rust
pub async fn read(&self, file_id: FileId, offset: u64, size: u32) -> Result<Bytes, ReaderError> {
let manifest = self.get_or_fetch_manifest(file_id).await?;
// ... offset/end calculation unchanged ...
for chunk_ref in &manifest.chunks {
// ... range check unchanged ...
let chunk_data = match self.store.get(&chunk_ref.hash).await {
Ok(data) => data,
Err(CasError::IntegrityError { .. }) | Err(CasError::NotFound(_)) => {
// Chunk is corrupt or missing — try to re-fetch from origin
warn!(hash = %chunk_ref.hash, "Chunk corrupt/missing, attempting re-fetch");
if let Some(fetcher) = &self.fetcher {
// Re-fetch the entire file (will re-chunk and store)
let new_manifest = fetcher.fetch_file(file_id).await?;
// Update cached manifest
self.manifests.write().insert(file_id, new_manifest);
// Retry the get
self.store.get(&chunk_ref.hash).await?
} else {
return Err(ReaderError::Cas(CasError::NotFound(chunk_ref.hash.as_hex())));
}
}
Err(e) => return Err(ReaderError::Cas(e)),
};
// ... slice extraction unchanged ...
}
Ok(result.freeze())
}
```
**Important**: The re-fetch downloads the entire file from origin and re-chunks it. For a single corrupt chunk this is wasteful (fetches all chunks to fix one), but it's the simplest correct approach. Chunk-level re-fetch would require the origin to support byte-range reads mapped to chunk boundaries — possible but complex. The file-level approach reuses existing `fetch_file()` logic.
#### Step 4: Verify
```bash
cargo test -p musicfs-test-utils --test resilience -- test_corrupt_chunk
cargo test -p musicfs-test-utils --test resilience -- test_missing_chunk
```
**Note on test updates**: The existing RED tests reference `store.chunk_path()` which is private. The tests will need to either:
- Make `chunk_path()` pub(crate) or add a test helper
- Or construct the path manually using the sharding logic
The tests also need a `ContentFetcher` with a real `LocalOrigin` to re-fetch from. The current tests create a CAS store but no fetcher — they need to be updated to include the full pipeline.
---
### 4.6 Issue C9: PID File / flock
**Problem**: Two `musicfs mount` commands can run simultaneously, both writing to the same SQLite/sled files.
#### Step 1: No stubs needed
#### Step 2: Test
```rust
#[test]
fn test_pid_file_prevents_concurrent_mount() {
let dir = TempDir::new().unwrap();
let lock_path = dir.path().join("musicfs.lock");
// First lock succeeds
let lock1 = try_acquire_lock(&lock_path);
assert!(lock1.is_ok());
// Second lock fails
let lock2 = try_acquire_lock(&lock_path);
assert!(lock2.is_err());
// Release first, second succeeds
drop(lock1);
let lock3 = try_acquire_lock(&lock_path);
assert!(lock3.is_ok());
}
```
#### Step 3: Implementation
**File**: `musicfs-cli/src/main.rs`
```rust
use std::fs::File;
use std::os::unix::io::AsRawFd;
struct LockFile {
_file: File,
}
fn try_acquire_lock(path: &Path) -> Result<LockFile> {
let file = File::create(path).context("Failed to create lock file")?;
let fd = file.as_raw_fd();
let ret = unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) };
if ret != 0 {
let err = std::io::Error::last_os_error();
if err.kind() == std::io::ErrorKind::WouldBlock {
anyhow::bail!("MusicFS is already running (lock file: {:?})", path);
}
return Err(err).context("Failed to acquire lock");
}
// Write PID for debugging
use std::io::Write;
let mut f = &file;
writeln!(f, "{}", std::process::id())?;
Ok(LockFile { _file: file })
}
```
Call in `run_mount()` before mounting:
```rust
let lock_path = cache_dir.join("musicfs.lock");
let _lock = try_acquire_lock(&lock_path)
.context("Failed to acquire lock — is another instance running?")?;
```
Lock is released automatically when `_lock` is dropped (process exit or scope end).
---
### 4.7 Issue 5.3: fd Exhaustion Handling
**Problem**: When fd limit is hit, operations fail with EMFILE. Currently this propagates as panics or unhelpful errors.
#### Step 1: Replace the `todo!()` test
#### Step 2: Test
```rust
#[test]
#[cfg(target_os = "linux")]
fn test_fd_exhaustion_handling() {
use rlimit::{Resource, setrlimit, getrlimit};
let (orig_soft, orig_hard) = getrlimit(Resource::NOFILE).unwrap();
// Set very low limit
setrlimit(Resource::NOFILE, 64, 64).unwrap();
let dir = TempDir::new().unwrap();
let rt = tokio::runtime::Runtime::new().unwrap();
let result = rt.block_on(async {
CasStore::open(CasConfig {
chunks_dir: dir.path().join("chunks"),
max_size: 1_000_000,
shard_levels: 2,
}).await
});
// Should either succeed (sled uses fewer than 64 fds) or fail gracefully
// Must NOT panic
match result {
Ok(_store) => { /* lucky — enough fds */ }
Err(e) => {
// Error message should be meaningful
let msg = format!("{}", e);
assert!(!msg.contains("panic"), "Should not panic on fd exhaustion");
}
}
setrlimit(Resource::NOFILE, orig_soft, orig_hard).unwrap();
}
```
#### Step 3: Implementation
This is primarily a **test** — verifying that existing code handles fd exhaustion without panicking. The fix is ensuring all I/O paths return `Result` rather than `.unwrap()` on file operations. Phase A's RwLock migration already removed the biggest panic source. The remaining `.unwrap()` calls are in test code only.
No production code change required if existing error paths handle I/O errors correctly. The test validates this.
---
## 5. Cross-Cutting Concerns
### 5.1 Observability
- Health check timeout logged at WARN with origin_id and timeout duration
- FUSE read timeout logged at WARN with inode, offset, size
- CAS chunk re-fetch logged at WARN with chunk hash
- PID file path logged at INFO on lock acquisition
### 5.2 Performance
- Health checks now parallel: O(1) wall-clock time instead of O(N) per check cycle
- FUSE read timeout: 30s cap prevents indefinite hangs but doesn't improve happy-path latency
- `calculate_size()` recursive scan: runs once at startup, negligible cost
### 5.3 Testing
| Test | Status Before | Status After | Issue |
|------|---------------|--------------|-------|
| `test_local_origin_health_check_has_timeout` | ❌ FAILED | ✅ GREEN | D1 |
| `test_health_checks_run_in_parallel` | ❌ FAILED | ✅ GREEN | D2 |
| `test_fd_exhaustion_handling` | ❌ todo!() | ✅ GREEN | 5.3 |
| `test_corrupt_chunk_auto_refetched` | ❌ FAILED | ✅ GREEN | 6.4 |
| `test_missing_chunk_triggers_origin_fetch` | ❌ FAILED | ✅ GREEN | 6.4 |
| `test_passthrough_mode_when_cache_disk_dead` | ❌ todo!() | ✅ GREEN | 6.6 |
| `test_cas_size_tracking_is_correct` | NEW | ✅ GREEN | C6 |
| `test_pid_file_prevents_concurrent_mount` | NEW | ✅ GREEN | C9 |
**Note on passthrough mode** (6.6): The test expects reads to succeed when the cache dir is read-only. With chunk auto-re-fetch (4.5), this partially works — if the origin is alive and the chunk isn't in cache, the fetcher reads from origin. But the fetcher tries to _write_ the chunk to CAS, which will fail on a read-only cache dir. The implementation needs a fallback path: if CAS write fails after origin fetch, return the data anyway without caching. This makes `test_passthrough_mode_when_cache_disk_dead` pass.
---
## 6. Alternatives Considered
### 6.1 Per-Origin Configurable Timeout vs Universal 5s
Could allow `health_check_timeout_ms` per origin config. Rejected for Phase C — universal 5s is correct for all current origin types. Can be made configurable later.
### 6.2 Chunk-Level Re-Fetch vs File-Level Re-Fetch
When one chunk is corrupt, we could re-fetch just that chunk's byte range from origin. Requires the origin to support byte-range reads and the system to know which byte range maps to which chunk. Complex. File-level re-fetch reuses existing `fetch_file()` and is correct, just slightly wasteful. Good enough for Phase C.
### 6.3 `advisory-lock` Crate vs Raw `flock`
The `advisory-lock` crate wraps flock nicely but adds a dependency for 10 lines of code. Raw `libc::flock` is simple enough and avoids the dependency.
---
## 7. Implementation Plan
### 7.1 Task Sequence
| Day | Task | Issue | Effort | Tests |
|-----|------|-------|--------|-------|
| 1 (morning) | Health check timeout in `check_one()` | D1 | 1h | `test_local_origin_health_check_has_timeout` → GREEN |
| 1 (morning) | Parallel `check_all()` with `join_all` | D2 | 1h | `test_health_checks_run_in_parallel` → GREEN |
| 1 (afternoon) | Fix `calculate_size()` recursion | C6 | 1h | `test_cas_size_tracking_is_correct` → GREEN |
| 1 (afternoon) | FUSE read timeout wrapper | C7 | 1h | New timeout test |
| 2 (morning) | CAS chunk auto-re-fetch on corruption/missing | 6.4 | 3h | `test_corrupt_chunk_auto_refetched` + `test_missing_chunk_triggers_origin_fetch` → GREEN |
| 2 (afternoon) | Passthrough fallback (CAS write fails → return data anyway) | 6.6 | 1h | `test_passthrough_mode_when_cache_disk_dead` → GREEN |
| 3 (morning) | PID file / flock | C9 | 1h | `test_pid_file_prevents_concurrent_mount` → GREEN |
| 3 (morning) | fd exhaustion test | 5.3 | 1h | `test_fd_exhaustion_handling` → GREEN |
| 3 (afternoon) | Integration + regression testing | — | 2h | Full `cargo test` |
| 4 | Buffer | — | 4h | — |
### 7.2 Verification Checklist
After all tasks:
- [ ] `cargo check` — zero errors, zero warnings
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass
- [ ] `cargo test -p musicfs-test-utils --test resilience` — **25 passed, 0 failed** (all RED tests GREEN)
- [ ] `cargo clippy` — no new warnings
---
## 8. Files Changed
| File | Change | Issue |
|------|--------|-------|
| `musicfs-origins/src/health.rs` | Timeout in `check_one()`, `join_all` in `check_all()` | D1, D2 |
| `musicfs-origins/Cargo.toml` | Add `futures` dependency (for `join_all`) | D2 |
| `musicfs-cas/src/store.rs` | Recursive `calculate_size()`, skip `index.sled` dir | C6 |
| `musicfs-fuse/src/filesystem.rs` | `tokio::time::timeout(30s)` around reader.read() | C7 |
| `musicfs-cas/src/reader.rs` | Auto-re-fetch on `IntegrityError` / `NotFound` | 6.4 |
| `musicfs-cas/src/fetcher.rs` | Possible: make `fetch_file` return data even if CAS write fails | 6.6 |
| `musicfs-cli/src/main.rs` | PID file with flock, fd exhaustion handling | C9, 5.3 |
| `musicfs-test-utils/tests/resilience.rs` | Replace remaining todo!()s, add new tests, update chunk tests with fetcher pipeline | all |
---
## 9. Glossary / References
| Term | Definition |
|------|------------|
| **join_all** | `futures::future::join_all` — runs multiple futures concurrently, waits for all |
| **flock** | Advisory file locking syscall — `LOCK_EX | LOCK_NB` for exclusive non-blocking |
| **EMFILE** | "Too many open files" errno — returned when process fd limit is reached |
| **Passthrough mode** | When CAS is unavailable, read directly from origin without caching |
| Document | Path |
|----------|------|
| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
| Phase B plan | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) |
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
docs/v2/plans/resilience-fault-tolerance.md
-1417
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/resilience-testing.md
-1003
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-01-foundation.md
-1126
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-02-metadata.md
-771
View File
@@ -1,771 +0,0 @@
# Week 2: Metadata Extraction
**Phase**: 1 (MVP)
**Prerequisites**: Week 1 (Foundation)
**Estimated effort**: 5 days
---
## Objective
Implement audio metadata extraction using symphonia and create SQLite schema for metadata cache.
---
## Deliverables
| Task | Crate | Files | Done |
|------|-------|-------|------|
| Audio parsing | musicfs-metadata | `lib.rs`, `parser.rs` | [ ] |
| Format handlers | musicfs-metadata | `formats/*.rs` | [ ] |
| SQLite schema | musicfs-cache | `schema.sql`, `db.rs` | [ ] |
| Metadata cache | musicfs-cache | `metadata.rs` | [ ] |
---
## Task 0: Extend AudioMeta in `musicfs-core`
Add `lyrics` and `composer` fields to `AudioMeta` struct (FR-6.4):
```rust
// In musicfs-core/src/types.rs, add to AudioMeta:
pub struct AudioMeta {
// ... existing fields ...
pub lyrics: Option<String>,
pub composer: Option<String>,
}
```
---
## Task 1: Metadata Parser (`musicfs-metadata`)
### 1.1 Create `Cargo.toml`
```toml
[package]
name = "musicfs-metadata"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
symphonia = { version = "0.5", features = ["all"] }
thiserror.workspace = true
tracing.workspace = true
```
### 1.2 Create `src/lib.rs`
```rust
mod parser;
pub use parser::MetadataParser;
```
### 1.3 Create `src/parser.rs`
```rust
use musicfs_core::{AudioFormat, AudioMeta, Result, Error};
use std::io::{Read, Seek};
use std::path::Path;
use symphonia::core::codecs::CODEC_TYPE_NULL;
use symphonia::core::formats::FormatOptions;
use symphonia::core::io::MediaSourceStream;
use symphonia::core::meta::MetadataOptions;
use symphonia::core::probe::Hint;
use tracing::debug;
/// Metadata extraction using symphonia (FR-6.1-6.5)
pub struct MetadataParser;
impl MetadataParser {
pub fn new() -> Self {
Self
}
/// Extract metadata from audio file
pub fn parse_file(&self, path: &Path) -> Result<AudioMeta> {
let file = std::fs::File::open(path)?;
let ext = path.extension()
.and_then(|e| e.to_str())
.unwrap_or("");
self.parse_reader(file, ext)
}
/// Extract metadata from reader
pub fn parse_reader<R: Read + Seek + Send + Sync + 'static>(
&self,
reader: R,
extension: &str,
) -> Result<AudioMeta> {
let mss = MediaSourceStream::new(Box::new(reader), Default::default());
let mut hint = Hint::new();
if !extension.is_empty() {
hint.with_extension(extension);
}
let format_opts = FormatOptions {
enable_gapless: false,
..Default::default()
};
let metadata_opts = MetadataOptions::default();
let probed = symphonia::default::get_probe()
.format(&hint, mss, &format_opts, &metadata_opts)
.map_err(|e| Error::Cache(format!("Failed to probe format: {}", e)))?;
let mut format = probed.format;
let mut audio_meta = AudioMeta {
format: AudioFormat::from_extension(extension),
..Default::default()
};
// Extract metadata from container
if let Some(metadata) = format.metadata().current() {
self.extract_tags(&mut audio_meta, metadata);
}
// Also check probed metadata
if let Some(metadata) = probed.metadata.current() {
self.extract_tags(&mut audio_meta, metadata);
}
// Get duration and codec info from track
if let Some(track) = format.tracks().iter().find(|t| t.codec_params.codec != CODEC_TYPE_NULL) {
let params = &track.codec_params;
if let Some(n_frames) = params.n_frames {
if let Some(sample_rate) = params.sample_rate {
audio_meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64);
audio_meta.sample_rate = Some(sample_rate);
}
}
if let Some(bits_per_sample) = params.bits_per_sample {
if let Some(sample_rate) = params.sample_rate {
if let Some(channels) = params.channels {
audio_meta.bitrate = Some(
bits_per_sample * sample_rate * channels.count() as u32 / 1000
);
}
}
}
}
debug!("Parsed metadata: {:?}", audio_meta);
Ok(audio_meta)
}
fn extract_tags(&self, meta: &mut AudioMeta, metadata: &symphonia::core::meta::MetadataRevision) {
use symphonia::core::meta::StandardTagKey;
for tag in metadata.tags() {
if let Some(std_key) = tag.std_key {
let value = tag.value.to_string();
match std_key {
StandardTagKey::TrackTitle => meta.title = Some(value),
StandardTagKey::Artist => meta.artist = Some(value),
StandardTagKey::Album => meta.album = Some(value),
StandardTagKey::AlbumArtist => meta.album_artist = Some(value),
StandardTagKey::Genre => meta.genre = Some(value),
StandardTagKey::TrackNumber => {
meta.track = value.split('/').next()
.and_then(|s| s.parse().ok());
}
StandardTagKey::DiscNumber => {
meta.disc = value.split('/').next()
.and_then(|s| s.parse().ok());
}
StandardTagKey::Date | StandardTagKey::ReleaseDate => {
meta.year = value.chars().take(4).collect::<String>()
.parse().ok();
}
StandardTagKey::Lyrics => {
meta.lyrics = Some(value);
}
StandardTagKey::Composer => {
meta.composer = Some(value);
}
_ => {}
}
}
}
}
}
impl Default for MetadataParser {
fn default() -> Self {
Self::new()
}
}
```
---
## Task 2: Cache Database (`musicfs-cache`)
### 2.1 Create `Cargo.toml`
```toml
[package]
name = "musicfs-cache"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
rusqlite = { workspace = true, features = ["bundled"] }
sled.workspace = true
tokio.workspace = true
tracing.workspace = true
thiserror.workspace = true
serde.workspace = true
rmp-serde.workspace = true
```
### 2.2 Create `src/lib.rs`
```rust
mod db;
mod metadata;
pub use db::Database;
pub use metadata::MetadataCache;
```
### 2.3 Create `src/schema.sql`
```sql
-- MusicFS Metadata Cache Schema
-- Per architecture.md section 4.3.6
-- NOTE: Chunk index stored in sled (chunks.sled/), NOT SQLite
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
PRAGMA synchronous = NORMAL;
CREATE TABLE IF NOT EXISTS files (
id INTEGER PRIMARY KEY,
origin_id TEXT NOT NULL,
real_path TEXT NOT NULL,
virtual_path TEXT NOT NULL,
-- Audio metadata (FR-6.1-6.5)
title TEXT,
artist TEXT,
album TEXT,
album_artist TEXT,
genre TEXT,
year INTEGER,
track INTEGER,
disc INTEGER,
duration_ms INTEGER,
bitrate INTEGER,
sample_rate INTEGER,
format TEXT,
-- Sync state
origin_mtime INTEGER NOT NULL,
origin_size INTEGER NOT NULL,
content_hash TEXT, -- hex-encoded xxHash64
chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)]
last_sync INTEGER NOT NULL DEFAULT (strftime('%s', 'now')),
UNIQUE(origin_id, real_path)
);
CREATE TABLE IF NOT EXISTS artwork (
id INTEGER PRIMARY KEY,
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
art_type TEXT NOT NULL, -- 'front', 'back', 'disc'
chunk_hash TEXT NOT NULL, -- hex-encoded reference to CAS
width INTEGER,
height INTEGER,
mime_type TEXT,
UNIQUE(file_id, art_type)
);
CREATE TABLE IF NOT EXISTS collections (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
query_json TEXT NOT NULL, -- smart collection query
created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
);
-- Indexes for performance (NFR-1.1, NFR-1.2)
CREATE INDEX IF NOT EXISTS idx_files_virtual ON files(virtual_path);
CREATE INDEX IF NOT EXISTS idx_files_artist_album ON files(artist, album);
CREATE INDEX IF NOT EXISTS idx_files_content_hash ON files(content_hash);
CREATE INDEX IF NOT EXISTS idx_files_real ON files(origin_id, real_path); -- FR-7.3
CREATE INDEX IF NOT EXISTS idx_files_origin ON files(origin_id);
CREATE INDEX IF NOT EXISTS idx_files_last_sync ON files(last_sync);
CREATE INDEX IF NOT EXISTS idx_artwork_file ON artwork(file_id);
```
### 2.4 Create `src/db.rs`
```rust
use musicfs_core::{AudioMeta, ContentHash, Error, FileId, FileMeta, OriginId, RealPath, Result, VirtualPath};
use rusqlite::{params, Connection, OptionalExtension};
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::time::{SystemTime, UNIX_EPOCH};
use tracing::{debug, info};
const SCHEMA: &str = include_str!("schema.sql");
/// SQLite database connection manager
pub struct Database {
conn: Arc<Mutex<Connection>>,
}
impl Database {
/// Open or create database at path
pub fn open(path: &Path) -> Result<Self> {
info!("Opening database at {:?}", path);
let conn = Connection::open(path)
.map_err(|e| Error::Database(e.to_string()))?;
// Execute schema
conn.execute_batch(SCHEMA)
.map_err(|e| Error::Database(e.to_string()))?;
Ok(Self {
conn: Arc::new(Mutex::new(conn)),
})
}
/// Open in-memory database (for testing)
pub fn open_memory() -> Result<Self> {
let conn = Connection::open_in_memory()
.map_err(|e| Error::Database(e.to_string()))?;
conn.execute_batch(SCHEMA)
.map_err(|e| Error::Database(e.to_string()))?;
Ok(Self {
conn: Arc::new(Mutex::new(conn)),
})
}
/// Insert or update file metadata
pub fn upsert_file(
&self,
origin_id: &OriginId,
real_path: &Path,
virtual_path: &VirtualPath,
audio_meta: &AudioMeta,
origin_mtime: SystemTime,
origin_size: u64,
) -> Result<FileId> {
let conn = self.conn.lock().unwrap();
let mtime_secs = origin_mtime
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs() as i64;
conn.execute(
r#"
INSERT INTO files (
origin_id, real_path, virtual_path,
title, artist, album, album_artist, genre,
year, track, disc,
duration_ms, bitrate, sample_rate, format,
origin_mtime, origin_size
) VALUES (
?1, ?2, ?3,
?4, ?5, ?6, ?7, ?8,
?9, ?10, ?11,
?12, ?13, ?14, ?15,
?16, ?17
)
ON CONFLICT(origin_id, real_path) DO UPDATE SET
virtual_path = excluded.virtual_path,
title = excluded.title,
artist = excluded.artist,
album = excluded.album,
album_artist = excluded.album_artist,
genre = excluded.genre,
year = excluded.year,
track = excluded.track,
disc = excluded.disc,
duration_ms = excluded.duration_ms,
bitrate = excluded.bitrate,
sample_rate = excluded.sample_rate,
format = excluded.format,
origin_mtime = excluded.origin_mtime,
origin_size = excluded.origin_size,
last_sync = strftime('%s', 'now')
"#,
params![
&origin_id.0,
real_path.to_string_lossy(),
virtual_path.as_str(),
&audio_meta.title,
&audio_meta.artist,
&audio_meta.album,
&audio_meta.album_artist,
&audio_meta.genre,
&audio_meta.year,
&audio_meta.track,
&audio_meta.disc,
&audio_meta.duration_ms.map(|d| d as i64),
&audio_meta.bitrate,
&audio_meta.sample_rate,
format!("{:?}", audio_meta.format),
mtime_secs,
origin_size as i64,
],
).map_err(|e| Error::Database(e.to_string()))?;
let id = conn.last_insert_rowid();
debug!("Upserted file {} with id {}", virtual_path.as_str(), id);
Ok(FileId(id))
}
/// Get file by virtual path
pub fn get_file_by_virtual_path(&self, path: &VirtualPath) -> Result<Option<FileMeta>> {
let conn = self.conn.lock().unwrap();
conn.query_row(
r#"
SELECT id, origin_id, real_path, virtual_path,
title, artist, album, album_artist, genre,
year, track, disc,
duration_ms, bitrate, sample_rate, format,
origin_mtime, origin_size, content_hash
FROM files
WHERE virtual_path = ?1
"#,
params![path.as_str()],
|row| {
Ok(FileMeta {
id: FileId(row.get(0)?),
real_path: RealPath {
origin_id: OriginId(row.get(1)?),
path: PathBuf::from(row.get::<_, String>(2)?),
},
virtual_path: VirtualPath::new(row.get::<_, String>(3)?),
audio: Some(AudioMeta {
title: row.get(4)?,
artist: row.get(5)?,
album: row.get(6)?,
album_artist: row.get(7)?,
genre: row.get(8)?,
year: row.get(9)?,
track: row.get(10)?,
disc: row.get(11)?,
duration_ms: row.get::<_, Option<i64>>(12)?.map(|d| d as u64),
bitrate: row.get(13)?,
sample_rate: row.get(14)?,
format: musicfs_core::AudioFormat::Unknown, // TODO: parse
}),
size: row.get::<_, i64>(17)? as u64,
mtime: UNIX_EPOCH + std::time::Duration::from_secs(row.get::<_, i64>(16)? as u64),
content_hash: row.get::<_, Option<Vec<u8>>>(18)?
.map(|b| ContentHash(b.try_into().unwrap_or([0; 8]))),
})
},
)
.optional()
.map_err(|e| Error::Database(e.to_string()))
}
/// Get file by ID
pub fn get_file_by_id(&self, id: FileId) -> Result<Option<FileMeta>> {
let conn = self.conn.lock().unwrap();
conn.query_row(
"SELECT virtual_path FROM files WHERE id = ?1",
params![id.0],
|row| row.get::<_, String>(0),
)
.optional()
.map_err(|e| Error::Database(e.to_string()))?
.map(|vp| self.get_file_by_virtual_path(&VirtualPath::new(vp)))
.transpose()
.map(|o| o.flatten())
}
/// List all files for an origin
pub fn list_files(&self, origin_id: &OriginId) -> Result<Vec<FileMeta>> {
let conn = self.conn.lock().unwrap();
let mut stmt = conn.prepare(
"SELECT virtual_path FROM files WHERE origin_id = ?1"
).map_err(|e| Error::Database(e.to_string()))?;
let paths: Vec<String> = stmt
.query_map(params![&origin_id.0], |row| row.get(0))
.map_err(|e| Error::Database(e.to_string()))?
.filter_map(|r| r.ok())
.collect();
drop(stmt);
drop(conn);
paths
.into_iter()
.filter_map(|p| self.get_file_by_virtual_path(&VirtualPath::new(p)).ok().flatten())
.collect::<Vec<_>>()
.pipe(Ok)
}
/// Delete file by ID
pub fn delete_file(&self, id: FileId) -> Result<()> {
let conn = self.conn.lock().unwrap();
conn.execute("DELETE FROM files WHERE id = ?1", params![id.0])
.map_err(|e| Error::Database(e.to_string()))?;
Ok(())
}
/// Get file count
pub fn file_count(&self) -> Result<u64> {
let conn = self.conn.lock().unwrap();
conn.query_row("SELECT COUNT(*) FROM files", [], |row| row.get::<_, i64>(0))
.map(|c| c as u64)
.map_err(|e| Error::Database(e.to_string()))
}
}
trait Pipe: Sized {
fn pipe<T>(self, f: impl FnOnce(Self) -> T) -> T {
f(self)
}
}
impl<T> Pipe for T {}
```
### 2.5 Create `src/metadata.rs`
```rust
use crate::db::Database;
use musicfs_core::{AudioMeta, FileMeta, OriginId, Result, VirtualPath};
use std::path::Path;
use std::sync::Arc;
use std::time::SystemTime;
/// High-level metadata cache interface
pub struct MetadataCache {
db: Arc<Database>,
}
impl MetadataCache {
pub fn new(db: Arc<Database>) -> Self {
Self { db }
}
/// Store file metadata
pub fn store(
&self,
origin_id: &OriginId,
real_path: &Path,
virtual_path: &VirtualPath,
audio_meta: &AudioMeta,
origin_mtime: SystemTime,
origin_size: u64,
) -> Result<()> {
self.db.upsert_file(
origin_id,
real_path,
virtual_path,
audio_meta,
origin_mtime,
origin_size,
)?;
Ok(())
}
/// Lookup by virtual path
pub fn lookup(&self, path: &VirtualPath) -> Result<Option<FileMeta>> {
self.db.get_file_by_virtual_path(path)
}
/// Check if file exists and is fresh
pub fn is_fresh(
&self,
origin_id: &OriginId,
real_path: &Path,
current_mtime: SystemTime,
) -> Result<bool> {
// TODO: Compare mtime with cached value
Ok(false)
}
}
```
---
## Tests
### Unit Tests (`musicfs-metadata`)
```rust
#[cfg(test)]
mod tests {
use super::*;
use std::io::Cursor;
#[test]
fn test_parse_flac_metadata() {
// Use a real FLAC file for testing
// For CI, embed a small test file or use a fixture
let parser = MetadataParser::new();
// This would need a real file path
// let meta = parser.parse_file(Path::new("test.flac")).unwrap();
// assert!(meta.title.is_some());
}
#[test]
fn test_audio_format_detection() {
assert_eq!(AudioFormat::from_extension("flac"), AudioFormat::Flac);
assert_eq!(AudioFormat::from_extension("mp3"), AudioFormat::Mp3);
assert_eq!(AudioFormat::from_extension("opus"), AudioFormat::Opus);
}
}
```
### Unit Tests (`musicfs-cache`)
```rust
#[cfg(test)]
mod tests {
use super::*;
use musicfs_core::{AudioFormat, AudioMeta, OriginId, VirtualPath};
use std::time::UNIX_EPOCH;
#[test]
fn test_database_creation() {
let db = Database::open_memory().unwrap();
assert_eq!(db.file_count().unwrap(), 0);
}
#[test]
fn test_upsert_and_retrieve() {
let db = Database::open_memory().unwrap();
let origin_id = OriginId::from("local");
let real_path = Path::new("/music/test.flac");
let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac");
let audio_meta = AudioMeta {
title: Some("Track".to_string()),
artist: Some("Artist".to_string()),
album: Some("Album".to_string()),
track: Some(1),
format: AudioFormat::Flac,
..Default::default()
};
let id = db.upsert_file(
&origin_id,
real_path,
&virtual_path,
&audio_meta,
UNIX_EPOCH,
1000,
).unwrap();
let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap();
assert_eq!(retrieved.id, id);
assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Track".to_string()));
}
#[test]
fn test_upsert_updates_existing() {
let db = Database::open_memory().unwrap();
let origin_id = OriginId::from("local");
let real_path = Path::new("/music/test.flac");
let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac");
// First insert
let meta1 = AudioMeta {
title: Some("Original".to_string()),
..Default::default()
};
db.upsert_file(&origin_id, real_path, &virtual_path, &meta1, UNIX_EPOCH, 1000).unwrap();
// Update
let meta2 = AudioMeta {
title: Some("Updated".to_string()),
..Default::default()
};
db.upsert_file(&origin_id, real_path, &virtual_path, &meta2, UNIX_EPOCH, 1000).unwrap();
// Should still be 1 file
assert_eq!(db.file_count().unwrap(), 1);
// Title should be updated
let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap();
assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Updated".to_string()));
}
#[test]
fn test_metadata_persistence() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("test.db");
// Create and populate
{
let db = Database::open(&db_path).unwrap();
db.upsert_file(
&OriginId::from("local"),
Path::new("/test.flac"),
&VirtualPath::new("/Test.flac"),
&AudioMeta::default(),
UNIX_EPOCH,
100,
).unwrap();
}
// Reopen and verify
{
let db = Database::open(&db_path).unwrap();
assert_eq!(db.file_count().unwrap(), 1);
}
}
}
```
---
## Exit Criteria
- [ ] Parse FLAC metadata (title, artist, album, track, duration)
- [ ] Parse MP3 metadata (ID3v2 and ID3v1 fallback)
- [ ] Parse Opus/Vorbis comments
- [ ] Parse M4A/AAC metadata
- [ ] Handle missing metadata gracefully (FR-6.5)
- [ ] SQLite schema creates all tables
- [ ] Metadata persists across daemon restarts (FR-7.4)
- [ ] Upsert correctly updates existing records
---
## Verification Commands
```bash
# Run metadata tests
cargo test -p musicfs-metadata
# Run cache tests
cargo test -p musicfs-cache
# Test with real audio file
cargo run --example parse_metadata -- /path/to/test.flac
```
---
## Next Week
Week 3 will implement the virtual path resolver and tree cache, connecting metadata to the FUSE operations.
docs/v2/plans/week-03-virtual-tree.md
-1069
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-04-cas-caching.md
-1089
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-04b-origin-connector.md
-501
View File
@@ -1,501 +0,0 @@
# Week 4b: Origin-CAS Connector
**Phase**: 1 (MVP)
**Prerequisites**: Week 4 (CAS & Chunk Caching)
**Estimated effort**: 1 day
---
## Objective
Bridge the gap between Origin (source files) and CAS (chunk cache) to enable actual file reads through FUSE. This implements the "cache miss" flow from architecture section 4.3.5.
**Problem**: Week 4 implemented CAS storage and FileReader, but there's no code that:
1. Detects when requested chunks aren't cached
2. Fetches data from Origin
3. Stores chunks in CAS
4. Creates ChunkManifest for the file
**Solution**: Create `ContentFetcher` that orchestrates Origin → CAS data flow on cache miss.
---
## Architecture Reference
From architecture.md section 4.3.5 (Read Operation Activity):
```
|CAS|
:compute chunk range for [offset, offset+size];
if (all chunks cached?) then (yes)
:read from local chunk files;
else (no)
|OriginFederation|
:select healthy origin by priority;
:fetch missing byte range;
|CAS|
:chunk fetched data (CDC);
:store chunks by hash;
:update chunk manifest;
endif
```
---
## Deliverables
| Task | Crate | Files | Done |
|------|-------|-------|------|
| ContentFetcher implementation | musicfs-cas | `fetcher.rs` | [ ] |
| FileId → FileMeta resolver | musicfs-cas | `fetcher.rs` | [ ] |
| Update FileReader for cache-miss | musicfs-cas | `reader.rs` | [ ] |
| Update FUSE with fetcher | musicfs-fuse | `filesystem.rs` | [ ] |
| E2E test: cat file through FUSE | tests | `integration.rs` | [ ] |
---
## Task 1: ContentFetcher
### 1.1 Create `musicfs-cas/src/fetcher.rs`
```rust
use crate::{CasStore, ChunkManifest, ChunkRef};
use musicfs_core::{Event, EventBus, FileId, FileMeta, OriginId, RealPath};
use musicfs_origins::Origin;
use std::collections::HashMap;
use std::path::Path;
use std::sync::{Arc, RwLock};
use tracing::{debug, info};
pub struct ContentFetcher {
store: Arc<CasStore>,
origins: RwLock<HashMap<OriginId, Arc<dyn Origin>>>,
file_meta: RwLock<HashMap<FileId, FileMeta>>,
event_bus: Option<Arc<EventBus>>,
}
impl ContentFetcher {
pub fn new(store: Arc<CasStore>) -> Self {
Self {
store,
origins: RwLock::new(HashMap::new()),
file_meta: RwLock::new(HashMap::new()),
event_bus: None,
}
}
pub fn with_event_bus(store: Arc<CasStore>, event_bus: Arc<EventBus>) -> Self {
Self {
store,
origins: RwLock::new(HashMap::new()),
file_meta: RwLock::new(HashMap::new()),
event_bus: Some(event_bus),
}
}
pub fn register_origin(&self, origin: Arc<dyn Origin>) {
let id = origin.id().clone();
self.origins.write().unwrap().insert(id, origin);
}
pub fn register_file(&self, meta: FileMeta) {
self.file_meta.write().unwrap().insert(meta.id, meta);
}
pub fn register_files(&self, files: impl IntoIterator<Item = FileMeta>) {
let mut map = self.file_meta.write().unwrap();
for meta in files {
map.insert(meta.id, meta);
}
}
pub async fn fetch_file(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
let meta = {
let files = self.file_meta.read().unwrap();
files.get(&file_id).cloned()
.ok_or(FetchError::FileNotFound(file_id))?
};
let origin = {
let origins = self.origins.read().unwrap();
origins.get(&meta.real_path.origin_id).cloned()
.ok_or_else(|| FetchError::OriginNotFound(meta.real_path.origin_id.clone()))?
};
info!("Fetching file {:?} from origin {}", file_id, origin.id());
let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await
.map_err(|e| FetchError::OriginRead(e.to_string()))?;
let hash = self.store.put(&data).await
.map_err(FetchError::Store)?;
let manifest = ChunkManifest {
file_id,
total_size: meta.size,
chunks: vec![ChunkRef {
hash,
offset: 0,
size: data.len() as u32,
}],
};
debug!("Created manifest for {:?}: {} bytes, 1 chunk", file_id, meta.size);
Ok(manifest)
}
pub fn emit_access_event(&self, meta: &FileMeta, offset: u64, size: u32) {
if let Some(bus) = &self.event_bus {
bus.publish(Event::FileAccessed {
path: meta.virtual_path.clone(),
origin_id: meta.real_path.origin_id.clone(),
offset,
size,
});
}
}
pub async fn ensure_cached(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
self.fetch_file(file_id).await
}
pub fn get_file_meta(&self, file_id: FileId) -> Option<FileMeta> {
self.file_meta.read().unwrap().get(&file_id).cloned()
}
}
#[derive(Debug, thiserror::Error)]
pub enum FetchError {
#[error("File not found: {0:?}")]
FileNotFound(FileId),
#[error("Origin not found: {0}")]
OriginNotFound(OriginId),
#[error("Origin read error: {0}")]
OriginRead(String),
#[error("Store error: {0}")]
Store(#[from] crate::CasError),
}
#[cfg(test)]
mod tests {
use super::*;
use crate::CasConfig;
use musicfs_core::VirtualPath;
use musicfs_origins::LocalOrigin;
use std::path::PathBuf;
use std::time::SystemTime;
use tempfile::TempDir;
#[tokio::test]
async fn test_fetch_file() {
let cas_dir = TempDir::new().unwrap();
let origin_dir = TempDir::new().unwrap();
std::fs::write(origin_dir.path().join("test.flac"), b"fake audio data").unwrap();
let config = CasConfig {
chunks_dir: cas_dir.path().join("chunks"),
..Default::default()
};
let store = Arc::new(CasStore::open(config).await.unwrap());
let fetcher = ContentFetcher::new(store.clone());
let origin = Arc::new(LocalOrigin::new("local", origin_dir.path()));
fetcher.register_origin(origin);
let meta = FileMeta {
id: FileId(1),
virtual_path: VirtualPath::new("/Artist/Album/test.flac"),
real_path: RealPath {
origin_id: OriginId::from("local"),
path: PathBuf::from("/test.flac"),
},
size: 15,
mtime: SystemTime::now(),
content_hash: None,
audio: None,
};
fetcher.register_file(meta);
let manifest = fetcher.fetch_file(FileId(1)).await.unwrap();
assert_eq!(manifest.total_size, 15);
assert_eq!(manifest.chunks.len(), 1);
let data = store.get(&manifest.chunks[0].hash).await.unwrap();
assert_eq!(&data[..], b"fake audio data");
}
#[tokio::test]
async fn test_fetch_file_not_found() {
let cas_dir = TempDir::new().unwrap();
let config = CasConfig {
chunks_dir: cas_dir.path().join("chunks"),
..Default::default()
};
let store = Arc::new(CasStore::open(config).await.unwrap());
let fetcher = ContentFetcher::new(store);
let result = fetcher.fetch_file(FileId(999)).await;
assert!(matches!(result, Err(FetchError::FileNotFound(_))));
}
#[tokio::test]
async fn test_fetch_emits_event() {
let cas_dir = TempDir::new().unwrap();
let origin_dir = TempDir::new().unwrap();
std::fs::write(origin_dir.path().join("test.flac"), b"audio").unwrap();
let config = CasConfig {
chunks_dir: cas_dir.path().join("chunks"),
..Default::default()
};
let store = Arc::new(CasStore::open(config).await.unwrap());
let event_bus = Arc::new(EventBus::default());
let mut rx = event_bus.subscribe();
let fetcher = ContentFetcher::with_event_bus(store, event_bus);
let origin = Arc::new(LocalOrigin::new("local", origin_dir.path()));
fetcher.register_origin(origin);
let meta = FileMeta {
id: FileId(1),
virtual_path: VirtualPath::new("/Artist/test.flac"),
real_path: RealPath {
origin_id: OriginId::from("local"),
path: PathBuf::from("/test.flac"),
},
size: 5,
mtime: SystemTime::now(),
content_hash: None,
audio: None,
};
fetcher.register_file(meta.clone());
fetcher.emit_access_event(&meta, 0, 5);
let event = rx.try_recv().unwrap();
assert!(matches!(event, Event::FileAccessed { .. }));
}
}
```
---
## Task 2: Update FileReader
### 2.1 Update `musicfs-cas/src/reader.rs`
Add fetcher integration for cache-miss handling:
```rust
use crate::fetcher::{ContentFetcher, FetchError};
pub struct FileReader {
store: Arc<CasStore>,
fetcher: Option<Arc<ContentFetcher>>,
manifests: RwLock<HashMap<FileId, ChunkManifest>>,
}
impl FileReader {
pub fn new(store: Arc<CasStore>) -> Self {
Self {
store,
fetcher: None,
manifests: RwLock::new(HashMap::new()),
}
}
pub fn with_fetcher(store: Arc<CasStore>, fetcher: Arc<ContentFetcher>) -> Self {
Self {
store,
fetcher: Some(fetcher),
manifests: RwLock::new(HashMap::new()),
}
}
pub async fn read(
&self,
file_id: FileId,
offset: u64,
size: u32,
) -> Result<Bytes, ReaderError> {
let manifest = self.get_or_fetch_manifest(file_id).await?;
if let Some(fetcher) = &self.fetcher {
if let Some(meta) = fetcher.get_file_meta(file_id) {
fetcher.emit_access_event(&meta, offset, size);
}
}
// ... rest of read logic unchanged
}
async fn get_or_fetch_manifest(&self, file_id: FileId) -> Result<ChunkManifest, ReaderError> {
{
let manifests = self.manifests.read().unwrap();
if let Some(m) = manifests.get(&file_id) {
return Ok(m.clone());
}
}
let Some(fetcher) = &self.fetcher else {
return Err(ReaderError::ManifestNotFound(file_id));
};
let manifest = fetcher.ensure_cached(file_id).await
.map_err(ReaderError::Fetch)?;
self.manifests.write().unwrap().insert(file_id, manifest.clone());
Ok(manifest)
}
}
#[derive(Debug, thiserror::Error)]
pub enum ReaderError {
#[error("Manifest not found for file {0:?}")]
ManifestNotFound(FileId),
#[error("Fetch error: {0}")]
Fetch(#[from] FetchError),
#[error("CAS error: {0}")]
Cas(#[from] crate::CasError),
}
```
---
## Task 3: Update lib.rs
### 3.1 Update `musicfs-cas/src/lib.rs`
```rust
mod chunks;
mod fetcher;
mod reader;
mod store;
pub use chunks::{ChunkLocation, ChunkRef};
pub use fetcher::{ContentFetcher, FetchError};
pub use reader::{ChunkManifest, FileReader, ReaderError};
pub use store::{CasConfig, CasError, CasStore, DedupStats};
```
---
## Task 4: Update Cargo.toml
### 4.1 Update `musicfs-cas/Cargo.toml`
```toml
[dependencies]
musicfs-core = { path = "../musicfs-core" }
musicfs-origins = { path = "../musicfs-origins" }
# ... rest unchanged
```
---
## Task 5: Update FUSE Integration
### 5.1 Update `musicfs-fuse/src/filesystem.rs`
```rust
use musicfs_cas::{ContentFetcher, FileReader};
pub struct MusicFs {
tree: Arc<RwLock<VirtualTree>>,
reader: Option<Arc<FileReader>>,
fetcher: Option<Arc<ContentFetcher>>,
uid: u32,
gid: u32,
}
impl MusicFs {
pub fn with_content_access(
tree: Arc<RwLock<VirtualTree>>,
reader: Arc<FileReader>,
fetcher: Arc<ContentFetcher>,
) -> Self {
Self {
tree,
reader: Some(reader),
fetcher: Some(fetcher),
uid: unsafe { libc::getuid() },
gid: unsafe { libc::getgid() },
}
}
}
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_fetch_file` | Unit | Origin → CAS fetch works |
| `test_fetch_file_not_found` | Unit | Missing file error |
| `test_fetch_emits_event` | Unit | FileAccessed event emitted (FR-18.1) |
| `test_reader_with_fetcher` | Unit | Cache-miss triggers fetch |
| `test_e2e_cat_file` | Integration | `cat` returns file content |
---
## Exit Criteria
- [ ] `ContentFetcher` fetches from Origin and stores in CAS
- [ ] `FileReader` calls fetcher on cache miss
- [ ] File metadata (FileId → FileMeta) is resolvable
- [ ] `cat /mnt/musicfs/Artist/Album/track.flac` returns actual audio data
- [ ] All existing tests still pass
---
## Dependencies
### `musicfs-cas/Cargo.toml`
```toml
[dependencies]
musicfs-origins = { path = "../musicfs-origins" }
```
---
## Implementation Notes
1. **Week 4 treated whole files as single chunks** - this continues that approach
2. **CDC chunking deferred to Week 5** - fetcher will be updated then
3. **No OriginFederation yet** - single origin lookup for MVP
4. **FileMeta registration** - caller must register files before they can be fetched
5. **EventBus integration** - emits `FileAccessed` event per FR-18.1 (P0)
6. **Full file fetch** - currently fetches entire file on cache miss; byte-range optimization deferred
## Architecture Compliance
| Architecture Section | Requirement | Status |
|---------------------|-------------|--------|
| 4.3.5 | Cache miss → fetch from origin | ✅ |
| 4.3.5 | Store chunks by hash | ✅ |
| 4.3.5 | Update chunk manifest | ✅ |
| 4.3.5 | Emit FileAccessed event | ✅ |
| 4.3.3 | OriginFederation (multi-origin) | ⏳ Deferred |
| 4.3.5 | Byte-range fetch | ⏳ Deferred |
| 4.3.5 | CDC chunking | ⏳ Week 5 |
---
## Next Steps
After this, the MVP is complete:
- Mount filesystem
- Browse virtual tree (Artist/Album/Track)
- Read actual file content through FUSE
- Audio playback works
Week 5 adds CDC chunking for efficient delta sync.
docs/v2/plans/week-05-cdc-delta.md
-1219
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-06-origin-federation.md
-1113
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-07-remote-origins.md
-1261
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-08-search-index.md
-1266
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-09-smart-features.md
-1686
View File
File diff suppressed because it is too large Load Diff
docs/v2/plans/week-10-plugin-system.md
-179
View File
@@ -1,179 +0,0 @@
# Week 10: Plugin System
**Phase**: 4 - Plugin System & Polish
**Goal**: Extensibility via native and WASM plugins
**Requirements**: FR-23.1-23.5, FR-24.1-24.3
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Plugin traits | musicfs-plugins | `traits.rs` | FR-23.1-23.4 |
| Native host | musicfs-plugins | `native.rs` | FR-23.2 |
| WASM host | musicfs-plugins | `wasm.rs` | FR-23.3 |
| Plugin lifecycle | musicfs-plugins | `manager.rs` | FR-23.5 |
| Example plugins | plugins/ | `example-origin/`, `example-format/` | FR-23.5 |
---
## Plugin Traits (`musicfs-plugins/src/traits.rs`)
```rust
/// Base plugin interface
pub trait Plugin: Send + Sync {
fn name(&self) -> &str;
fn version(&self) -> Version;
fn init(&mut self, config: Value) -> Result<(), PluginError>;
fn shutdown(&mut self) -> Result<(), PluginError>;
}
/// Origin plugin interface (per architecture 4.3.4)
pub trait OriginPlugin: Plugin {
fn origin_type(&self) -> &str;
fn create(&self, config: Value) -> Result<Box<dyn Origin>, PluginError>;
}
/// Metadata source plugin
pub trait MetadataPlugin: Plugin {
fn lookup(&self, query: &MetadataQuery) -> Result<Option<ExternalMetadata>, PluginError>;
}
/// Format plugin for custom audio formats (FR-24.1)
pub trait FormatPlugin: Plugin {
fn extensions(&self) -> &[&str];
fn can_handle(&self, extension: &str) -> bool;
fn parse(&self, reader: &mut dyn Read) -> Result<AudioMeta, PluginError>;
}
```
---
## Native Plugin Host (`musicfs-plugins/src/native.rs`)
```rust
pub struct NativePluginHost {
plugins: HashMap<String, LoadedPlugin>,
search_paths: Vec<PathBuf>,
}
struct LoadedPlugin {
library: libloading::Library,
instance: Box<dyn Plugin>,
}
impl NativePluginHost {
pub fn new() -> Self;
/// Load plugin from shared library (.so/.dylib)
pub fn load(&mut self, path: &Path) -> Result<PluginId, PluginError>;
/// Unload plugin (FR-23.5)
pub fn unload(&mut self, id: PluginId) -> Result<(), PluginError>;
/// Hot reload plugin without restart (FR-23.4)
pub fn reload(&mut self, id: PluginId) -> Result<(), PluginError>;
/// List loaded plugins
pub fn list(&self) -> Vec<PluginInfo>;
}
```
---
## WASM Plugin Host (`musicfs-plugins/src/wasm.rs`)
```rust
pub struct WasmPluginHost {
engine: wasmtime::Engine,
linker: wasmtime::Linker<PluginState>,
}
impl WasmPluginHost {
pub fn new() -> Result<Self, PluginError>;
/// Load WASM plugin with sandboxing (FR-23.3)
pub fn load(&mut self, wasm_bytes: &[u8]) -> Result<WasmPlugin, PluginError>;
/// Resource limits for sandboxed execution
pub fn set_limits(&mut self, limits: ResourceLimits);
}
pub struct ResourceLimits {
pub max_memory_mb: u32,
pub max_cpu_time_ms: u32,
pub allow_network: bool,
pub allow_filesystem: bool,
}
```
---
## Plugin Manager (`musicfs-plugins/src/manager.rs`)
```rust
pub struct PluginManager {
native_host: NativePluginHost,
wasm_host: WasmPluginHost,
registry: PluginRegistry,
}
impl PluginManager {
/// Initialize and load plugins from config
pub fn init(config: &PluginConfig) -> Result<Self, PluginError>;
/// Get all origin plugins
pub fn origin_plugins(&self) -> Vec<&dyn OriginPlugin>;
/// Get all format plugins
pub fn format_plugins(&self) -> Vec<&dyn FormatPlugin>;
/// Get all metadata plugins
pub fn metadata_plugins(&self) -> Vec<&dyn MetadataPlugin>;
/// Reload all plugins (hot reload)
pub fn reload_all(&mut self) -> Result<(), PluginError>;
}
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_native_plugin_load` | Unit | Native plugin loading (FR-23.2) |
| `test_native_plugin_unload` | Unit | Clean unload |
| `test_wasm_plugin_sandbox` | Unit | WASM isolation (FR-23.3) |
| `test_wasm_resource_limits` | Unit | Memory/CPU limits enforced |
| `test_plugin_hot_reload` | Integration | Reload without restart (FR-23.4) |
| `test_example_origin_plugin` | Integration | Custom origin works |
| `test_example_format_plugin` | Integration | Custom format works |
---
## Exit Criteria
- [ ] Native plugins loadable at runtime
- [ ] WASM plugins sandboxed with resource limits
- [ ] Example plugins functional
- [ ] Plugins hot-reloadable without daemon restart
- [ ] Plugin lifecycle management (load, unload, reload)
---
## Architecture Alignment
Per architecture.md section 4.3.4:
- Plugin loading: Built-in → Native (.so) → WASM
- Origin plugins create `Box<dyn Origin>`
- Format plugins register file extensions
- WASM runs in wasmtime sandbox
Per requirements.md:
- FR-23.1: Loadable plugins ✓
- FR-23.2: Stable plugin API ✓
- FR-23.3: Plugins for origins, metadata, formats ✓
- FR-23.4: WASM sandbox ✓
- FR-23.5: Plugin lifecycle ✓
docs/v2/plans/week-11-control-api.md
-539
View File
@@ -1,539 +0,0 @@
# Week 11: Control API & Production
**Phase**: 4 - Plugin System & Polish
**Goal**: gRPC control API, metrics, and production readiness
**Requirements**: FR-17.1-17.5, FR-18.1-18.4, NFR-6.1-6.4, NFR-10.1-10.4
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| gRPC server | musicfs-grpc | `server.rs` | FR-17.1-17.5 |
| Proto codegen | proto/ | `musicfs.proto`, `build.rs` | FR-17.2 |
| Event streaming | musicfs-grpc | `events.rs` | FR-18.1-18.3 |
| Webhook handler | musicfs-grpc | `webhook.rs` | FR-18.2 |
| Metrics export | musicfs-core | `metrics.rs` | NFR-6.1-6.4, NFR-10.2-10.4 |
| CLI completion | musicfs-cli | `main.rs` | FR-17 |
| systemd unit | dist/ | `musicfs.service` | Production |
| Packaging | dist/ | `PKGBUILD`, `musicfs.spec` | Production |
| E2E compatibility | tests/ | `e2e_players.rs` | NFR-12.1-12.3 |
---
## Proto Definitions (`proto/musicfs.proto`)
Per architecture.md section 4.3.7, implement full gRPC API:
```protobuf
syntax = "proto3";
package musicfs.v1;
service MusicFS {
// Daemon lifecycle
rpc GetStatus(Empty) returns (StatusResponse);
rpc Shutdown(ShutdownRequest) returns (Empty);
// Cache management
rpc GetCacheStats(Empty) returns (CacheStats);
rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse);
rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress);
// Origin management
rpc ListOrigins(Empty) returns (OriginsResponse);
rpc GetOriginHealth(OriginRequest) returns (OriginHealth);
rpc RescanOrigin(OriginRequest) returns (stream SyncProgress);
// Search (already implemented in Week 8)
rpc Search(SearchRequest) returns (SearchResponse);
rpc SearchStream(SearchRequest) returns (stream SearchResult);
// Events (server-streaming)
rpc SubscribeEvents(EventFilter) returns (stream Event);
}
```
Full message definitions in architecture.md section 4.3.7.
---
## gRPC Server (`musicfs-grpc/src/server.rs`)
```rust
pub struct MusicFsService {
core: Arc<MusicFsCore>,
events: broadcast::Sender<Event>,
metrics: Arc<MetricsCollector>,
}
#[tonic::async_trait]
impl musicfs::v1::music_fs_server::MusicFs for MusicFsService {
// Daemon lifecycle
async fn get_status(&self, _: Request<Empty>) -> Result<Response<StatusResponse>, Status>;
async fn shutdown(&self, req: Request<ShutdownRequest>) -> Result<Response<Empty>, Status>;
// Cache management
async fn get_cache_stats(&self, _: Request<Empty>) -> Result<Response<CacheStats>, Status>;
async fn clear_cache(&self, req: Request<ClearCacheRequest>) -> Result<Response<ClearCacheResponse>, Status>;
type PrefetchStream = ReceiverStream<Result<PrefetchProgress, Status>>;
async fn prefetch(&self, req: Request<PrefetchRequest>) -> Result<Response<Self::PrefetchStream>, Status>;
// Origin management
async fn list_origins(&self, _: Request<Empty>) -> Result<Response<OriginsResponse>, Status>;
async fn get_origin_health(&self, req: Request<OriginRequest>) -> Result<Response<OriginHealth>, Status>;
type RescanOriginStream = ReceiverStream<Result<SyncProgress, Status>>;
async fn rescan_origin(&self, req: Request<OriginRequest>) -> Result<Response<Self::RescanOriginStream>, Status>;
// Events
type SubscribeEventsStream = ReceiverStream<Result<Event, Status>>;
async fn subscribe_events(&self, req: Request<EventFilter>) -> Result<Response<Self::SubscribeEventsStream>, Status>;
}
```
---
## Event Streaming (`musicfs-grpc/src/events.rs`)
```rust
pub struct EventStreamer {
bus: Arc<EventBus>,
}
impl EventStreamer {
/// Convert internal events to gRPC Event messages
pub fn subscribe(&self, filter: EventFilter) -> impl Stream<Item = Event>;
/// Filter events by type and origin
fn matches(event: &Event, filter: &EventFilter) -> bool;
}
```
---
## Webhook Handler (`musicfs-grpc/src/webhook.rs`)
HTTP webhook notifications for external integrations (FR-18.2):
```rust
use reqwest::Client;
use serde::Serialize;
use tokio::sync::broadcast;
#[derive(Debug, Clone, Serialize)]
pub struct WebhookPayload {
pub event_type: String,
pub timestamp: i64,
pub data: serde_json::Value,
}
pub struct WebhookConfig {
pub url: String,
pub secret: Option<String>,
pub events: Vec<String>, // Filter: ["file_accessed", "sync_completed", ...]
pub retry_count: u32,
pub timeout_ms: u64,
}
pub struct WebhookHandler {
client: Client,
configs: Vec<WebhookConfig>,
}
impl WebhookHandler {
pub fn new(configs: Vec<WebhookConfig>) -> Self;
/// Start listening to event bus and dispatch webhooks
pub async fn run(&self, mut rx: broadcast::Receiver<Event>) {
while let Ok(event) = rx.recv().await {
for config in &self.configs {
if self.matches_filter(&event, config) {
self.dispatch(config, &event).await;
}
}
}
}
/// Dispatch webhook with retry logic
async fn dispatch(&self, config: &WebhookConfig, event: &Event) {
let payload = WebhookPayload {
event_type: event.event_type(),
timestamp: event.timestamp(),
data: event.to_json(),
};
let mut attempts = 0;
loop {
let result = self.client
.post(&config.url)
.timeout(Duration::from_millis(config.timeout_ms))
.header("X-MusicFS-Signature", self.sign(&payload, config))
.json(&payload)
.send()
.await;
match result {
Ok(resp) if resp.status().is_success() => break,
_ if attempts < config.retry_count => {
attempts += 1;
tokio::time::sleep(Duration::from_millis(100 * 2u64.pow(attempts))).await;
}
_ => {
tracing::warn!("Webhook delivery failed after {} attempts", attempts);
break;
}
}
}
}
/// HMAC-SHA256 signature if secret configured
fn sign(&self, payload: &WebhookPayload, config: &WebhookConfig) -> String;
fn matches_filter(&self, event: &Event, config: &WebhookConfig) -> bool;
}
```
Configuration in `config.toml`:
```toml
[[webhooks]]
url = "https://example.com/musicfs/events"
secret = "your-webhook-secret"
events = ["file_accessed", "sync_completed", "origin_health_changed"]
retry_count = 3
timeout_ms = 5000
```
---
## E2E Compatibility Tests (`tests/e2e_players.rs`)
Verify MusicFS works with common media players (NFR-12.1-12.3):
```rust
//! E2E tests for media player compatibility
//! Requires: mpv, vlc, file manager (nautilus/dolphin) installed
use std::process::Command;
use std::time::Duration;
/// Test mpv can play files from MusicFS (NFR-12.1)
#[test]
#[ignore] // Run manually: cargo test --ignored
fn test_mpv_playback() {
let mountpoint = setup_test_mount();
// mpv should be able to:
// 1. Open file without hanging
// 2. Read metadata (duration, format)
// 3. Play first few seconds
// 4. Seek forward
// 5. Exit cleanly
let output = Command::new("mpv")
.args([
"--no-video",
"--no-audio", // Silent playback
"--length=2", // Play 2 seconds only
"--msg-level=all=debug",
&format!("{}/Artist/Album/01 - Track.flac", mountpoint),
])
.output()
.expect("mpv must be installed");
assert!(output.status.success(), "mpv playback failed: {:?}", output);
}
/// Test VLC can browse and play (NFR-12.2)
#[test]
#[ignore]
fn test_vlc_playback() {
let mountpoint = setup_test_mount();
// VLC should handle:
// 1. Directory browsing
// 2. Playlist creation from folder
// 3. Metadata display
// 4. Gapless playback (if supported)
let output = Command::new("cvlc") // Command-line VLC
.args([
"--play-and-exit",
"--run-time=2",
&format!("{}/Artist/Album/", mountpoint),
])
.output()
.expect("vlc must be installed");
assert!(output.status.success(), "VLC playback failed");
}
/// Test file manager operations (NFR-12.3)
#[test]
#[ignore]
fn test_file_manager_operations() {
let mountpoint = setup_test_mount();
// File managers should be able to:
// 1. List directories without timeout
// 2. Show file previews/thumbnails
// 3. Display file properties
// 4. Copy files to local disk
// Test basic stat operations that file managers use
let entries: Vec<_> = std::fs::read_dir(&mountpoint)
.expect("read_dir failed")
.collect();
assert!(!entries.is_empty(), "mountpoint should have entries");
// Test stat on each entry (file managers do this for icons)
for entry in entries {
let entry = entry.expect("entry should be valid");
let metadata = entry.metadata().expect("metadata should work");
assert!(metadata.is_dir() || metadata.is_file());
}
}
/// Test concurrent access from multiple players
#[test]
#[ignore]
fn test_concurrent_player_access() {
let mountpoint = setup_test_mount();
// Spawn multiple players accessing different files
let handles: Vec<_> = (0..3)
.map(|i| {
let mp = mountpoint.clone();
std::thread::spawn(move || {
Command::new("mpv")
.args([
"--no-video", "--no-audio", "--length=1",
&format!("{}/Artist/Album/0{} - Track.flac", mp, i + 1),
])
.output()
})
})
.collect();
for handle in handles {
let output = handle.join().unwrap().expect("mpv should run");
assert!(output.status.success());
}
}
fn setup_test_mount() -> String {
// Returns path to test mount with sample files
std::env::var("MUSICFS_TEST_MOUNT")
.unwrap_or_else(|_| "/tmp/musicfs-test".to_string())
}
```
---
## Metrics (`musicfs-core/src/metrics.rs`)
Per architecture.md section 5.2:
```rust
use prometheus::{IntCounterVec, HistogramVec, IntGauge, register_*};
lazy_static! {
pub static ref FUSE_OPS: IntCounterVec = register_int_counter_vec!(
"musicfs_fuse_ops_total",
"Total FUSE operations",
&["op"]
).unwrap();
pub static ref FUSE_LATENCY: HistogramVec = register_histogram_vec!(
"musicfs_fuse_latency_seconds",
"FUSE operation latency",
&["op"],
vec![0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
).unwrap();
pub static ref CACHE_HITS: IntCounter = register_int_counter!(
"musicfs_cache_hits_total",
"Cache hits"
).unwrap();
pub static ref CACHE_MISSES: IntCounter = register_int_counter!(
"musicfs_cache_misses_total",
"Cache misses"
).unwrap();
pub static ref CACHE_SIZE_BYTES: IntGauge = register_int_gauge!(
"musicfs_cache_size_bytes",
"Current cache size in bytes"
).unwrap();
pub static ref ORIGIN_HEALTH: IntGaugeVec = register_int_gauge_vec!(
"musicfs_origin_health",
"Origin health status (1=healthy, 0=unhealthy)",
&["origin"]
).unwrap();
}
/// Expose metrics on HTTP endpoint
pub async fn serve_metrics(addr: SocketAddr) -> Result<(), MetricsError>;
```
---
## CLI Commands (`musicfs-cli/src/main.rs`)
```rust
#[derive(Parser)]
enum Command {
/// Mount filesystem
Mount {
#[arg(short, long)]
config: PathBuf,
mountpoint: PathBuf,
},
/// Get daemon status
Status,
/// Cache management
Cache {
#[command(subcommand)]
command: CacheCommand,
},
/// Search library
Search {
query: String,
#[arg(short, long, default_value = "100")]
limit: u32,
},
/// Origin management
Origin {
#[command(subcommand)]
command: OriginCommand,
},
/// Subscribe to events
Events {
#[arg(short, long)]
r#type: Option<String>,
},
}
#[derive(Subcommand)]
enum CacheCommand {
Stats,
Clear { origin: Option<String> },
Prefetch { paths: Vec<String> },
}
#[derive(Subcommand)]
enum OriginCommand {
List,
Health { origin_id: String },
Rescan { origin_id: String },
}
```
---
## systemd Service (`dist/musicfs.service`)
```ini
[Unit]
Description=MusicFS - Metadata-Organized Music Filesystem
After=network.target
[Service]
Type=notify
ExecStart=/usr/bin/musicfs mount --config /etc/musicfs/config.toml /mnt/music
ExecStop=/usr/bin/musicfs shutdown
Restart=on-failure
RestartSec=5
User=musicfs
Group=musicfs
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/cache/musicfs /mnt/music
PrivateTmp=true
[Install]
WantedBy=multi-user.target
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_grpc_status` | Unit | GetStatus RPC (FR-17.1) |
| `test_grpc_cache_stats` | Unit | GetCacheStats RPC |
| `test_grpc_cache_clear` | Unit | ClearCache RPC (FR-17.3) |
| `test_grpc_origins_list` | Unit | ListOrigins RPC |
| `test_grpc_origin_rescan` | Integration | RescanOrigin streaming |
| `test_grpc_events_stream` | Integration | Event streaming (FR-18.1) |
| `test_grpc_prefetch_stream` | Integration | Prefetch progress |
| `test_webhook_dispatch` | Unit | Webhook delivery (FR-18.2) |
| `test_webhook_retry` | Unit | Webhook retry on failure |
| `test_webhook_hmac_signature` | Unit | HMAC-SHA256 signing |
| `test_metrics_prometheus` | Unit | Prometheus format (NFR-6.1) |
| `test_metrics_http_endpoint` | Integration | HTTP metrics endpoint |
| `test_cli_commands` | Integration | CLI works |
| `test_systemd_service` | E2E | Service lifecycle |
| `test_mpv_playback` | E2E | mpv compatibility (NFR-12.1) |
| `test_vlc_playback` | E2E | VLC compatibility (NFR-12.2) |
| `test_file_manager_operations` | E2E | File manager browsing (NFR-12.3) |
| `test_concurrent_player_access` | E2E | Multiple players concurrently |
---
## Exit Criteria
- [ ] gRPC API fully functional (all RPCs from architecture.md 4.3.7)
- [ ] Event streaming works with filtering
- [ ] Webhook notifications delivered with HMAC signing
- [ ] Prometheus metrics exported on HTTP endpoint
- [ ] CLI feature-complete with all commands
- [ ] systemd service works (start, stop, restart)
- [ ] mpv, VLC playback verified (E2E tests)
- [ ] File manager browsing verified
- [ ] All acceptance tests pass
---
## Architecture Alignment
Per architecture.md section 4.3.7:
- gRPC over Unix socket ✓
- Protocol Buffers for type safety ✓
- Server-streaming for events, sync progress, prefetch ✓
- CLI wraps gRPC client ✓
Per architecture.md section 5.2:
- Prometheus metrics format ✓
- Golden signals: latency, traffic, errors, saturation ✓
Per requirements.md:
- FR-17.1: Unix socket control ✓
- FR-17.2: gRPC with Protocol Buffers ✓
- FR-17.3: Cache management commands ✓
- FR-17.4: Runtime configuration ✓
- FR-17.5: Graceful shutdown ✓
- FR-18.1: File access events ✓
- FR-18.2: Webhook notifications ✓ (HTTP webhooks with HMAC)
- FR-18.3: Event streaming ✓
- FR-18.4: Access pattern logging ✓
- NFR-10.1: Configurable logging ✓
- NFR-10.2: Metrics exposure ✓
- NFR-10.3: Health check ✓
- NFR-10.4: Prometheus integration ✓
- NFR-12.1: mpv compatibility ✓ (E2E tests)
- NFR-12.2: VLC compatibility ✓ (E2E tests)
- NFR-12.3: File manager compatibility ✓ (E2E tests)
docs/v2/plans/week-12-external-metadata.md
-624
View File
@@ -1,624 +0,0 @@
# Week 12: External Metadata Integration
**Phase**: 5 - P1 Feature Completion
**Goal**: Integrate external metadata sources for automatic tagging and artwork
**Requirements**: FR-21.1-21.4, FR-16.5
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| MusicBrainz client | musicfs-external | `musicbrainz.rs` | FR-21.1 |
| Discogs client | musicfs-external | `discogs.rs` | FR-21.2 |
| Last.fm client | musicfs-external | `lastfm.rs` | FR-21.3 |
| AcoustID/Chromaprint | musicfs-external | `acoustid.rs` | FR-21.4 |
| Online artwork fetch | musicfs-external | `artwork_fetch.rs` | FR-16.5 |
| Metadata enrichment | musicfs-external | `enrichment.rs` | All |
| Plugin integration | musicfs-plugins | `metadata_plugin.rs` | FR-21.5 |
---
## Task 1: Create `musicfs-external` Crate
### 1.1 `Cargo.toml`
```toml
[package]
name = "musicfs-external"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
reqwest = { version = "0.11", features = ["json"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
tokio.workspace = true
tracing.workspace = true
thiserror.workspace = true
chromaprint = "0.6" # Audio fingerprinting
base64 = "0.21"
[dev-dependencies]
wiremock = "0.5" # Mock HTTP responses
tokio-test = "0.4"
```
### 1.2 `src/lib.rs`
```rust
pub mod musicbrainz;
pub mod discogs;
pub mod lastfm;
pub mod acoustid;
pub mod artwork_fetch;
pub mod enrichment;
pub use enrichment::MetadataEnricher;
```
---
## Task 2: MusicBrainz Client (`musicfs-external/src/musicbrainz.rs`)
```rust
use serde::Deserialize;
const MB_API: &str = "https://musicbrainz.org/ws/2";
const USER_AGENT: &str = "MusicFS/0.1.0 (https://github.com/user/musicfs)";
#[derive(Debug, Deserialize)]
pub struct MbRecording {
pub id: String,
pub title: String,
pub length: Option<u64>,
#[serde(rename = "artist-credit")]
pub artist_credit: Vec<ArtistCredit>,
pub releases: Option<Vec<MbRelease>>,
}
#[derive(Debug, Deserialize)]
pub struct MbRelease {
pub id: String,
pub title: String,
pub date: Option<String>,
#[serde(rename = "release-group")]
pub release_group: Option<MbReleaseGroup>,
}
#[derive(Debug, Deserialize)]
pub struct MbReleaseGroup {
pub id: String,
#[serde(rename = "primary-type")]
pub primary_type: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct ArtistCredit {
pub artist: MbArtist,
}
#[derive(Debug, Deserialize)]
pub struct MbArtist {
pub id: String,
pub name: String,
#[serde(rename = "sort-name")]
pub sort_name: String,
}
pub struct MusicBrainzClient {
client: reqwest::Client,
rate_limiter: RateLimiter, // 1 req/sec per MB guidelines
}
impl MusicBrainzClient {
pub fn new() -> Self {
let client = reqwest::Client::builder()
.user_agent(USER_AGENT)
.build()
.expect("client build");
Self {
client,
rate_limiter: RateLimiter::new(Duration::from_secs(1)),
}
}
/// Search by recording title + artist (FR-21.1)
pub async fn search_recording(
&self,
title: &str,
artist: Option<&str>,
) -> Result<Vec<MbRecording>, ExternalError> {
self.rate_limiter.wait().await;
let mut query = format!("recording:{}", title);
if let Some(artist) = artist {
query.push_str(&format!(" AND artist:{}", artist));
}
let resp = self.client
.get(format!("{}/recording", MB_API))
.query(&[
("query", query.as_str()),
("fmt", "json"),
("limit", "5"),
])
.send()
.await?;
let body: SearchResponse<MbRecording> = resp.json().await?;
Ok(body.recordings)
}
/// Get release artwork from Cover Art Archive
pub async fn get_cover_art(&self, release_id: &str) -> Result<Option<Vec<u8>>, ExternalError> {
let url = format!("https://coverartarchive.org/release/{}/front-500", release_id);
let resp = self.client.get(&url).send().await?;
if resp.status() == 404 {
return Ok(None);
}
let bytes = resp.bytes().await?;
Ok(Some(bytes.to_vec()))
}
/// Lookup recording by MusicBrainz ID
pub async fn get_recording(&self, mbid: &str) -> Result<MbRecording, ExternalError> {
self.rate_limiter.wait().await;
let resp = self.client
.get(format!("{}/recording/{}", MB_API, mbid))
.query(&[
("inc", "artist-credits+releases+release-groups"),
("fmt", "json"),
])
.send()
.await?;
Ok(resp.json().await?)
}
}
struct RateLimiter {
interval: Duration,
last_request: Mutex<Instant>,
}
impl RateLimiter {
fn new(interval: Duration) -> Self {
Self {
interval,
last_request: Mutex::new(Instant::now() - interval),
}
}
async fn wait(&self) {
let mut last = self.last_request.lock().await;
let elapsed = last.elapsed();
if elapsed < self.interval {
tokio::time::sleep(self.interval - elapsed).await;
}
*last = Instant::now();
}
}
```
---
## Task 3: Discogs Client (`musicfs-external/src/discogs.rs`)
```rust
const DISCOGS_API: &str = "https://api.discogs.com";
pub struct DiscogsClient {
client: reqwest::Client,
token: Option<String>,
rate_limiter: RateLimiter, // 60 req/min authenticated
}
impl DiscogsClient {
pub fn new(token: Option<String>) -> Self;
/// Search releases (FR-21.2)
pub async fn search(
&self,
query: &str,
artist: Option<&str>,
) -> Result<Vec<DiscogsRelease>, ExternalError>;
/// Get master release details
pub async fn get_master(&self, id: u64) -> Result<DiscogsMaster, ExternalError>;
/// Get release images
pub async fn get_images(&self, release_id: u64) -> Result<Vec<DiscogsImage>, ExternalError>;
}
#[derive(Debug, Deserialize)]
pub struct DiscogsRelease {
pub id: u64,
pub title: String,
pub year: Option<u16>,
pub thumb: Option<String>,
pub master_id: Option<u64>,
}
#[derive(Debug, Deserialize)]
pub struct DiscogsImage {
pub uri: String,
pub width: u32,
pub height: u32,
#[serde(rename = "type")]
pub image_type: String, // "primary" or "secondary"
}
```
---
## Task 4: Last.fm Client (`musicfs-external/src/lastfm.rs`)
```rust
const LASTFM_API: &str = "https://ws.audioscrobbler.com/2.0";
pub struct LastFmClient {
client: reqwest::Client,
api_key: String,
}
impl LastFmClient {
pub fn new(api_key: String) -> Self;
/// Get track info with play counts, tags (FR-21.3)
pub async fn get_track_info(
&self,
track: &str,
artist: &str,
) -> Result<LastFmTrack, ExternalError>;
/// Get album info with artwork
pub async fn get_album_info(
&self,
album: &str,
artist: &str,
) -> Result<LastFmAlbum, ExternalError>;
/// Get artist info
pub async fn get_artist_info(&self, artist: &str) -> Result<LastFmArtist, ExternalError>;
}
#[derive(Debug, Deserialize)]
pub struct LastFmTrack {
pub name: String,
pub playcount: Option<u64>,
pub listeners: Option<u64>,
pub duration: Option<u64>,
pub toptags: Option<Tags>,
pub album: Option<LastFmAlbumRef>,
}
#[derive(Debug, Deserialize)]
pub struct LastFmAlbum {
pub name: String,
pub artist: String,
pub image: Vec<LastFmImage>,
pub tracks: Option<Tracks>,
}
#[derive(Debug, Deserialize)]
pub struct LastFmImage {
#[serde(rename = "#text")]
pub url: String,
pub size: String, // "small", "medium", "large", "extralarge", "mega"
}
```
---
## Task 5: AcoustID/Chromaprint (`musicfs-external/src/acoustid.rs`)
```rust
use chromaprint::{Fingerprinter, Configuration};
const ACOUSTID_API: &str = "https://api.acoustid.org/v2/lookup";
pub struct AcoustIdClient {
client: reqwest::Client,
api_key: String,
}
impl AcoustIdClient {
pub fn new(api_key: String) -> Self;
/// Generate fingerprint from audio data (FR-21.4)
pub fn fingerprint(&self, samples: &[i16], sample_rate: u32) -> Result<String, ExternalError> {
let config = Configuration::preset_test1();
let mut fp = Fingerprinter::new(&config);
fp.start(sample_rate, 1)?; // mono
fp.feed(samples)?;
fp.finish()?;
Ok(fp.fingerprint().to_string())
}
/// Lookup fingerprint on AcoustID database
pub async fn lookup(
&self,
fingerprint: &str,
duration: u32,
) -> Result<Vec<AcoustIdResult>, ExternalError> {
let resp = self.client
.get(ACOUSTID_API)
.query(&[
("client", self.api_key.as_str()),
("fingerprint", fingerprint),
("duration", &duration.to_string()),
("meta", "recordings+releasegroups"),
])
.send()
.await?;
let body: AcoustIdResponse = resp.json().await?;
Ok(body.results)
}
}
#[derive(Debug, Deserialize)]
pub struct AcoustIdResult {
pub id: String,
pub score: f32,
pub recordings: Option<Vec<AcoustIdRecording>>,
}
#[derive(Debug, Deserialize)]
pub struct AcoustIdRecording {
pub id: String, // MusicBrainz recording ID
pub title: Option<String>,
pub artists: Option<Vec<AcoustIdArtist>>,
}
```
---
## Task 6: Online Artwork Fetch (`musicfs-external/src/artwork_fetch.rs`)
```rust
pub struct ArtworkFetcher {
musicbrainz: MusicBrainzClient,
discogs: Option<DiscogsClient>,
lastfm: Option<LastFmClient>,
}
impl ArtworkFetcher {
/// Fetch missing artwork from online sources (FR-16.5)
/// Tries sources in order: MusicBrainz Cover Art Archive → Discogs → Last.fm
pub async fn fetch_artwork(
&self,
artist: &str,
album: &str,
size: ArtworkSize,
) -> Result<Option<ArtworkData>, ExternalError> {
// 1. Try MusicBrainz release search → Cover Art Archive
if let Some(art) = self.try_musicbrainz(artist, album, size).await? {
return Ok(Some(art));
}
// 2. Try Discogs
if let Some(discogs) = &self.discogs {
if let Some(art) = self.try_discogs(discogs, artist, album, size).await? {
return Ok(Some(art));
}
}
// 3. Try Last.fm
if let Some(lastfm) = &self.lastfm {
if let Some(art) = self.try_lastfm(lastfm, artist, album, size).await? {
return Ok(Some(art));
}
}
Ok(None)
}
async fn try_musicbrainz(
&self,
artist: &str,
album: &str,
size: ArtworkSize,
) -> Result<Option<ArtworkData>, ExternalError> {
// Search for release, get cover art from Cover Art Archive
let releases = self.musicbrainz.search_release(album, Some(artist)).await?;
for release in releases.iter().take(3) {
if let Some(art) = self.musicbrainz.get_cover_art(&release.id).await? {
return Ok(Some(ArtworkData {
data: art,
source: ArtworkSource::MusicBrainz,
mime_type: "image/jpeg".to_string(),
}));
}
}
Ok(None)
}
}
#[derive(Debug)]
pub struct ArtworkData {
pub data: Vec<u8>,
pub source: ArtworkSource,
pub mime_type: String,
}
#[derive(Debug)]
pub enum ArtworkSource {
MusicBrainz,
Discogs,
LastFm,
Embedded,
}
pub enum ArtworkSize {
Small, // 150px
Medium, // 300px
Large, // 500px
Original,
}
```
---
## Task 7: Metadata Enrichment (`musicfs-external/src/enrichment.rs`)
```rust
pub struct MetadataEnricher {
musicbrainz: MusicBrainzClient,
acoustid: Option<AcoustIdClient>,
artwork_fetcher: ArtworkFetcher,
}
impl MetadataEnricher {
/// Enrich metadata from external sources
pub async fn enrich(&self, meta: &AudioMeta) -> Result<EnrichedMetadata, ExternalError> {
let mut enriched = EnrichedMetadata::from(meta);
// If we have title + artist, search MusicBrainz
if let (Some(title), Some(artist)) = (&meta.title, &meta.artist) {
let recordings = self.musicbrainz.search_recording(title, Some(artist)).await?;
if let Some(best) = recordings.first() {
enriched.musicbrainz_recording_id = Some(best.id.clone());
// Enrich with release info
if let Some(releases) = &best.releases {
if let Some(release) = releases.first() {
enriched.musicbrainz_release_id = Some(release.id.clone());
}
}
}
}
Ok(enriched)
}
/// Identify unknown track by audio fingerprint
pub async fn identify_by_fingerprint(
&self,
samples: &[i16],
sample_rate: u32,
duration: u32,
) -> Result<Option<IdentifiedTrack>, ExternalError> {
let acoustid = self.acoustid.as_ref()
.ok_or(ExternalError::ServiceNotConfigured("AcoustID"))?;
let fingerprint = acoustid.fingerprint(samples, sample_rate)?;
let results = acoustid.lookup(&fingerprint, duration).await?;
// Return best match above threshold
results.into_iter()
.filter(|r| r.score > 0.8)
.flat_map(|r| r.recordings)
.flatten()
.next()
.map(|rec| IdentifiedTrack {
title: rec.title,
musicbrainz_id: Some(rec.id),
artists: rec.artists.map(|a| a.into_iter().map(|x| x.name).collect()),
})
.pipe(Ok)
}
}
#[derive(Debug)]
pub struct EnrichedMetadata {
pub original: AudioMeta,
pub musicbrainz_recording_id: Option<String>,
pub musicbrainz_release_id: Option<String>,
pub musicbrainz_artist_id: Option<String>,
pub genres: Vec<String>,
pub play_count: Option<u64>,
}
#[derive(Debug)]
pub struct IdentifiedTrack {
pub title: Option<String>,
pub musicbrainz_id: Option<String>,
pub artists: Option<Vec<String>>,
}
```
---
## Configuration
```toml
[external]
# MusicBrainz (no auth required, rate limited to 1 req/sec)
musicbrainz.enabled = true
# Discogs (optional, requires token for higher rate limits)
discogs.enabled = true
discogs.token = "your_discogs_token"
# Last.fm (requires API key)
lastfm.enabled = true
lastfm.api_key = "your_lastfm_api_key"
# AcoustID (requires API key)
acoustid.enabled = true
acoustid.api_key = "your_acoustid_api_key"
# Artwork fetching behavior
artwork.fetch_missing = true
artwork.cache_fetched = true
artwork.preferred_size = "large" # small, medium, large, original
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_musicbrainz_search` | Integration | Recording search (FR-21.1) |
| `test_musicbrainz_cover_art` | Integration | Cover Art Archive |
| `test_discogs_search` | Integration | Release search (FR-21.2) |
| `test_lastfm_track_info` | Integration | Track metadata (FR-21.3) |
| `test_acoustid_fingerprint` | Unit | Chromaprint generation |
| `test_acoustid_lookup` | Integration | Fingerprint lookup (FR-21.4) |
| `test_artwork_fetch_cascade` | Integration | Multi-source artwork (FR-16.5) |
| `test_metadata_enrichment` | Integration | Full enrichment flow |
| `test_rate_limiting` | Unit | Rate limiter works |
| `test_mock_responses` | Unit | Offline testing with mocks |
---
## Exit Criteria
- [ ] MusicBrainz search returns relevant recordings
- [ ] Cover Art Archive artwork downloads work
- [ ] Discogs integration retrieves release info
- [ ] Last.fm integration retrieves track/artist info
- [ ] AcoustID fingerprinting identifies tracks
- [ ] Artwork fetcher tries all sources in cascade
- [ ] Metadata enricher adds external IDs
- [ ] Rate limiting prevents API abuse
- [ ] All tests pass with mock HTTP responses
---
## Architecture Alignment
Per requirements.md:
- FR-21.1: MusicBrainz for canonical metadata ✓
- FR-21.2: Discogs for release info, artwork ✓
- FR-21.3: Last.fm for play counts, tags ✓
- FR-21.4: AcoustID for audio fingerprinting ✓
- FR-16.5: Fetch missing artwork from online ✓
Per architecture.md section 4.3.4:
- External metadata via `MetadataPlugin` trait ✓
- Plugin architecture allows adding more sources ✓
docs/v2/plans/week-13-import-export.md
-699
View File
@@ -1,699 +0,0 @@
# Week 13: Import & Export
**Phase**: 5 - P1 Feature Completion
**Goal**: Import metadata from existing library managers, export library data
**Requirements**: FR-22.1-22.3
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Beets database import | musicfs-import | `beets.rs` | FR-22.1 |
| iTunes/Apple Music import | musicfs-import | `itunes.rs` | FR-22.2 |
| Library export | musicfs-import | `export.rs` | FR-22.3 |
| Import CLI | musicfs-cli | `import.rs` | All |
---
## Task 1: Create `musicfs-import` Crate
### 1.1 `Cargo.toml`
```toml
[package]
name = "musicfs-import"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
musicfs-cache = { path = "../musicfs-cache" }
rusqlite = { workspace = true, features = ["bundled"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
plist = "1.5" # For iTunes XML parsing
tokio.workspace = true
tracing.workspace = true
thiserror.workspace = true
csv = "1.3"
url = "2.4"
percent-encoding = "2.3"
chrono = { version = "0.4", features = ["serde"] }
[dev-dependencies]
tempfile.workspace = true
```
### 1.2 `src/lib.rs`
```rust
pub mod beets;
pub mod itunes;
pub mod export;
use musicfs_core::Result;
/// Common import result
#[derive(Debug, Default)]
pub struct ImportResult {
pub imported: usize,
pub skipped: usize,
pub errors: Vec<ImportError>,
}
#[derive(Debug)]
pub struct ImportError {
pub path: String,
pub reason: String,
}
/// Import progress callback
pub type ProgressCallback = Box<dyn Fn(ImportProgress) + Send>;
#[derive(Debug, Clone)]
pub struct ImportProgress {
pub current: usize,
pub total: usize,
pub current_file: String,
}
```
---
## Task 2: Beets Database Import (`musicfs-import/src/beets.rs`)
```rust
use rusqlite::{Connection, params};
use std::path::Path;
/// Beets database schema (simplified)
/// Full schema: https://beets.readthedocs.io/en/stable/dev/db.html
#[derive(Debug)]
pub struct BeetsItem {
pub id: i64,
pub path: String,
pub title: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>,
pub year: Option<i32>,
pub track: Option<i32>,
pub disc: Option<i32>,
pub length: Option<f64>,
pub bitrate: Option<i32>,
pub sample_rate: Option<i32>,
pub format: Option<String>,
pub mb_trackid: Option<String>,
pub mb_albumid: Option<String>,
pub mb_artistid: Option<String>,
pub mtime: f64,
}
pub struct BeetsImporter {
beets_db: Connection,
target_db: Arc<Database>,
}
impl BeetsImporter {
/// Open beets database for import (FR-22.1)
pub fn new(beets_db_path: &Path, target_db: Arc<Database>) -> Result<Self, ImportError> {
let conn = Connection::open_with_flags(
beets_db_path,
rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY,
)?;
// Verify this is a beets database
let tables: Vec<String> = conn
.prepare("SELECT name FROM sqlite_master WHERE type='table'")?
.query_map([], |row| row.get(0))?
.filter_map(|r| r.ok())
.collect();
if !tables.contains(&"items".to_string()) {
return Err(ImportError::InvalidDatabase("Not a beets database"));
}
Ok(Self {
beets_db: conn,
target_db,
})
}
/// Count items to import
pub fn count_items(&self) -> Result<usize, ImportError> {
self.beets_db
.query_row("SELECT COUNT(*) FROM items", [], |row| row.get(0))
.map_err(Into::into)
}
/// Import all items with progress callback
pub fn import_all(&self, progress: Option<ProgressCallback>) -> Result<ImportResult, ImportError> {
let total = self.count_items()?;
let mut result = ImportResult::default();
let mut stmt = self.beets_db.prepare(r#"
SELECT id, path, title, artist, album, albumartist, genre,
year, track, disc, length, bitrate, samplerate, format,
mb_trackid, mb_albumid, mb_artistid, mtime
FROM items
"#)?;
let items = stmt.query_map([], |row| {
Ok(BeetsItem {
id: row.get(0)?,
path: row.get(1)?,
title: row.get(2)?,
artist: row.get(3)?,
album: row.get(4)?,
album_artist: row.get(5)?,
genre: row.get(6)?,
year: row.get(7)?,
track: row.get(8)?,
disc: row.get(9)?,
length: row.get(10)?,
bitrate: row.get(11)?,
sample_rate: row.get(12)?,
format: row.get(13)?,
mb_trackid: row.get(14)?,
mb_albumid: row.get(15)?,
mb_artistid: row.get(16)?,
mtime: row.get(17)?,
})
})?;
for (idx, item) in items.enumerate() {
match item {
Ok(item) => {
if let Some(ref cb) = progress {
cb(ImportProgress {
current: idx + 1,
total,
current_file: item.path.clone(),
});
}
match self.import_item(&item) {
Ok(_) => result.imported += 1,
Err(e) => {
result.errors.push(ImportError {
path: item.path,
reason: e.to_string(),
});
}
}
}
Err(e) => {
result.skipped += 1;
result.errors.push(ImportError {
path: format!("item_{}", idx),
reason: e.to_string(),
});
}
}
}
Ok(result)
}
fn import_item(&self, item: &BeetsItem) -> Result<(), ImportError> {
let path = Path::new(&item.path);
// Convert to our AudioMeta
let audio_meta = AudioMeta {
title: item.title.clone(),
artist: item.artist.clone(),
album: item.album.clone(),
album_artist: item.album_artist.clone(),
genre: item.genre.clone(),
year: item.year.map(|y| y as u32),
track: item.track.map(|t| t as u32),
disc: item.disc.map(|d| d as u32),
duration_ms: item.length.map(|l| (l * 1000.0) as u64),
bitrate: item.bitrate.map(|b| b as u32),
sample_rate: item.sample_rate.map(|s| s as u32),
format: AudioFormat::from_extension(
path.extension().and_then(|e| e.to_str()).unwrap_or("")
),
..Default::default()
};
// Generate virtual path using our resolver
let virtual_path = VirtualPath::from_metadata(&audio_meta, path);
// Import to our database
self.target_db.upsert_file(
&OriginId::from("beets-import"),
path,
&virtual_path,
&audio_meta,
std::time::UNIX_EPOCH + std::time::Duration::from_secs_f64(item.mtime),
std::fs::metadata(path).map(|m| m.len()).unwrap_or(0),
)?;
Ok(())
}
}
```
---
## Task 3: iTunes/Apple Music Import (`musicfs-import/src/itunes.rs`)
```rust
use plist::Value;
use std::collections::HashMap;
use url::Url;
/// iTunes Library XML format
#[derive(Debug)]
pub struct ItunesTrack {
pub track_id: u64,
pub name: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>,
pub year: Option<u32>,
pub track_number: Option<u32>,
pub disc_number: Option<u32>,
pub total_time: Option<u64>, // milliseconds
pub bit_rate: Option<u32>,
pub sample_rate: Option<u32>,
pub location: Option<String>, // file:// URL
pub date_added: Option<String>,
}
pub struct ItunesImporter {
tracks: Vec<ItunesTrack>,
target_db: Arc<Database>,
}
impl ItunesImporter {
/// Parse iTunes Library.xml (FR-22.2)
pub fn from_xml(xml_path: &Path, target_db: Arc<Database>) -> Result<Self, ImportError> {
let file = std::fs::File::open(xml_path)?;
let plist: Value = plist::from_reader(file)?;
let dict = plist.as_dictionary()
.ok_or(ImportError::InvalidFormat("Expected dictionary at root"))?;
let tracks_dict = dict.get("Tracks")
.and_then(|v| v.as_dictionary())
.ok_or(ImportError::InvalidFormat("Missing Tracks dictionary"))?;
let mut tracks = Vec::new();
for (_, track_value) in tracks_dict {
if let Some(track_dict) = track_value.as_dictionary() {
tracks.push(Self::parse_track(track_dict)?);
}
}
Ok(Self { tracks, target_db })
}
fn parse_track(dict: &plist::Dictionary) -> Result<ItunesTrack, ImportError> {
Ok(ItunesTrack {
track_id: dict.get("Track ID")
.and_then(|v| v.as_unsigned_integer())
.unwrap_or(0),
name: dict.get("Name").and_then(|v| v.as_string()).map(String::from),
artist: dict.get("Artist").and_then(|v| v.as_string()).map(String::from),
album: dict.get("Album").and_then(|v| v.as_string()).map(String::from),
album_artist: dict.get("Album Artist").and_then(|v| v.as_string()).map(String::from),
genre: dict.get("Genre").and_then(|v| v.as_string()).map(String::from),
year: dict.get("Year").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
track_number: dict.get("Track Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
disc_number: dict.get("Disc Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
total_time: dict.get("Total Time").and_then(|v| v.as_unsigned_integer()),
bit_rate: dict.get("Bit Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
sample_rate: dict.get("Sample Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
location: dict.get("Location").and_then(|v| v.as_string()).map(String::from),
date_added: dict.get("Date Added").and_then(|v| v.as_string()).map(String::from),
})
}
/// Convert file:// URL to path
fn url_to_path(url_str: &str) -> Option<PathBuf> {
Url::parse(url_str).ok()
.filter(|u| u.scheme() == "file")
.and_then(|u| u.to_file_path().ok())
}
pub fn count_tracks(&self) -> usize {
self.tracks.len()
}
/// Import all tracks
pub fn import_all(&self, progress: Option<ProgressCallback>) -> Result<ImportResult, ImportError> {
let total = self.tracks.len();
let mut result = ImportResult::default();
for (idx, track) in self.tracks.iter().enumerate() {
if let Some(ref cb) = progress {
cb(ImportProgress {
current: idx + 1,
total,
current_file: track.name.clone().unwrap_or_default(),
});
}
// Skip tracks without location
let Some(ref location) = track.location else {
result.skipped += 1;
continue;
};
let Some(path) = Self::url_to_path(location) else {
result.skipped += 1;
result.errors.push(ImportError {
path: location.clone(),
reason: "Invalid file URL".to_string(),
});
continue;
};
match self.import_track(track, &path) {
Ok(_) => result.imported += 1,
Err(e) => {
result.errors.push(ImportError {
path: path.display().to_string(),
reason: e.to_string(),
});
}
}
}
Ok(result)
}
fn import_track(&self, track: &ItunesTrack, path: &Path) -> Result<(), ImportError> {
let audio_meta = AudioMeta {
title: track.name.clone(),
artist: track.artist.clone(),
album: track.album.clone(),
album_artist: track.album_artist.clone(),
genre: track.genre.clone(),
year: track.year,
track: track.track_number,
disc: track.disc_number,
duration_ms: track.total_time,
bitrate: track.bit_rate,
sample_rate: track.sample_rate,
format: AudioFormat::from_extension(
path.extension().and_then(|e| e.to_str()).unwrap_or("")
),
..Default::default()
};
let virtual_path = VirtualPath::from_metadata(&audio_meta, path);
let mtime = std::fs::metadata(path)
.map(|m| m.modified().unwrap_or(std::time::UNIX_EPOCH))
.unwrap_or(std::time::UNIX_EPOCH);
let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
self.target_db.upsert_file(
&OriginId::from("itunes-import"),
path,
&virtual_path,
&audio_meta,
mtime,
size,
)?;
Ok(())
}
}
```
---
## Task 4: Library Export (`musicfs-import/src/export.rs`)
```rust
use csv::Writer;
use serde::Serialize;
#[derive(Debug, Serialize)]
pub struct ExportedTrack {
pub virtual_path: String,
pub real_path: String,
pub title: String,
pub artist: String,
pub album: String,
pub album_artist: String,
pub genre: String,
pub year: Option<u32>,
pub track: Option<u32>,
pub disc: Option<u32>,
pub duration_ms: Option<u64>,
pub format: String,
pub musicbrainz_id: Option<String>,
}
pub struct LibraryExporter {
db: Arc<Database>,
}
impl LibraryExporter {
pub fn new(db: Arc<Database>) -> Self {
Self { db }
}
/// Export library to CSV (FR-22.3)
pub fn export_csv(&self, output: &Path) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let mut writer = Writer::from_path(output)?;
let mut count = 0;
for file in files {
let audio = file.audio.as_ref();
writer.serialize(ExportedTrack {
virtual_path: file.virtual_path.as_str().to_string(),
real_path: file.real_path.path.display().to_string(),
title: audio.and_then(|a| a.title.clone()).unwrap_or_default(),
artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(),
album: audio.and_then(|a| a.album.clone()).unwrap_or_default(),
album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(),
genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(),
year: audio.and_then(|a| a.year),
track: audio.and_then(|a| a.track),
disc: audio.and_then(|a| a.disc),
duration_ms: audio.and_then(|a| a.duration_ms),
format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(),
musicbrainz_id: None, // TODO: Include if enriched
})?;
count += 1;
}
writer.flush()?;
Ok(count)
}
/// Export library to JSON
pub fn export_json(&self, output: &Path) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let tracks: Vec<ExportedTrack> = files.iter()
.map(|file| {
let audio = file.audio.as_ref();
ExportedTrack {
virtual_path: file.virtual_path.as_str().to_string(),
real_path: file.real_path.path.display().to_string(),
title: audio.and_then(|a| a.title.clone()).unwrap_or_default(),
artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(),
album: audio.and_then(|a| a.album.clone()).unwrap_or_default(),
album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(),
genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(),
year: audio.and_then(|a| a.year),
track: audio.and_then(|a| a.track),
disc: audio.and_then(|a| a.disc),
duration_ms: audio.and_then(|a| a.duration_ms),
format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(),
musicbrainz_id: None,
}
})
.collect();
let json = serde_json::to_string_pretty(&tracks)?;
std::fs::write(output, json)?;
Ok(tracks.len())
}
/// Export to M3U playlist format
pub fn export_m3u(&self, output: &Path, base_path: Option<&Path>) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let mut content = String::from("#EXTM3U\n");
for file in &files {
let duration = file.audio.as_ref()
.and_then(|a| a.duration_ms)
.map(|d| d / 1000)
.unwrap_or(0);
let title = file.audio.as_ref()
.and_then(|a| a.title.clone())
.unwrap_or_else(|| file.virtual_path.as_str().to_string());
let artist = file.audio.as_ref()
.and_then(|a| a.artist.clone())
.unwrap_or_default();
content.push_str(&format!(
"#EXTINF:{},{} - {}\n",
duration, artist, title
));
// Use virtual path relative to base, or absolute real path
let path = if let Some(base) = base_path {
base.join(file.virtual_path.as_str().trim_start_matches('/'))
.display().to_string()
} else {
file.real_path.path.display().to_string()
};
content.push_str(&path);
content.push('\n');
}
std::fs::write(output, content)?;
Ok(files.len())
}
}
```
---
## Task 5: Import CLI Commands (`musicfs-cli/src/import.rs`)
```rust
#[derive(Subcommand)]
pub enum ImportCommand {
/// Import from beets database
Beets {
/// Path to beets library.db
#[arg(short, long)]
db: PathBuf,
},
/// Import from iTunes Library.xml
Itunes {
/// Path to iTunes Library.xml
#[arg(short, long)]
xml: PathBuf,
},
/// Export library
Export {
/// Output file path
#[arg(short, long)]
output: PathBuf,
/// Format: csv, json, m3u
#[arg(short, long, default_value = "csv")]
format: String,
},
}
pub async fn handle_import(cmd: ImportCommand, db: Arc<Database>) -> Result<()> {
match cmd {
ImportCommand::Beets { db: beets_path } => {
println!("Importing from beets database: {:?}", beets_path);
let importer = BeetsImporter::new(&beets_path, db)?;
let total = importer.count_items()?;
println!("Found {} items to import", total);
let pb = ProgressBar::new(total as u64);
let result = importer.import_all(Some(Box::new(move |p| {
pb.set_position(p.current as u64);
})))?;
println!("\nImport complete:");
println!(" Imported: {}", result.imported);
println!(" Skipped: {}", result.skipped);
println!(" Errors: {}", result.errors.len());
}
ImportCommand::Itunes { xml } => {
println!("Importing from iTunes Library: {:?}", xml);
let importer = ItunesImporter::from_xml(&xml, db)?;
let total = importer.count_tracks();
println!("Found {} tracks to import", total);
let pb = ProgressBar::new(total as u64);
let result = importer.import_all(Some(Box::new(move |p| {
pb.set_position(p.current as u64);
})))?;
println!("\nImport complete:");
println!(" Imported: {}", result.imported);
println!(" Skipped: {}", result.skipped);
println!(" Errors: {}", result.errors.len());
}
ImportCommand::Export { output, format } => {
let exporter = LibraryExporter::new(db);
let count = match format.as_str() {
"csv" => exporter.export_csv(&output)?,
"json" => exporter.export_json(&output)?,
"m3u" => exporter.export_m3u(&output, None)?,
_ => return Err(anyhow::anyhow!("Unknown format: {}", format)),
};
println!("Exported {} tracks to {:?}", count, output);
}
}
Ok(())
}
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_beets_import_valid` | Integration | Beets database parsing (FR-22.1) |
| `test_beets_import_missing_fields` | Unit | Handle incomplete metadata |
| `test_itunes_xml_parsing` | Unit | iTunes XML parsing (FR-22.2) |
| `test_itunes_url_to_path` | Unit | file:// URL conversion |
| `test_itunes_import_tracks` | Integration | Full iTunes import |
| `test_export_csv` | Unit | CSV export (FR-22.3) |
| `test_export_json` | Unit | JSON export |
| `test_export_m3u` | Unit | M3U playlist export |
| `test_import_preserves_musicbrainz_ids` | Integration | External IDs preserved |
| `test_import_deduplication` | Integration | No duplicates on re-import |
---
## Exit Criteria
- [ ] Beets database import works with real beets.db
- [ ] iTunes Library.xml import parses all tracks
- [ ] CSV/JSON/M3U export generates valid files
- [ ] Progress reporting works during import
- [ ] Errors are reported without crashing
- [ ] Import is idempotent (re-import updates, doesn't duplicate)
- [ ] MusicBrainz IDs from beets are preserved
---
## Architecture Alignment
Per requirements.md:
- FR-22.1: Import from beets database ✓
- FR-22.2: Import from iTunes/Apple Music ✓
- FR-22.3: Export library metadata ✓
docs/v2/plans/week-14-extended-formats.md
-633
View File
@@ -1,633 +0,0 @@
# Week 14: Extended Formats & Audio Fingerprinting
**Phase**: 5 - P1 Feature Completion
**Goal**: Audio fingerprint search and audiobook format support
**Requirements**: FR-14.4, FR-24.2
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Fingerprint indexing | musicfs-search | `fingerprint.rs` | FR-14.4 |
| Fingerprint search | musicfs-search | `fingerprint_search.rs` | FR-14.4 |
| M4B audiobook support | musicfs-metadata | `formats/m4b.rs` | FR-24.2 |
| Chapter extraction | musicfs-metadata | `chapters.rs` | FR-24.2 |
| Virtual chapter files | musicfs-fuse | `ops/chapters.rs` | FR-24.2 |
---
## Task 1: Audio Fingerprint Generation
### 1.1 Add Dependencies
```toml
# In musicfs-search/Cargo.toml
[dependencies]
chromaprint = "0.6"
symphonia = { version = "0.5", features = ["all"] }
```
### 1.2 Fingerprint Generation (`musicfs-search/src/fingerprint.rs`)
```rust
use chromaprint::{Configuration, Fingerprinter};
use symphonia::core::audio::SampleBuffer;
use symphonia::core::codecs::DecoderOptions;
use std::path::Path;
/// Audio fingerprint using Chromaprint algorithm
#[derive(Debug, Clone)]
pub struct AudioFingerprint {
pub raw: Vec<u32>,
pub duration_secs: u32,
}
impl AudioFingerprint {
/// Generate fingerprint from audio file (FR-14.4)
pub fn from_file(path: &Path) -> Result<Self, FingerprintError> {
let file = std::fs::File::open(path)?;
let mss = MediaSourceStream::new(Box::new(file), Default::default());
let probed = symphonia::default::get_probe()
.format(&Hint::new(), mss, &FormatOptions::default(), &MetadataOptions::default())?;
let mut format = probed.format;
let track = format.tracks()
.iter()
.find(|t| t.codec_params.codec != CODEC_TYPE_NULL)
.ok_or(FingerprintError::NoAudioTrack)?;
let sample_rate = track.codec_params.sample_rate
.ok_or(FingerprintError::NoSampleRate)?;
let mut decoder = symphonia::default::get_codecs()
.make(&track.codec_params, &DecoderOptions::default())?;
// Chromaprint configuration
let config = Configuration::preset_test1();
let mut fingerprinter = Fingerprinter::new(&config);
fingerprinter.start(sample_rate, 1)?; // Mono
let mut samples: Vec<i16> = Vec::new();
let mut duration_samples = 0u64;
// Decode and collect samples (first 120 seconds max)
let max_samples = sample_rate as u64 * 120;
loop {
match format.next_packet() {
Ok(packet) => {
let decoded = decoder.decode(&packet)?;
let mut sample_buf = SampleBuffer::<i16>::new(
decoded.capacity() as u64,
*decoded.spec(),
);
sample_buf.copy_interleaved_ref(decoded);
// Convert to mono if stereo
let mono: Vec<i16> = if decoded.spec().channels.count() > 1 {
sample_buf.samples()
.chunks(decoded.spec().channels.count())
.map(|chunk| (chunk.iter().map(|&s| s as i32).sum::<i32>() / chunk.len() as i32) as i16)
.collect()
} else {
sample_buf.samples().to_vec()
};
samples.extend(&mono);
duration_samples += mono.len() as u64;
if duration_samples >= max_samples {
break;
}
}
Err(symphonia::core::errors::Error::IoError(e))
if e.kind() == std::io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e.into()),
}
}
// Feed samples to fingerprinter
fingerprinter.feed(&samples)?;
fingerprinter.finish()?;
let raw = fingerprinter.fingerprint().to_vec();
let duration_secs = (duration_samples / sample_rate as u64) as u32;
Ok(Self { raw, duration_secs })
}
/// Compress fingerprint for storage
pub fn to_bytes(&self) -> Vec<u8> {
// Use chromaprint's compressed format
chromaprint::encode_fingerprint(&self.raw, chromaprint::Algorithm::Test1)
}
/// Decompress fingerprint
pub fn from_bytes(bytes: &[u8]) -> Result<Self, FingerprintError> {
let (raw, _) = chromaprint::decode_fingerprint(bytes)?;
Ok(Self { raw, duration_secs: 0 })
}
}
#[derive(Debug, thiserror::Error)]
pub enum FingerprintError {
#[error("No audio track found")]
NoAudioTrack,
#[error("No sample rate")]
NoSampleRate,
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Decode error: {0}")]
Decode(String),
#[error("Chromaprint error: {0}")]
Chromaprint(String),
}
```
---
## Task 2: Fingerprint Search (`musicfs-search/src/fingerprint_search.rs`)
```rust
use crate::fingerprint::AudioFingerprint;
/// Fingerprint similarity search using bit-level comparison
pub struct FingerprintIndex {
db: Arc<Database>,
}
impl FingerprintIndex {
pub fn new(db: Arc<Database>) -> Self {
Self { db }
}
/// Index a file's fingerprint
pub fn index(&self, file_id: FileId, fingerprint: &AudioFingerprint) -> Result<(), SearchError> {
let bytes = fingerprint.to_bytes();
self.db.store_fingerprint(file_id, &bytes, fingerprint.duration_secs)?;
Ok(())
}
/// Search by fingerprint similarity (FR-14.4)
pub fn search(
&self,
query: &AudioFingerprint,
threshold: f32, // 0.0-1.0, higher = more similar
limit: usize,
) -> Result<Vec<FingerprintMatch>, SearchError> {
let candidates = self.db.get_fingerprints_by_duration(
query.duration_secs.saturating_sub(10),
query.duration_secs + 10,
)?;
let mut matches: Vec<FingerprintMatch> = candidates
.into_iter()
.filter_map(|(file_id, fp_bytes, duration)| {
let fp = AudioFingerprint::from_bytes(&fp_bytes).ok()?;
let similarity = self.compare(&query.raw, &fp.raw);
if similarity >= threshold {
Some(FingerprintMatch { file_id, similarity, duration })
} else {
None
}
})
.collect();
// Sort by similarity descending
matches.sort_by(|a, b| b.similarity.partial_cmp(&a.similarity).unwrap());
matches.truncate(limit);
Ok(matches)
}
/// Compare two fingerprints using bit error rate
fn compare(&self, a: &[u32], b: &[u32]) -> f32 {
let len = a.len().min(b.len());
if len == 0 {
return 0.0;
}
let mut matching_bits = 0u32;
let mut total_bits = 0u32;
for i in 0..len {
let xor = a[i] ^ b[i];
matching_bits += 32 - xor.count_ones();
total_bits += 32;
}
matching_bits as f32 / total_bits as f32
}
/// Find duplicates by fingerprint
pub fn find_duplicates(&self, threshold: f32) -> Result<Vec<DuplicateGroup>, SearchError> {
let all_fps = self.db.get_all_fingerprints()?;
let mut groups: Vec<DuplicateGroup> = Vec::new();
let mut processed: HashSet<FileId> = HashSet::new();
for (file_id, fp_bytes, duration) in &all_fps {
if processed.contains(file_id) {
continue;
}
let fp = AudioFingerprint::from_bytes(fp_bytes)?;
let matches = self.search(&fp, threshold, 100)?;
if matches.len() > 1 {
let group = DuplicateGroup {
files: matches.iter().map(|m| m.file_id).collect(),
similarity: matches.iter().map(|m| m.similarity).sum::<f32>() / matches.len() as f32,
};
for m in &matches {
processed.insert(m.file_id);
}
groups.push(group);
}
}
Ok(groups)
}
}
#[derive(Debug)]
pub struct FingerprintMatch {
pub file_id: FileId,
pub similarity: f32,
pub duration: u32,
}
#[derive(Debug)]
pub struct DuplicateGroup {
pub files: Vec<FileId>,
pub similarity: f32,
}
```
---
## Task 3: M4B Audiobook Support (`musicfs-metadata/src/formats/m4b.rs`)
```rust
use symphonia::core::meta::StandardTagKey;
/// M4B audiobook metadata (FR-24.2)
#[derive(Debug, Clone, Default)]
pub struct AudiobookMeta {
pub title: Option<String>,
pub author: Option<String>, // Maps to "artist" in audio
pub narrator: Option<String>,
pub series: Option<String>,
pub series_part: Option<u32>,
pub description: Option<String>,
pub publisher: Option<String>,
pub year: Option<u32>,
pub duration_ms: Option<u64>,
pub chapters: Vec<Chapter>,
}
#[derive(Debug, Clone)]
pub struct Chapter {
pub index: u32,
pub title: String,
pub start_ms: u64,
pub end_ms: u64,
}
impl Chapter {
pub fn duration_ms(&self) -> u64 {
self.end_ms - self.start_ms
}
}
pub struct M4bParser;
impl M4bParser {
/// Parse M4B audiobook with chapters
pub fn parse(&self, path: &Path) -> Result<AudiobookMeta, MetadataError> {
let file = std::fs::File::open(path)?;
let mss = MediaSourceStream::new(Box::new(file), Default::default());
let mut hint = Hint::new();
hint.with_extension("m4b");
let probed = symphonia::default::get_probe()
.format(&hint, mss, &FormatOptions::default(), &MetadataOptions::default())?;
let mut meta = AudiobookMeta::default();
let format = probed.format;
// Extract metadata
if let Some(metadata) = format.metadata().current() {
for tag in metadata.tags() {
if let Some(std_key) = tag.std_key {
let value = tag.value.to_string();
match std_key {
StandardTagKey::TrackTitle | StandardTagKey::Album => {
meta.title = Some(value);
}
StandardTagKey::Artist => {
meta.author = Some(value);
}
StandardTagKey::Composer => {
meta.narrator = Some(value);
}
StandardTagKey::Description => {
meta.description = Some(value);
}
StandardTagKey::Label => {
meta.publisher = Some(value);
}
StandardTagKey::Date => {
meta.year = value.chars().take(4).collect::<String>().parse().ok();
}
_ => {}
}
}
}
}
// Extract chapters from MP4 chpl atom
meta.chapters = self.extract_chapters(&format)?;
// Get total duration
if let Some(track) = format.tracks().first() {
if let (Some(n_frames), Some(sample_rate)) =
(track.codec_params.n_frames, track.codec_params.sample_rate)
{
meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64);
}
}
Ok(meta)
}
fn extract_chapters(&self, format: &dyn FormatReader) -> Result<Vec<Chapter>, MetadataError> {
let mut chapters = Vec::new();
// Symphonia exposes chapters via cues
if let Some(cues) = format.cues() {
for (idx, cue) in cues.iter().enumerate() {
let start_ms = (cue.start_ts as f64 / cue.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64;
// End time is start of next chapter or track end
let end_ms = cues.get(idx + 1)
.map(|next| (next.start_ts as f64 / next.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64)
.unwrap_or(u64::MAX); // Will be clamped to duration
chapters.push(Chapter {
index: idx as u32,
title: cue.tags.iter()
.find(|t| t.std_key == Some(StandardTagKey::TrackTitle))
.map(|t| t.value.to_string())
.unwrap_or_else(|| format!("Chapter {}", idx + 1)),
start_ms,
end_ms,
});
}
}
Ok(chapters)
}
}
```
---
## Task 4: Chapter Extraction (`musicfs-metadata/src/chapters.rs`)
```rust
/// Generic chapter support for various formats
pub trait ChapterSource {
fn chapters(&self) -> &[Chapter];
fn chapter_at(&self, position_ms: u64) -> Option<&Chapter>;
}
impl ChapterSource for AudiobookMeta {
fn chapters(&self) -> &[Chapter] {
&self.chapters
}
fn chapter_at(&self, position_ms: u64) -> Option<&Chapter> {
self.chapters.iter()
.find(|c| position_ms >= c.start_ms && position_ms < c.end_ms)
}
}
/// Virtual chapter file generator
pub struct ChapterFileGenerator;
impl ChapterFileGenerator {
/// Generate virtual files for each chapter
/// Example: book.m4b -> book/01 - Introduction.m4b.chapter
pub fn generate_virtual_files(&self, meta: &AudiobookMeta, base_path: &VirtualPath) -> Vec<VirtualChapterFile> {
meta.chapters.iter()
.map(|chapter| {
let filename = format!(
"{:02} - {}.chapter",
chapter.index + 1,
sanitize_filename(&chapter.title)
);
VirtualChapterFile {
path: base_path.join(&filename),
chapter_index: chapter.index,
start_ms: chapter.start_ms,
end_ms: chapter.end_ms,
title: chapter.title.clone(),
}
})
.collect()
}
}
#[derive(Debug)]
pub struct VirtualChapterFile {
pub path: VirtualPath,
pub chapter_index: u32,
pub start_ms: u64,
pub end_ms: u64,
pub title: String,
}
fn sanitize_filename(name: &str) -> String {
name.chars()
.map(|c| match c {
'/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
_ => c,
})
.collect()
}
```
---
## Task 5: Virtual Chapter Files (`musicfs-fuse/src/ops/chapters.rs`)
```rust
use crate::VirtualFs;
impl VirtualFs {
/// Handle reads from virtual chapter files
/// These return a byte-range reference to the parent M4B file
pub async fn read_chapter(
&self,
chapter_file: &VirtualChapterFile,
offset: u64,
size: usize,
) -> Result<Vec<u8>, FuseError> {
// Get the parent audiobook file
let parent = self.get_parent_audiobook(&chapter_file.path)?;
// Calculate byte range for this chapter
// This requires knowing the audio bitrate to convert ms -> bytes
let meta = self.get_audiobook_meta(&parent)?;
let bitrate_bps = meta.bitrate.unwrap_or(128_000); // Default 128kbps
let bytes_per_ms = bitrate_bps / 8 / 1000;
let chapter_start_bytes = chapter_file.start_ms * bytes_per_ms;
let chapter_end_bytes = chapter_file.end_ms * bytes_per_ms;
// Adjust offset to be within chapter
let actual_offset = chapter_start_bytes + offset;
let max_size = (chapter_end_bytes - actual_offset) as usize;
let read_size = size.min(max_size);
// Read from the actual file
self.read_file(&parent, actual_offset, read_size).await
}
/// List chapter files for an audiobook
pub fn list_chapters(&self, audiobook_path: &VirtualPath) -> Result<Vec<DirEntry>, FuseError> {
let meta = self.get_audiobook_meta(audiobook_path)?;
let generator = ChapterFileGenerator;
let chapters = generator.generate_virtual_files(&meta, audiobook_path);
Ok(chapters.into_iter()
.map(|c| DirEntry {
name: c.path.filename().to_string(),
kind: FileType::RegularFile,
size: self.estimate_chapter_size(&c),
})
.collect())
}
fn estimate_chapter_size(&self, chapter: &VirtualChapterFile) -> u64 {
// Estimate based on duration and typical bitrate
let duration_secs = (chapter.end_ms - chapter.start_ms) / 1000;
duration_secs * 128_000 / 8 // 128kbps assumption
}
}
```
---
## Task 6: Fingerprint Search Virtual Directory
```rust
/// Virtual directory for fingerprint search
/// /.search/fingerprint/{base64_fingerprint} -> matching files
impl SearchOps {
pub async fn search_by_fingerprint(
&self,
fingerprint_path: &str,
) -> Result<Vec<SearchResult>, SearchError> {
// Path format: /.search/fingerprint/{base64_encoded_fingerprint}
let fp_bytes = base64::decode(fingerprint_path)
.map_err(|_| SearchError::InvalidQuery)?;
let fingerprint = AudioFingerprint::from_bytes(&fp_bytes)?;
let matches = self.fingerprint_index.search(&fingerprint, 0.8, 20)?;
let mut results = Vec::new();
for m in matches {
if let Some(file) = self.db.get_file_by_id(m.file_id)? {
results.push(SearchResult {
path: file.virtual_path,
score: m.similarity,
snippet: format!("Similarity: {:.1}%", m.similarity * 100.0),
});
}
}
Ok(results)
}
}
```
---
## Database Schema Additions
```sql
-- Fingerprint storage
CREATE TABLE IF NOT EXISTS fingerprints (
file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE,
fingerprint BLOB NOT NULL, -- Compressed chromaprint
duration INTEGER NOT NULL, -- Duration in seconds
indexed_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
);
CREATE INDEX IF NOT EXISTS idx_fingerprints_duration ON fingerprints(duration);
-- Audiobook chapters
CREATE TABLE IF NOT EXISTS chapters (
id INTEGER PRIMARY KEY,
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
chapter_idx INTEGER NOT NULL,
title TEXT NOT NULL,
start_ms INTEGER NOT NULL,
end_ms INTEGER NOT NULL,
UNIQUE(file_id, chapter_idx)
);
CREATE INDEX IF NOT EXISTS idx_chapters_file ON chapters(file_id);
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_fingerprint_generation` | Unit | Chromaprint from audio (FR-14.4) |
| `test_fingerprint_similarity` | Unit | Bit comparison algorithm |
| `test_fingerprint_search` | Integration | Find similar tracks |
| `test_fingerprint_duplicates` | Integration | Detect duplicate audio |
| `test_m4b_parsing` | Unit | M4B metadata extraction (FR-24.2) |
| `test_chapter_extraction` | Unit | Chapter list from M4B |
| `test_virtual_chapter_files` | Integration | Chapter files appear in listing |
| `test_chapter_read` | Integration | Read chapter content |
| `test_audiobook_navigation` | E2E | Browse audiobook chapters |
---
## Exit Criteria
- [ ] Audio fingerprints generated from audio files
- [ ] Fingerprint similarity search finds matching tracks
- [ ] Duplicate detection works across library
- [ ] M4B files parsed with full metadata
- [ ] Chapters extracted and stored
- [ ] Virtual chapter files appear in directory listing
- [ ] Chapter files are readable (return correct byte range)
- [ ] All tests pass
---
## Architecture Alignment
Per requirements.md:
- FR-14.4: Audio fingerprint search ✓
- FR-24.2: Audiobook formats with chapters ✓
Per architecture.md section 4.3.4:
- FormatPlugin trait for M4B support ✓
- Chapter extraction via symphonia ✓
docs/v2/requirements.md
-649
View File
@@ -1,649 +0,0 @@
# Music Library FUSE Filesystem - Requirements Specification
**Version**: 1.0
**Date**: 2026-05-12
**Status**: Draft
## 1. Introduction
### 1.1 Purpose
This document specifies the requirements for a FUSE-based virtual filesystem that presents a music library organized by metadata. The system overlays metadata onto audio files without modifying originals and operates as a read-only client against the origin storage.
### 1.2 Scope
The system provides:
- Virtual filesystem accessible via standard POSIX operations
- Metadata-based directory structure (artist/album/track)
- Local caching with delta synchronization
- Support for local and remote origin storage
### 1.3 Definitions
| Term | Definition |
|------|------------|
| **Origin** | The source storage containing original audio files (local FS, NFS, S3, etc.) |
| **Virtual path** | The metadata-derived path shown to users (e.g., `/Artist/Album/Track.flac`) |
| **Real path** | The actual path on origin storage |
| **Metadata overlay** | Serving synthesized file headers from cached metadata |
| **CDC** | Content-Defined Chunking - algorithm for stable file segmentation |
---
## 2. System Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ User Applications │
│ (mpv, Rhythmbox, Plex, etc.) │
└─────────────────────────────┬───────────────────────────────────┘
│ POSIX (read-only)
▼
┌─────────────────────────────────────────────────────────────────┐
│ FUSE Interface │
├─────────────────────────────────────────────────────────────────┤
│ Plugin Host │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Origin │ │ Metadata │ │ Format │ │
│ │ Plugins │ │ Plugins │ │ Plugins │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Core Services │
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
│ │ Virtual │ │ Event │ │ Search │ │ Control │ │
│ │ Path │ │ Bus │ │ Index │ │ API │ │
│ │ Resolver │ │ │ │ │ │ │ │
│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Storage Layer │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Content-Addressable Chunk Store │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Metadata │ │ Content │ │ Tree │ │ │
│ │ │ Cache │ │ Chunks │ │ Cache │ │ │
│ │ │ (SQLite) │ │ (CAS) │ │ │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
│ └─────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│ Origin Federation │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Local │ │ NFS │ │ S3 │ │ SFTP │ │
│ │ FS │ │ │ │ │ │ │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ read-only
▼
┌─────────────────────────────────────────────────────────────────┐
│ Origin Storage(s) │
│ (original audio files) │
└─────────────────────────────────────────────────────────────────┘
```
---
## 3. Functional Requirements
### 3.1 Filesystem Operations
#### FR-1: Mount/Unmount
| ID | Requirement |
|----|-------------|
| FR-1.1 | The system SHALL mount as a FUSE filesystem at a user-specified mountpoint |
| FR-1.2 | The system SHALL return control to the caller within 500ms of mount initiation |
| FR-1.3 | The system SHALL unmount cleanly via `fusermount -u` |
| FR-1.4 | The system SHALL release all resources (file handles, connections) on unmount |
#### FR-2: Directory Operations
| ID | Requirement |
|----|-------------|
| FR-2.1 | The system SHALL present files organized by metadata path format |
| FR-2.2 | The system SHALL support configurable path templates (e.g., `$artist/$album/$track - $title.$format`) |
| FR-2.3 | The system SHALL return directory listings via `readdir()` |
| FR-2.4 | The system SHALL support nested directory traversal to arbitrary depth |
| FR-2.5 | The system SHALL handle directories with 100,000+ entries |
#### FR-3: File Operations (Read)
| ID | Requirement |
|----|-------------|
| FR-3.1 | The system SHALL support `open()` for reading |
| FR-3.2 | The system SHALL support `read()` with arbitrary offset and size |
| FR-3.3 | The system SHALL support `seek()` operations for random access |
| FR-3.4 | The system SHALL return file attributes via `stat()` / `fstat()` |
| FR-3.5 | The system SHALL support concurrent reads from multiple processes |
#### FR-4: Read-Only Constraint
| ID | Requirement |
|----|-------------|
| FR-4.1 | The system SHALL NOT modify original files on the origin storage |
| FR-4.2 | The system SHALL NOT push any changes to the origin server |
| FR-4.3 | The system SHALL return `EROFS` (Read-only filesystem) for write operations |
| FR-4.4 | The system SHALL return `EROFS` for `create()`, `mkdir()`, `unlink()`, `rmdir()` |
| FR-4.5 | The system SHALL return `EROFS` for `rename()`, `chmod()`, `chown()`, `truncate()` |
### 3.2 Metadata Handling
#### FR-5: Metadata Overlay
| ID | Requirement |
|----|-------------|
| FR-5.1 | The system SHALL extract metadata from audio files on first access |
| FR-5.2 | The system SHALL cache extracted metadata in a local database |
| FR-5.3 | The system SHALL serve file headers with metadata from cache |
| FR-5.4 | The system SHALL support FLAC Vorbis comments |
| FR-5.5 | The system SHALL support MP3 ID3v2 tags |
| FR-5.6 | The system SHOULD support additional formats (OGG, M4A, OPUS) |
#### FR-6: Metadata Fields
| ID | Requirement |
|----|-------------|
| FR-6.1 | The system SHALL extract and cache: title, artist, album, genre |
| FR-6.2 | The system SHALL extract and cache: year, track number, disc number |
| FR-6.3 | The system SHALL extract and cache: duration, bitrate, sample rate |
| FR-6.4 | The system SHOULD extract: composer, album artist, lyrics |
| FR-6.5 | The system SHALL handle missing metadata gracefully with defaults |
### 3.3 Caching
#### FR-7: Metadata Cache
| ID | Requirement |
|----|-------------|
| FR-7.1 | The system SHALL persist metadata cache across restarts |
| FR-7.2 | The system SHALL store metadata in SQLite database |
| FR-7.3 | The system SHALL index by both virtual path and real path |
| FR-7.4 | The system SHALL invalidate cache entries when origin file changes |
#### FR-8: Content Cache
| ID | Requirement |
|----|-------------|
| FR-8.1 | The system SHALL cache file content in fixed-size chunks |
| FR-8.2 | The system SHALL use content-defined chunking for cache efficiency |
| FR-8.3 | The system SHALL store chunk hashes for delta detection |
| FR-8.4 | The system SHALL evict chunks under memory/disk pressure |
#### FR-9: Directory Tree Cache
| ID | Requirement |
|----|-------------|
| FR-9.1 | The system SHALL cache directory listings locally |
| FR-9.2 | The system SHALL serve `readdir()` from cache without origin access |
| FR-9.3 | The system SHALL refresh tree cache based on configurable policy |
| FR-9.4 | The system SHALL support forced refresh via signal or special file |
### 3.4 Synchronization
#### FR-10: Change Detection
| ID | Requirement |
|----|-------------|
| FR-10.1 | The system SHALL detect changes to origin files |
| FR-10.2 | The system SHALL use inotify for local filesystem origins |
| FR-10.3 | The system SHALL use polling for remote origins without push support |
| FR-10.4 | The system SHALL compare mtime and size for change detection |
| FR-10.5 | The system SHALL support content-hash verification on demand |
#### FR-11: Delta Sync
| ID | Requirement |
|----|-------------|
| FR-11.1 | The system SHALL download only changed portions of files |
| FR-11.2 | The system SHALL use CDC to identify changed chunks |
| FR-11.3 | The system SHALL preserve unchanged chunks in cache |
| FR-11.4 | The system SHALL handle file additions and deletions |
### 3.5 Origin Support
#### FR-12: Origin Types
| ID | Requirement |
|----|-------------|
| FR-12.1 | The system SHALL support local filesystem as origin |
| FR-12.2 | The system SHOULD support NFS mounted filesystems |
| FR-12.3 | The system SHOULD support SMB/CIFS shares |
| FR-12.4 | The system SHOULD support S3-compatible object storage |
| FR-12.5 | The system SHOULD support SFTP servers |
| FR-12.6 | The system SHALL provide pluggable origin interface |
#### FR-13: Multiple Origins [P0]
| ID | Requirement |
|----|-------------|
| FR-13.1 | The system SHALL support multiple simultaneous origins |
| FR-13.2 | The system SHALL present unified virtual tree across origins |
| FR-13.3 | The system SHALL support origin priority/preference ordering |
| FR-13.4 | The system SHALL handle duplicate files across origins |
| FR-13.5 | The system SHALL support per-origin configuration |
### 3.6 Search & Discovery
#### FR-14: Full-Text Search [P1]
| ID | Requirement |
|----|-------------|
| FR-14.1 | The system SHALL index metadata for full-text search |
| FR-14.2 | The system SHALL expose search via virtual directory (`/.search/query/`) |
| FR-14.3 | The system SHALL support fuzzy matching |
| FR-14.4 | The system SHOULD support search by audio fingerprint |
#### FR-15: Smart Collections [P1]
| ID | Requirement |
|----|-------------|
| FR-15.1 | The system SHALL support query-based virtual folders |
| FR-15.2 | The system SHALL support saved searches as directories |
| FR-15.3 | The system SHALL support dynamic playlists (recently played, most played) |
| FR-15.4 | The system SHOULD support user-defined metadata fields |
### 3.7 Album Art
#### FR-16: Cover Art Handling [P1]
| ID | Requirement |
|----|-------------|
| FR-16.1 | The system SHALL extract embedded album art |
| FR-16.2 | The system SHALL expose art as virtual files (`/Artist/Album/cover.jpg`) |
| FR-16.3 | The system SHALL cache artwork separately from audio |
| FR-16.4 | The system SHALL support multiple art sizes (thumbnail, medium, full) |
| FR-16.5 | The system SHOULD fetch missing art from online sources |
### 3.8 Control & API
#### FR-17: Control Interface [P0]
| ID | Requirement |
|----|-------------|
| FR-17.1 | The system SHALL expose control via Unix socket (gRPC) |
| FR-17.2 | The system SHALL use gRPC with Protocol Buffers for all control APIs |
| FR-17.3 | The system SHALL support cache management commands (clear, refresh, stats) |
| FR-17.4 | The system SHALL support runtime configuration changes |
| FR-17.5 | The system SHALL support graceful shutdown with drain |
#### FR-18: Event System [P0]
| ID | Requirement |
|----|-------------|
| FR-18.1 | The system SHALL emit events for file access |
| FR-18.2 | The system SHALL support webhook notifications |
| FR-18.3 | The system SHOULD support event streaming (SSE/WebSocket) |
| FR-18.4 | The system SHALL log access patterns for analysis |
### 3.9 Caching Enhancements
#### FR-19: Intelligent Prefetching [P1]
| ID | Requirement |
|----|-------------|
| FR-19.1 | The system SHALL learn access patterns |
| FR-19.2 | The system SHALL support playlist-aware prefetching |
| FR-19.3 | The system SHOULD support time-based prefetching |
| FR-19.4 | The system SHALL support manual prefetch hints (`/.prefetch/path/`) |
#### FR-20: Content-Addressable Storage [P0]
| ID | Requirement |
|----|-------------|
| FR-20.1 | The system SHALL store chunks by content hash |
| FR-20.2 | The system SHALL detect identical files across library |
| FR-20.3 | The system SHALL report deduplication statistics |
| FR-20.4 | The system SHALL enable cache sharing via content addressing |
### 3.10 Integration
#### FR-21: Metadata Sources [P1]
| ID | Requirement |
|----|-------------|
| FR-21.1 | The system SHOULD integrate with MusicBrainz |
| FR-21.2 | The system SHOULD integrate with Discogs |
| FR-21.3 | The system SHOULD integrate with Last.fm |
| FR-21.4 | The system SHOULD support AcoustID fingerprinting |
| FR-21.5 | The system SHALL support custom metadata plugins |
#### FR-22: Import & Migration [P1]
| ID | Requirement |
|----|-------------|
| FR-22.1 | The system SHALL import from beets database |
| FR-22.2 | The system SHOULD import from iTunes/Apple Music library |
| FR-22.3 | The system SHALL export library metadata |
### 3.11 Extensibility
#### FR-23: Plugin System [P0]
| ID | Requirement |
|----|-------------|
| FR-23.1 | The system SHALL support loadable plugins |
| FR-23.2 | The system SHALL define stable plugin API |
| FR-23.3 | The system SHALL support plugins for: origins, metadata extractors, formats |
| FR-23.4 | The system SHOULD support WASM plugins for sandboxed execution |
| FR-23.5 | The system SHALL provide plugin lifecycle management (load, unload, reload) |
#### FR-24: Format Extensibility [P1]
| ID | Requirement |
|----|-------------|
| FR-24.1 | The system SHALL support pluggable codec modules |
| FR-24.2 | The system SHOULD support audiobook formats (M4B, chapters) |
| FR-24.3 | The system SHALL allow format plugins to register file extensions |
### 3.12 High Availability [P3]
#### FR-25: Resilience
| ID | Requirement |
|----|-------------|
| FR-25.1 | The system SHOULD support active-passive failover |
| FR-25.2 | The system SHOULD support read replicas |
| FR-25.3 | The system SHALL support zero-downtime upgrades |
| FR-25.4 | The system SHALL support cache backup/restore |
| FR-25.5 | The system SHALL validate cache integrity on startup |
---
## 4. Non-Functional Requirements
### 4.1 Performance
#### NFR-1: Latency
| ID | Requirement | Target | Maximum |
|----|-------------|--------|---------|
| NFR-1.1 | `stat()` on cached file | <1ms | 5ms |
| NFR-1.2 | `readdir()` on cached directory | <10ms | 50ms |
| NFR-1.3 | `open()` on cached file | <5ms | 20ms |
| NFR-1.4 | `read()` from cache | <1ms | 5ms |
| NFR-1.5 | `read()` cache miss (local origin) | <50ms | 200ms |
| NFR-1.6 | `read()` cache miss (remote origin) | <200ms | 1000ms |
| NFR-1.7 | Mount completion | <100ms | 500ms |
#### NFR-2: Throughput
| ID | Requirement | Target |
|----|-------------|--------|
| NFR-2.1 | Sequential read throughput (cached) | >500 MB/s |
| NFR-2.2 | Sequential read throughput (local origin) | >200 MB/s |
| NFR-2.3 | Metadata operations per second | >1000 ops/s |
| NFR-2.4 | Concurrent file handles | >1000 |
#### NFR-3: Scalability
| ID | Requirement |
|----|-------------|
| NFR-3.1 | The system SHALL handle libraries with 1,000,000+ files |
| NFR-3.2 | The system SHALL handle directories with 100,000+ entries |
| NFR-3.3 | The system SHALL maintain O(1) mount time regardless of library size |
| NFR-3.4 | The system SHALL maintain O(log n) lookup time for paths |
| NFR-3.5 | The system SHOULD handle libraries with 10,000,000+ files [P3] |
| NFR-3.6 | The system SHOULD support 100+ concurrent clients [P3] |
| NFR-3.7 | The system SHOULD achieve <100μs cached stat for high-performance use [P3] |
### 4.2 Resource Usage
#### NFR-4: Memory
| ID | Requirement | Limit |
|----|-------------|-------|
| NFR-4.1 | Idle memory usage | <50 MB |
| NFR-4.2 | Active usage (1000 files accessed) | <200 MB |
| NFR-4.3 | Peak usage under load | <500 MB |
| NFR-4.4 | Per-file metadata overhead | <1 KB |
| NFR-4.5 | The system SHALL NOT load entire files into memory |
#### NFR-5: Disk
| ID | Requirement |
|----|-------------|
| NFR-5.1 | Metadata cache size SHALL be configurable (default: 100 MB) |
| NFR-5.2 | Content cache size SHALL be configurable (default: 10 GB) |
| NFR-5.3 | The system SHALL evict cache entries under disk pressure |
| NFR-5.4 | The system SHALL function with cache disabled (passthrough mode) |
#### NFR-6: Network
| ID | Requirement |
|----|-------------|
| NFR-6.1 | The system SHALL minimize network round-trips via batching |
| NFR-6.2 | The system SHALL use connection pooling for remote origins |
| NFR-6.3 | The system SHALL support bandwidth limiting (configurable) |
| NFR-6.4 | Delta sync SHALL achieve >90% bandwidth reduction vs full copy |
### 4.3 Reliability
#### NFR-7: Availability
| ID | Requirement |
|----|-------------|
| NFR-7.1 | The system SHALL serve cached data when origin is unavailable |
| NFR-7.2 | The system SHALL gracefully degrade with network failures |
| NFR-7.3 | The system SHALL retry failed operations with exponential backoff |
| NFR-7.4 | The system SHALL not crash on malformed audio files |
#### NFR-8: Data Integrity
| ID | Requirement |
|----|-------------|
| NFR-8.1 | The system SHALL verify chunk integrity via checksums |
| NFR-8.2 | The system SHALL use ACID transactions for cache database |
| NFR-8.3 | The system SHALL recover from interrupted synchronization |
| NFR-8.4 | The system SHALL detect and report cache corruption |
### 4.4 Usability
#### NFR-9: Configuration
| ID | Requirement |
|----|-------------|
| NFR-9.1 | The system SHALL support configuration via file (TOML/YAML) |
| NFR-9.2 | The system SHALL support configuration via command-line arguments |
| NFR-9.3 | The system SHALL support configuration via environment variables |
| NFR-9.4 | The system SHALL provide sensible defaults for all options |
#### NFR-10: Observability
| ID | Requirement |
|----|-------------|
| NFR-10.1 | The system SHALL log operations at configurable verbosity |
| NFR-10.2 | The system SHALL expose metrics (cache hit rate, latency, etc.) |
| NFR-10.3 | The system SHALL support health check endpoint/signal |
| NFR-10.4 | The system SHOULD support integration with Prometheus/StatsD |
### 4.5 Compatibility
#### NFR-11: Platform Support
| ID | Requirement |
|----|-------------|
| NFR-11.1 | The system SHALL run on Linux (kernel 4.x+) |
| NFR-11.2 | The system SHOULD run on macOS (via macFUSE) |
| NFR-11.3 | The system SHALL require FUSE kernel module |
| NFR-11.4 | The system SHALL run without root privileges (user-space FUSE) |
#### NFR-12: Application Compatibility
| ID | Requirement |
|----|-------------|
| NFR-12.1 | The system SHALL work with standard media players (mpv, VLC, etc.) |
| NFR-12.2 | The system SHALL work with media servers (Plex, Jellyfin) |
| NFR-12.3 | The system SHALL work with file managers (Nautilus, Dolphin) |
| NFR-12.4 | The system SHALL correctly report file sizes and timestamps |
### 4.6 Security
#### NFR-13: Access Control
| ID | Requirement |
|----|-------------|
| NFR-13.1 | The system SHALL respect origin file permissions |
| NFR-13.2 | The system SHALL run as unprivileged user |
| NFR-13.3 | The system SHALL support credential storage for remote origins |
| NFR-13.4 | The system SHALL NOT expose credentials in logs or process list |
### 4.7 Maintainability
#### NFR-14: Code Quality
| ID | Requirement |
|----|-------------|
| NFR-14.1 | The system SHALL be implemented in a memory-safe language |
| NFR-14.2 | The system SHALL have no global interpreter lock (no Python/Ruby) |
| NFR-14.3 | The system SHALL use async I/O for concurrent operations |
| NFR-14.4 | The system SHALL have modular architecture with pluggable components |
---
## 5. Constraints
### 5.1 Technical Constraints
| ID | Constraint |
|----|------------|
| C-1 | Must use FUSE for filesystem interface |
| C-2 | Must not require kernel module development |
| C-3 | Must work with existing audio file formats (no transcoding) |
| C-4 | Cache database must be portable (no external database server) |
### 5.2 Operational Constraints
| ID | Constraint |
|----|------------|
| C-5 | Client is read-only; no writes propagate to origin |
| C-6 | Must function offline with cached data |
| C-7 | Must not corrupt origin files under any circumstances |
---
## 6. Assumptions
| ID | Assumption |
|----|------------|
| A-1 | Origin storage is accessible via supported protocol |
| A-2 | Audio files contain valid metadata headers |
| A-3 | Sufficient local disk space for caching is available |
| A-4 | FUSE kernel module is installed and accessible |
| A-5 | Network connectivity is intermittent but generally available |
---
## 7. Dependencies
| ID | Dependency | Purpose |
|----|------------|---------|
| D-1 | FUSE library (fuser/libfuse) | Filesystem interface |
| D-2 | SQLite | Metadata and tree cache |
| D-3 | Audio parsing library (symphonia) | Metadata extraction |
| D-4 | Async runtime (tokio) | Concurrent I/O |
| D-5 | CDC library (fastcdc) | Content chunking |
| D-6 | Full-text search (tantivy) | Search index [P1] |
| D-7 | Image processing (image) | Album art thumbnails [P1] |
| D-8 | HTTP client (reqwest) | Remote origins, metadata APIs |
| D-9 | WASM runtime (wasmtime) | Plugin sandboxing [P0] |
| D-10 | Hash library (xxhash/blake3) | Content addressing [P0] |
---
## 8. Acceptance Criteria
### 8.1 Functional Acceptance
| ID | Criterion |
|----|-----------|
| AC-1 | Mount filesystem and browse directories via `ls` |
| AC-2 | Play audio file through mounted filesystem with media player |
| AC-3 | Seek within audio file without full download |
| AC-4 | Directory listing completes without network access (when cached) |
| AC-5 | Confirm write operations return EROFS |
| AC-6 | Detect and sync changes from origin within configured interval |
### 8.2 Performance Acceptance
| ID | Criterion |
|----|-----------|
| AC-7 | Mount completes in <500ms for library of any size |
| AC-8 | Cached stat() completes in <5ms (p99) |
| AC-9 | Memory stays under 500MB with 10,000 files accessed |
| AC-10 | Tag-only change syncs <10KB of data |
### 8.3 Reliability Acceptance
| ID | Criterion |
|----|-----------|
| AC-11 | Filesystem remains accessible when origin is offline |
| AC-12 | No data corruption after unclean unmount |
| AC-13 | Recovers automatically when origin comes back online |
### 8.4 Multi-Origin Acceptance [P0]
| ID | Criterion |
|----|-----------|
| AC-14 | Configure and mount multiple origins simultaneously |
| AC-15 | Browse unified tree showing content from all origins |
| AC-16 | Access same file from preferred origin when duplicated |
### 8.5 Search & Discovery Acceptance [P1]
| ID | Criterion |
|----|-----------|
| AC-17 | Search for tracks by partial artist/album/title match |
| AC-18 | Browse smart collection (e.g., "Jazz from 1960s") |
| AC-19 | View album art via virtual cover.jpg file |
### 8.6 Plugin Acceptance [P0]
| ID | Criterion |
|----|-----------|
| AC-20 | Load custom origin plugin at runtime |
| AC-21 | Control daemon via Unix socket (cache stats, refresh) |
| AC-22 | Receive webhook on file access event |
### 8.7 Deduplication Acceptance [P0]
| ID | Criterion |
|----|-----------|
| AC-23 | Identical chunks stored once regardless of file count |
| AC-24 | Deduplication stats visible via control API |
---
## 9. Appendix
### 9.1 Comparison with beetfs
| Requirement Area | beetfs | This Specification |
|------------------|--------|-------------------|
| Mount time | O(N), 5-120s | O(1), <500ms (NFR-1.7) |
| Memory per file | Full file size | <1KB (NFR-4.4) |
| Write to origin | Yes (DB updates) | No (FR-4.1, FR-4.2) |
| Delta sync | None | Required (FR-11) |
| Remote origins | None | Required (FR-12) |
| Offline access | No | Required (NFR-7.1) |
| Cache persistence | No | Required (FR-7.1) |
### 9.2 Path Template Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `$artist` | Track artist | "Metallica" |
| `$album` | Album name | "72 Seasons" |
| `$title` | Track title | "Lux Æterna" |
| `$track` | Track number (zero-padded) | "03" |
| `$disc` | Disc number | "1" |
| `$year` | Release year | "2023" |
| `$genre` | Genre | "Metal" |
| `$format` | File extension | "flac" |
| `$format_upper` | File extension (uppercase) | "FLAC" |
### 9.3 Error Codes
| Operation | Error | Code |
|-----------|-------|------|
| Any write operation | Read-only filesystem | EROFS (30) |
| File not found | No such file | ENOENT (2) |
| Origin unavailable | I/O error | EIO (5) |
| Permission denied | Access denied | EACCES (13) |
docs/v2/week-07-performance-review.md
-179
View File
@@ -1,179 +0,0 @@
# MusicFS Week 7 Performance Review
**Date**: 2026-05-12
**Commit**: `09f0197` (Week 7 Remote Origins)
**Baseline**: `d5ef68c` (Week 6 Origin Federation)
**System**: Linux, NixOS
**Test**: Synthetic benchmarks (CDC chunking, hashing, chunk reuse)
---
## Executive Summary
**Week 7 Remote Origins adds no performance regression.** The core CDC and hashing algorithms remain unchanged; Week 7 adds I/O wrappers (NFS, SMB, S3, SFTP) that are network-bound, not CPU-bound. All NFR targets continue to be met or exceeded.
---
## Benchmark Results
### CDC Chunker Throughput
| Metric | Week 6 | Week 7 | Delta | NFR Target | Status |
|--------|--------|--------|-------|------------|--------|
| CDC Throughput | 3148.7 MB/s | 3007.9 MB/s | -4.5% | N/A* | ✅ |
| Chunks per 10MB | 137 | 137 | 0% | — | ✅ |
*CDC throughput is internal; NFR-2.1/2.2 measure end-to-end read throughput (>500 MB/s cached, >200 MB/s local origin). CDC at ~3 GB/s confirms chunking is not a bottleneck.
### Hash Computation Throughput
| Metric | Week 6 | Week 7 | Delta | Status |
|--------|--------|--------|-------|--------|
| xxHash64 Throughput | 16330.7 MB/s | 16274.6 MB/s | -0.3% | ✅ |
Hash computation at ~16 GB/s is CPU-limited and far exceeds any I/O bottleneck.
### Chunk Reuse (NFR-6.4)
| Metric | Week 6 | Week 7 | NFR-6.4 Target | Status |
|--------|--------|--------|----------------|--------|
| Chunk Reuse | 99.1% | 99.1% | >90% | ✅ PASS |
| Reused Chunks | 107/108 | 107/108 | — | — |
| Edit Size | 100 bytes | 100 bytes | — | — |
**NFR-6.4**: *"Delta sync SHALL achieve >90% bandwidth reduction vs full copy"*
Result: **99.1% bandwidth reduction** for mid-file metadata edits (100 bytes changed in 2MB file). This exceeds the >90% requirement by 9.1 percentage points.
---
## Requirements Compliance
### NFR-2: Throughput
| ID | Requirement | Target | Measured | Status |
|----|-------------|--------|----------|--------|
| NFR-2.1 | Sequential read (cached) | >500 MB/s | ~3000 MB/s* | ✅ |
| NFR-2.2 | Sequential read (local origin) | >200 MB/s | ~3000 MB/s* | ✅ |
*Measured at CDC layer. End-to-end throughput demonstrated in MVP review (2-3 GB/s).
### NFR-6: Network
| ID | Requirement | Target | Measured | Status |
|----|-------------|--------|----------|--------|
| NFR-6.4 | Delta sync bandwidth reduction | >90% | 99.1% | ✅ |
### NFR-7: Availability (Week 7 Additions)
| ID | Requirement | Implementation | Status |
|----|-------------|----------------|--------|
| NFR-7.3 | Retry with exponential backoff | NFS: ESTALE retry (100ms→200ms→400ms) | ✅ |
| NFR-7.3 | Retry with exponential backoff | SMB: ENOTCONN retry (100ms fixed) | ✅ |
---
## Week 7 Changes Analysis
### What Changed (No Performance Impact Expected)
| Component | Change | Performance Impact |
|-----------|--------|-------------------|
| `credentials.rs` | New CredentialStore with redacted Debug | None (startup only) |
| `nfs.rs` | NfsOrigin with ESTALE retry, 5s health timeout | None (error path only) |
| `smb.rs` | SmbOrigin with ENOTCONN retry, 5s health timeout | None (error path only) |
| `s3.rs` | Feature-gated stub | None (not compiled) |
| `sftp.rs` | Feature-gated stub | None (not compiled) |
| `error.rs` | New error variants | None (enum extension) |
### Why ~4.5% CDC Variance is Noise
The 4.5% difference (3148.7 → 3007.9 MB/s) is within expected benchmark noise:
1. **No code path changed** — FastCDC algorithm unchanged
2. **CPU frequency variation** — Turbo boost, thermal throttling
3. **Memory subsystem** — Cache line evictions, NUMA effects
4. **OS scheduler** — Process placement, interrupt handling
A 4.5% variance over 10 iterations of 10MB data is statistically insignificant. To detect real regressions, we'd need:
- Warmup iterations (discard first N)
- Statistical analysis (mean, stddev, p-value)
- Dedicated benchmark infrastructure (criterion.rs)
---
## Comparison with MVP Performance Review
| Metric | MVP Review | Week 7 | Change |
|--------|-----------|--------|--------|
| Single file read | 3.2 GB/s (warm) | N/A | — |
| CDC Throughput | Not measured | 3.0 GB/s | Baseline |
| Chunk Reuse | Not measured | 99.1% | Baseline |
| Mount time | ~8ms | N/A | — |
| stat() latency | 3ms | N/A | — |
MVP review focused on end-to-end FUSE operations. Week 7 review focuses on CDC/sync layer since remote origins add I/O wrappers, not CPU-bound logic.
---
## Test Details
```
Test Type: Synthetic microbenchmarks
Data Size: 10 MB (CDC), 64 KB × 10000 (hash), 2 MB (reuse)
Iterations: 10 (CDC), 10000 (hash), 1 (reuse)
Build: cargo build --release
Rust: stable (via nix develop)
```
### Benchmark Code
CDC and hash throughput measured with in-memory data to isolate algorithm performance from I/O. Chunk reuse measured with simulated metadata edit (100 bytes changed mid-file).
---
## Recommendations
### 1. Add Formal Benchmarks (Priority: Medium)
Current benchmarks are ad-hoc. Add criterion.rs for:
- Reproducible measurements with statistical analysis
- Regression detection in CI
- Historical tracking
```toml
[dev-dependencies]
criterion = "0.5"
```
### 2. Add Integration Benchmarks (Priority: Low)
Week 7 adds NFS/SMB wrappers. Add benchmarks for:
- ESTALE retry overhead
- Health check timeout behavior
- Connection pool performance (when S3/SFTP implemented)
### 3. Test with Real Network Origins (Priority: High for Week 8+)
Current benchmarks use local mounts. Before deploying:
- Benchmark against real NFS server
- Measure latency distribution (p50, p95, p99)
- Test failure scenarios (network partition, slow origin)
---
## Conclusion
**Week 7 introduces no performance regression.** The 4.5% CDC throughput variance is within noise margin. NFR-6.4 (>90% bandwidth reduction) continues to be exceeded at 99.1%.
Remote origin wrappers (NFS, SMB) are I/O-bound and will only affect performance when accessing remote storage. The retry logic (ESTALE, ENOTCONN) and health timeouts are error-path-only and have no impact on happy-path performance.
**All 102 tests pass with 0 warnings.**
---
## References
- [Requirements Specification](requirements.md) — NFR-2 (Throughput), NFR-6 (Network), NFR-7 (Availability)
- [MVP Performance Review](mvp-performance-review.md) — Baseline end-to-end measurements
- [Week 7 Plan](plans/week-07-remote-origins.md) — Remote origins implementation