From 53e6dbaef0c33555b7aeb11682eb2a052520bb4c Mon Sep 17 00:00:00 2001 From: Alexander Date: Sat, 27 Jun 2026 16:46:06 +0200 Subject: [PATCH] Remove old docs --- docs/api/search.md | 199 -- docs/api/smart-features.md | 315 --- docs/templates/bluedoc.md | 96 - docs/templates/greendoc.md | 56 - docs/v1/README.md | 118 -- docs/v1/analysis.md | 263 --- docs/v1/benchmark-plan.md | 403 ---- docs/v1/benchmark-results.md | 101 - docs/v1/components.md | 550 ------ docs/v1/data-flow.md | 412 ---- docs/v1/drawbacks.md | 479 ----- docs/v1/e2e-test-plan.md | 493 ----- docs/v1/features.md | 249 --- docs/v1/modernization.md | 459 ----- docs/v1/rust-migration.md | 451 ----- docs/v2/architecture.md | 1151 ----------- docs/v2/development-plan.md | 1387 -------------- .../metadata-enrichment-standalone.md | 579 ------ docs/v2/features/metadata-overlay.md | 1010 ---------- docs/v2/features/mkdir.md | 105 - docs/v2/features/mv.md | 94 - docs/v2/features/rm.md | 166 -- docs/v2/mvp-performance-review.md | 239 --- docs/v2/plans/logging-comprehensive.md | 982 ---------- docs/v2/plans/persistent-state-impl.md | 796 -------- docs/v2/plans/persistent-state.md | 353 ---- docs/v2/plans/phase-a-stop-dying.md | 569 ------ docs/v2/plans/phase-b-crash-recovery.md | 830 -------- docs/v2/plans/phase-c-hardening.md | 598 ------ docs/v2/plans/resilience-fault-tolerance.md | 1417 -------------- docs/v2/plans/resilience-testing.md | 1003 ---------- docs/v2/plans/week-01-foundation.md | 1126 ----------- docs/v2/plans/week-02-metadata.md | 771 -------- docs/v2/plans/week-03-virtual-tree.md | 1069 ----------- docs/v2/plans/week-04-cas-caching.md | 1089 ----------- docs/v2/plans/week-04b-origin-connector.md | 501 ----- docs/v2/plans/week-05-cdc-delta.md | 1219 ------------ docs/v2/plans/week-06-origin-federation.md | 1113 ----------- docs/v2/plans/week-07-remote-origins.md | 1261 ------------ docs/v2/plans/week-08-search-index.md | 1266 ------------- docs/v2/plans/week-09-smart-features.md | 1686 ----------------- docs/v2/plans/week-10-plugin-system.md | 179 -- docs/v2/plans/week-11-control-api.md | 539 ------ docs/v2/plans/week-12-external-metadata.md | 624 ------ docs/v2/plans/week-13-import-export.md | 699 ------- docs/v2/plans/week-14-extended-formats.md | 633 ------- docs/v2/requirements.md | 649 ------- docs/v2/week-07-performance-review.md | 179 -- 48 files changed, 30526 deletions(-) delete mode 100644 docs/api/search.md delete mode 100644 docs/api/smart-features.md delete mode 100644 docs/templates/bluedoc.md delete mode 100644 docs/templates/greendoc.md delete mode 100644 docs/v1/README.md delete mode 100644 docs/v1/analysis.md delete mode 100644 docs/v1/benchmark-plan.md delete mode 100644 docs/v1/benchmark-results.md delete mode 100644 docs/v1/components.md delete mode 100644 docs/v1/data-flow.md delete mode 100644 docs/v1/drawbacks.md delete mode 100644 docs/v1/e2e-test-plan.md delete mode 100644 docs/v1/features.md delete mode 100644 docs/v1/modernization.md delete mode 100644 docs/v1/rust-migration.md delete mode 100644 docs/v2/architecture.md delete mode 100644 docs/v2/development-plan.md delete mode 100644 docs/v2/features/metadata-enrichment-standalone.md delete mode 100644 docs/v2/features/metadata-overlay.md delete mode 100644 docs/v2/features/mkdir.md delete mode 100644 docs/v2/features/mv.md delete mode 100644 docs/v2/features/rm.md delete mode 100644 docs/v2/mvp-performance-review.md delete mode 100644 docs/v2/plans/logging-comprehensive.md delete mode 100644 docs/v2/plans/persistent-state-impl.md delete mode 100644 docs/v2/plans/persistent-state.md delete mode 100644 docs/v2/plans/phase-a-stop-dying.md delete mode 100644 docs/v2/plans/phase-b-crash-recovery.md delete mode 100644 docs/v2/plans/phase-c-hardening.md delete mode 100644 docs/v2/plans/resilience-fault-tolerance.md delete mode 100644 docs/v2/plans/resilience-testing.md delete mode 100644 docs/v2/plans/week-01-foundation.md delete mode 100644 docs/v2/plans/week-02-metadata.md delete mode 100644 docs/v2/plans/week-03-virtual-tree.md delete mode 100644 docs/v2/plans/week-04-cas-caching.md delete mode 100644 docs/v2/plans/week-04b-origin-connector.md delete mode 100644 docs/v2/plans/week-05-cdc-delta.md delete mode 100644 docs/v2/plans/week-06-origin-federation.md delete mode 100644 docs/v2/plans/week-07-remote-origins.md delete mode 100644 docs/v2/plans/week-08-search-index.md delete mode 100644 docs/v2/plans/week-09-smart-features.md delete mode 100644 docs/v2/plans/week-10-plugin-system.md delete mode 100644 docs/v2/plans/week-11-control-api.md delete mode 100644 docs/v2/plans/week-12-external-metadata.md delete mode 100644 docs/v2/plans/week-13-import-export.md delete mode 100644 docs/v2/plans/week-14-extended-formats.md delete mode 100644 docs/v2/requirements.md delete mode 100644 docs/v2/week-07-performance-review.md diff --git a/docs/api/search.md b/docs/api/search.md deleted file mode 100644 index f57578f..0000000 --- a/docs/api/search.md +++ /dev/null @@ -1,199 +0,0 @@ -# Search API Documentation - -## Overview - -MusicFS provides two search interfaces: -1. **FUSE Virtual Directory** - `/.search/query/` for file manager integration -2. **gRPC API** - `Search` and `SearchStream` RPCs for programmatic access (planned) - ---- - -## FUSE Search Interface - -### Endpoint: `/.search/{query}/` - -Browse search results as symlinks in a virtual directory. - -### Happy Path - -1. User navigates to `/.search/metallica/` -2. FUSE returns directory listing of symlinks -3. Each symlink points to absolute path: `/mnt/music/Metallica/Album/Track.flac` -4. User can open symlink directly in media player - -**Example:** -```bash -$ ls -la /mnt/musicfs/.search/metallica/ -001. Metallica - Enter Sandman.flac -> /mnt/musicfs/Metallica/Black Album/Enter Sandman.flac -002. Metallica - Battery.flac -> /mnt/musicfs/Metallica/Master of Puppets/Battery.flac -``` - -### Error Cases - -| Scenario | Behavior | FUSE Error | -|----------|----------|------------| -| Empty query | Empty directory | (none) | -| No results | Empty directory | (none) | -| Query too long (>256 chars) | Truncated | (none) | -| Invalid UTF-8 in query | EINVAL | `libc::EINVAL` | -| Index corrupted | ENOENT | `libc::ENOENT` | -| Index writer shutdown | EIO | `libc::EIO` | - -### Cache Behavior - -- Results cached for 5 minutes (TTL) -- Maximum 1000 cached queries (LRU eviction) -- Cache miss triggers tantivy query - ---- - -## gRPC Search API - -> **Note:** gRPC API is planned for implementation. See architecture docs for design. - -### `Search(SearchRequest) -> SearchResponse` - -Single request/response search. - -#### Request Schema - -```protobuf -message SearchRequest { - string query = 1; // Required: tantivy query string - optional uint32 limit = 2; // Default: 100, max: 10000 - optional uint32 offset = 3; // Default: 0, for pagination - optional string origin_id = 4; // Filter by origin (optional) -} -``` - -#### Response Schema - -```protobuf -message SearchResponse { - repeated SearchResult results = 1; - uint64 total_matches = 2; // Approximate total - uint32 query_time_ms = 3; // Query execution time -} - -message SearchResult { - int64 file_id = 1; - string virtual_path = 2; - optional string artist = 3; - optional string album = 4; - optional string title = 5; - float score = 6; // Relevance score - map highlights = 7; // Matched fragments -} -``` - -### Error Cases - -| Scenario | gRPC Status | Details | -|----------|-------------|---------| -| Empty query | `INVALID_ARGUMENT` | "Query cannot be empty" | -| Malformed query syntax | `INVALID_ARGUMENT` | tantivy parse error message | -| limit > 10000 | `INVALID_ARGUMENT` | "Limit exceeds maximum (10000)" | -| Index unavailable | `UNAVAILABLE` | "Search index not ready" | -| Index corrupted | `INTERNAL` | "Search index corrupted" | -| Timeout (>5s) | `DEADLINE_EXCEEDED` | Client-specified deadline | - ---- - -## Query Syntax - -MusicFS uses tantivy query syntax with custom fuzzy support. - -### Supported Operators - -| Operator | Example | Description | -|----------|---------|-------------| -| Term | `metallica` | Match in any default field | -| Field | `artist:metallica` | Match specific field | -| Phrase | `"enter sandman"` | Exact phrase match | -| Fuzzy | `metalica~1` | 1-character edit distance | -| Boolean | `metallica AND 1991` | Combine conditions | -| Range | `year:[1980 TO 1989]` | Numeric range | - -### Searchable Fields - -| Field | Type | Notes | -|-------|------|-------| -| `artist` | TEXT | Full-text searchable, default field | -| `album` | TEXT | Full-text searchable, default field | -| `album_artist` | TEXT | Full-text searchable, default field | -| `title` | TEXT | Full-text searchable, default field | -| `genre` | TEXT | Full-text searchable, default field | -| `composer` | TEXT | Full-text searchable, default field | -| `year` | u64 | Range queries only | - -### Fuzzy Query Implementation - -Fuzzy queries use the `term~N` syntax where N is the maximum edit distance (0-2). - -When a fuzzy query is detected: -1. Query is parsed to extract term and distance -2. `FuzzyTermQuery` is created for each default field -3. Results are combined with `BooleanQuery` (OR semantics) - -Example: `metalica~1` matches "Metallica" (edit distance 1). - ---- - -## Performance - -| Metric | Target | Notes | -|--------|--------|-------| -| Query latency (1M tracks) | <500ms | tantivy optimized | -| Index throughput | >1000 files/sec | Batch commits recommended | -| Memory per 1M tracks | <500MB | mmap-based index | - ---- - -## Architecture - -### Index Schema - -```rust -pub struct SearchSchema { - file_id: Field, // INDEXED | STORED - for deletion - virtual_path: Field, // STORED - symlink target - artist: Field, // TEXT | STORED - album: Field, // TEXT | STORED - album_artist: Field, // TEXT | STORED - title: Field, // TEXT | STORED - genre: Field, // TEXT | STORED - composer: Field, // TEXT | STORED - year: Field, // INDEXED | STORED - duration_ms: Field, // STORED - bitrate: Field, // STORED - sample_rate: Field, // STORED -} -``` - -### Writer Pattern - -Uses `Arc>` per tantivy best practices: -- `add_document()` and `delete_term()` require READ lock -- `commit()` requires WRITE lock -- Single writer, multiple concurrent indexers - -### Event Integration - -The `Indexer` subscribes to `EventBus` for: -- `FileAdded` - Index new file via `MetadataLookup` -- `FileRemoved` - Remove from index by file_id -- `FileModified` - Update index entry - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_search_basic` | Unit | Basic search returns results | -| `test_search_fuzzy` | Unit | Typo tolerance (FR-14.3) | -| `test_search_genre` | Unit | Field-specific search | -| `test_index_persistence` | Unit | Index survives restart | -| `test_remove_file` | Unit | Deletion works correctly | -| `test_index_batch` | Unit | Batch indexing via Indexer | -| `test_search_ops_*` | Unit | FUSE SearchOps integration | diff --git a/docs/api/smart-features.md b/docs/api/smart-features.md deleted file mode 100644 index 8fe5cdf..0000000 --- a/docs/api/smart-features.md +++ /dev/null @@ -1,315 +0,0 @@ -# Smart Features API Documentation - -## Overview - -MusicFS Week 9 introduces three intelligent features: -1. **Smart Collections** - Dynamic playlists based on queries, time ranges, and listening patterns -2. **Artwork Extraction & Caching** - Extract and serve album art in multiple sizes -3. **Predictive Prefetching** - Learn listening patterns to preload likely-next tracks - ---- - -## Smart Collections - -### CollectionStore - -Manages persistent smart collections using SQLite. - -```rust -pub struct CollectionStore { - db: rusqlite::Connection, -} - -pub struct Collection { - pub id: i64, - pub name: String, - pub query: CollectionQuery, - pub created_at: SystemTime, - pub updated_at: SystemTime, -} -``` - -### CollectionQuery Types - -| Query Type | Description | Example | -|------------|-------------|---------| -| `Match(String)` | tantivy search query | `"artist:Metallica"` | -| `DateRange { start, end }` | Files added within range | Last 30 days | -| `RecentlyAdded(days)` | Files added in last N days | `RecentlyAdded(7)` | -| `RecentlyPlayed(days)` | Files played in last N days | `RecentlyPlayed(30)` | -| `MostPlayed(limit)` | Top N most played tracks | `MostPlayed(100)` | -| `Genre(String)` | All tracks matching genre | `"Progressive Rock"` | -| `Compound(Vec)` | AND combination of queries | Multiple conditions | - -### API - -```rust -impl CollectionStore { - fn create(&self, name: &str, query: CollectionQuery) -> Result; - fn get(&self, id: i64) -> Result, CollectionError>; - fn list(&self) -> Result, CollectionError>; - fn update(&self, id: i64, name: &str, query: CollectionQuery) -> Result<(), CollectionError>; - fn delete(&self, id: i64) -> Result<(), CollectionError>; - fn evaluate(&self, id: i64, index: &SearchIndex, patterns: &PatternStore) -> Result, CollectionError>; -} -``` - -### FUSE Integration (Planned) - -Collections will appear as virtual directories under `/.collections/`: - -```bash -$ ls /mnt/musicfs/.collections/ -Recent Additions/ -Most Played/ -80s Metal/ - -$ ls /mnt/musicfs/.collections/Most\ Played/ -001. Track1.flac -> /mnt/musicfs/Artist/Album/Track1.flac -002. Track2.flac -> /mnt/musicfs/Artist/Album/Track2.flac -``` - ---- - -## Artwork Extraction & Caching - -### ArtworkExtractor - -Extracts embedded artwork from audio files. - -```rust -pub struct Artwork { - pub data: Vec, - pub mime_type: String, - pub art_type: ArtType, - pub width: u32, - pub height: u32, -} - -pub enum ArtType { - Front, - Back, - Other, -} - -pub enum ArtSize { - Thumbnail, // 150x150 max - Medium, // 300x300 max - Full, // Original size -} -``` - -### API - -```rust -impl ArtworkExtractor { - fn extract(&self, path: &Path) -> Result, ArtworkError>; - fn extract_first(&self, path: &Path) -> Result, ArtworkError>; - fn resize(data: &[u8], size: ArtSize) -> Result, ArtworkError>; -} -``` - -### ArtworkCache - -Caches artwork in CAS (Content-Addressable Storage). - -```rust -impl ArtworkCache { - async fn store(&self, file_id: i64, artwork: &Artwork) -> Result; - async fn get(&self, file_id: i64, art_type: &str, size: ArtSize) -> Result>, ArtworkError>; - async fn has(&self, file_id: i64, art_type: &str) -> Result; -} -``` - -### Size Specifications - -| Size | Max Dimension | Use Case | -|------|---------------|----------| -| Thumbnail | 150px | List views, grids | -| Medium | 300px | Detail panels | -| Full | Original | High-res display | - -### Caching Strategy - -1. Original artwork stored in CAS with content hash -2. SQLite maps `(file_id, art_type)` → `chunk_hash` -3. Resizing performed on-demand, not cached (saves storage) -4. Max input size: 10MB (reject larger images) - ---- - -## Predictive Prefetching - -### Access Patterns (PatternStore) - -Tracks file access history to predict next tracks. - -```rust -pub struct AccessPattern { - pub file_id: FileId, - pub timestamp: SystemTime, - pub context: AccessContext, - pub hour_of_day: u8, -} - -pub struct AccessContext { - pub album_id: Option, - pub track_number: Option, - pub artist: Option, -} -``` - -### Pattern Learning - -| Pattern Type | Description | Use Case | -|--------------|-------------|----------| -| Sequential | A → B → C transitions | Album playback | -| Time-based | Hour-of-day preferences | Morning playlist | -| Frequency | Most played tracks | Popular content | - -### API - -```rust -impl PatternStore { - fn record(&self, file_id: FileId, context: AccessContext) -> Result<(), PatternError>; - fn predict_next(&self, current: FileId, limit: usize) -> Vec; - fn predict_for_time(&self, hour: u8, limit: usize) -> Vec; - fn recently_played(&self, days: u32) -> Result, PatternError>; - fn most_played(&self, limit: u32) -> Result, PatternError>; -} -``` - -### PrefetchEngine - -Background engine that listens for file access events and prefetches predicted content. - -```rust -pub struct PrefetchConfig { - pub lookahead: usize, // How many tracks to prefetch (default: 3) - pub max_concurrent: usize, // Concurrent prefetch limit (default: 2) - pub cooldown: Duration, // Delay between prefetch bursts (default: 100ms) - pub enabled: bool, // Master switch -} -``` - -### Architecture - -``` -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ EventBus │────▶│ PrefetchEngine │────▶│ ContentFetcher │ -│ (FileAccessed) │ │ (predictions) │ │ (CAS storage) │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ - ▼ - ┌─────────────────┐ - │ PatternStore │ - │ (SQLite DB) │ - └─────────────────┘ -``` - -### FUSE Interface - -Virtual directory `/.prefetch/` exposes prefetch status and hints: - -```bash -$ cat /mnt/musicfs/.prefetch/status -MusicFS Prefetch Status -======================= -running: true -in_flight: 2 -most_played: [42, 57, 103, 89, 12] - -$ ls /mnt/musicfs/.prefetch/ -status -hint_0042 -hint_0057 -hint_0103 - -$ cat /mnt/musicfs/.prefetch/hint_0042 -57 -103 -89 -``` - ---- - -## Performance Targets - -| Feature | Metric | Target | -|---------|--------|--------| -| Collection evaluation | Latency | <100ms for 100k files | -| Artwork extraction | Throughput | >10 files/sec | -| Artwork resize | Latency | <50ms per image | -| Pattern prediction | Latency | <10ms | -| Prefetch hit rate | Accuracy | >70% for sequential play | - ---- - -## Error Handling - -### CollectionError - -| Error | Description | -|-------|-------------| -| `Database(rusqlite::Error)` | SQLite operation failed | -| `NotFound` | Collection ID doesn't exist | -| `InvalidQuery` | Query failed to serialize | -| `Search(SearchError)` | tantivy query failed | -| `Pattern(PatternError)` | Pattern lookup failed | - -### ArtworkError - -| Error | Description | -|-------|-------------| -| `Database(rusqlite::Error)` | Cache DB operation failed | -| `Cas(CasError)` | CAS storage operation failed | -| `InvalidHash` | Stored hash is malformed | -| `NotFound` | Artwork not in cache | -| `ImageTooLarge(usize)` | Input exceeds 10MB limit | -| `InvalidImage` | Cannot decode image data | -| `ResizeFailed` | Image resize operation failed | - -### PatternError - -| Error | Description | -|-------|-------------| -| `Database(rusqlite::Error)` | SQLite operation failed | - ---- - -## Configuration - -### Default Settings - -```toml -[prefetch] -enabled = true -lookahead = 3 -max_concurrent = 2 -cooldown_ms = 100 - -[artwork] -max_input_size_mb = 10 -thumbnail_size = 150 -medium_size = 300 - -[patterns] -max_history_days = 30 -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_collection_crud` | Unit | Create, read, update, delete | -| `test_collection_evaluate_match` | Unit | Match query evaluation | -| `test_collection_persistence` | Unit | Collections survive restart | -| `test_artwork_extract_flac` | Unit | FLAC artwork extraction | -| `test_artwork_cache_store_get` | Unit | Cache round-trip | -| `test_artwork_resize` | Unit | Resize produces valid output | -| `test_pattern_prediction` | Unit | Sequential pattern learning | -| `test_pattern_persistence` | Unit | Patterns survive restart | -| `test_prefetch_config_defaults` | Unit | Default config values | -| `test_prefetch_ops_*` | Unit | FUSE PrefetchOps integration | diff --git a/docs/templates/bluedoc.md b/docs/templates/bluedoc.md deleted file mode 100644 index c380ee1..0000000 --- a/docs/templates/bluedoc.md +++ /dev/null @@ -1,96 +0,0 @@ -# [Project Name]: Design Doc - -**Authors:** [Author Name(s)] -**Status:** [Draft / In-Review / Approved / Obsolete] -**Last Updated:** YYYY-MM-DD -**Reviewers:** [List of reviewers, usually @usernames] -**Approvers:** [List of final decision makers] -**Document Link:** [Link to this file or rendered version] - ---- - -## 1. Abstract -A high-level summary (1–3 paragraphs) of what the project is, what problem it solves, and the proposed solution. This should be readable by a non-expert. - -## 2. Background -Context for why this project exists. -- What is the current state? -- What are the pain points? -- Are there existing systems that this will replace or interact with? -- Include links to relevant PRDs (Product Requirement Documents) or previous design docs. - -## 3. Goals & Non-Goals -Clarity on scope is critical to prevent scope creep. - -### 3.1. Goals -* **Primary Goal:** The most important outcome. -* Metric-driven goals (e.g., "Reduce latency by 20%"). -* Functional requirements (e.g., "Allow users to edit comments"). - -### 3.2. Non-Goals -* Features that might seem related but are explicitly out of scope. -* Future improvements that are deferred. - -## 4. Proposed Design -The "meat" of the document. Start with the high-level architecture and zoom in. - -### 4.1. High-Level Architecture -Provide a high-level diagram or description of how the system fits together. -> *Tip: Use Mermaid.js or link to an embedded image.* - -### 4.2. Detailed Design -Go into specific components, APIs, and data models. -* **API Definitions:** Describe new endpoints, Protobuf definitions, or CLI commands. -* **Data Schema:** Database tables, key-value structures, or file formats. -* **Workflows:** Step-by-step logic for complex operations (e.g., auth flow). - -## 5. Cross-Cutting Concerns -Google design docs place heavy emphasis on these "standard" reviews. - -### 5.1. Security & Privacy -- How is data encrypted? -- What are the access control lists (ACLs)? -- Does this handle PII (Personally Identifiable Information)? - -### 5.2. Observability (Monitoring & Logging) -- What metrics will be exported (e.g., RPC error rates, latency)? -- What logging is required for debugging? -- What are the "Golden Signals" for the dashboard? - -### 5.3. Scalability & Performance -- What are the expected QPS (Queries Per Second)? -- How does the system scale (Horizontal vs. Vertical)? -- What are the resource requirements (CPU, RAM, Storage)? - -### 5.4. Testing Plan -- Unit tests, integration tests, and end-to-end tests. -- Strategy for load testing or "chaos" testing. - -## 6. Alternatives Considered -*A BlueDoc is not just about the chosen path, but why others were rejected.* -* **Alternative A:** Briefly describe it and why it was rejected (e.g., "Too complex," "High latency"). -* **Alternative B:** Why "Doing Nothing" is not an option. - -## 7. Implementation Plan -- **Phase 1:** Minimum Viable Product (MVP). -- **Phase 2:** Feature parity or migrations. -- **Rollout/Rollback:** How will the feature be toggled? (e.g., feature flags). - -## 8. Glossary / References -- Links to external libraries. -- Definitions for project-specific acronyms. - ---- - -### Markdown Style Tips (Google Conventions): -1. **Line Length:** Google's internal style guide suggests a soft limit of **80 characters** per line for source Markdown to make it easier to review in code-diff tools. -2. **Headings:** Use `#` for title, `##` for sections, and `###` for subsections. -3. **TOC:** If your environment supports it, use `[TOC]` at the top to generate a Table of Contents. -4. **Diagrams:** Use **Mermaid** blocks if using GitHub/GitLab, otherwise link to a stable SVG/PNG. - ```mermaid - graph TD; - A-->B; - A-->C; - B-->D; - C-->D; - ``` diff --git a/docs/templates/greendoc.md b/docs/templates/greendoc.md deleted file mode 100644 index 0011a0c..0000000 --- a/docs/templates/greendoc.md +++ /dev/null @@ -1,56 +0,0 @@ -# [Project Name]: Design One-Pager (GreenDoc) - -**Author(s):** [Name] -**Status:** [Draft / Approved / Shipped] -**Last Updated:** YYYY-MM-DD -**Estimated Effort:** [e.g., 2 weeks, 1 sprint] - ---- - -## 1. Summary -A 2–3 sentence overview of the change. What are you doing and why? - -## 2. Problem Statement -Describe the specific pain point or "broken" state this project addresses. -* *Example: Currently, users cannot filter their search history by date, leading to high latency in manual lookup.* - -## 3. Proposed Solution -Explain the high-level logic of the fix or feature. -* What is the specific code change or configuration update? -* How does it interact with existing systems? -* *Note: Use a single simple diagram if the logic is non-trivial.* - -## 4. Risks & Trade-offs -Even small changes have risks. Address them upfront. -* **Performance:** Will this increase memory usage? -* **Complexity:** Does this add a new dependency? -* **Backwards Compatibility:** Will this break existing clients? -* **Alternatives:** Why did you choose this over a "quicker" or "better" fix? - -## 5. Success Criteria (Metrics) -How will you know this worked? -* [ ] Primary metric (e.g., "Feature usage > 5%") -* [ ] Guardrail metric (e.g., "Latency does not increase by > 10ms") - -## 6. Implementation & Rollout -A brief bulleted list of the steps to ship. -1. Feature flag implementation. -2. Canary to 1% of users. -3. Full rollout. - ---- - -### Comparison: BlueDoc vs. GreenDoc - -| Feature | BlueDoc (Standard) | GreenDoc (One-Pager) | -| :--- | :--- | :--- | -| **Scope** | Major systems, new services. | Small features, optimizations, bug fixes. | -| **Length** | 5–20+ pages. | 1–2 pages. | -| **Review** | Cross-functional committees (SRE, Security). | Peer-level or Team Lead review. | -| **Focus** | Long-term scalability and architecture. | Immediate impact and implementation. | - -### Markdown Tips for GreenDocs: -* **Be Brutally Concise:** If a section requires more than three paragraphs, consider upgrading to a **BlueDoc**. -* **Checklists:** Use `[ ]` to show remaining work or requirements. -* **Inline Links:** Link directly to the relevant code files or bug tracker (Jira/Buganizer) to keep the doc self-contained. - diff --git a/docs/v1/README.md b/docs/v1/README.md deleted file mode 100644 index 7e7ca1a..0000000 --- a/docs/v1/README.md +++ /dev/null @@ -1,118 +0,0 @@ -# beetfs - Reverse Engineered Documentation - -> **Status**: Archived project (2010-2013), Python 2, fuse-python API -> **Fork**: git@github.com:LichHunter/beetfs.git -> **Original**: https://github.com/jbaiter/beetfs - -## Overview - -beetfs is a FUSE filesystem that presents audio files with **metadata from a database** while **passing through audio data unchanged** from original files. This enables transparent metadata modification without touching the underlying files. - -### The Core Concept - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ APPLICATION (VLC, Jellyfin, etc.) │ -│ │ -│ read("/mount/Artist/Album/track.flac") │ -└─────────────────────────────────┬───────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────┐ -│ beetfs (FUSE Layer) │ -│ ┌────────────────────────────────────────────────────────────────┐ │ -│ │ FileHandler │ │ -│ │ ┌──────────────────────────────────────────────────────────┐ │ │ -│ │ │ if offset < header_boundary: │ │ │ -│ │ │ return MODIFIED_HEADER (from beets database) │ │ │ -│ │ │ else: │ │ │ -│ │ │ return ORIGINAL_AUDIO (from real file on disk) │ │ │ -│ │ └──────────────────────────────────────────────────────────┘ │ │ -│ └────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────┘ - │ │ - ┌───────────┘ └───────────┐ - ▼ ▼ -┌───────────────────┐ ┌───────────────────┐ -│ Beets Database │ │ Original File │ -│ (SQLite - tags) │ │ (untouched) │ -│ │ │ │ -│ title: "Fixed" │ │ [FLAC header] │ -│ artist: "Corr" │ │ [Audio frames] │ -│ album: "Right" │ │ │ -└───────────────────┘ └───────────────────┘ -``` - -## Key Features - -| Feature | Description | -|---------|-------------| -| **Metadata Overlay** | Returns tags from database, not from file | -| **Audio Passthrough** | Original audio data served unchanged | -| **Write Interception** | Tag edits saved to database, not to file | -| **Virtual Organization** | Presents files in template-based directory structure | -| **Format Support** | FLAC (full), MP3 (partial - read-only) | - -## File Structure - -``` -beetfs/ -├── beetsplug/ -│ ├── __init__.py # Package initialization -│ └── beetFs.py # ALL code (~1144 lines) -├── README.rst # Original readme -└── COPYING # GPLv3 license -``` - -## Quick Architecture Summary - -| Component | Lines | Purpose | -|-----------|-------|---------| -| `beetFs` (plugin) | 188-191 | Beets plugin hook | -| `mount()` | 119-183 | CLI entry point, builds virtual tree | -| `FSNode` | 390-436 | Virtual directory tree node | -| `FileHandler` | 439-565 | **CORE**: Metadata interpolation | -| `InterpolatedFLAC` | 274-388 | FLAC header generation | -| `InterpolatedID3` | 200-271 | ID3 tag generation (incomplete) | -| `beetFileSystem` | 622-1144 | FUSE operations implementation | -| `Stat` | 568-619 | File stat structure | - -## Documentation Index - -1. **[Architecture Overview](./architecture.md)** - System design and component interaction -2. **[Components Deep Dive](./components.md)** - Detailed component analysis -3. **[Data Flow](./data-flow.md)** - Read/write operation flows -4. **[Performance Analysis](./analysis.md)** - Latency, memory footprint, I/O patterns -5. **[Drawbacks & Limitations](./drawbacks.md)** - Known issues and missing features -6. **[Modernization Guide](./modernization.md)** - Notes for updating to Python 3 - -## Critical Issues Summary - -| Issue | Severity | Impact | -|-------|----------|--------| -| Full file loaded into RAM | 🔴 Critical | OOM on large libraries | -| MP3 support disabled | 🔴 Critical | Only FLAC works | -| Python 2 only | 🔴 Critical | EOL, security risk | -| Single-threaded | 🟡 Major | Poor concurrency | -| 4 of 17 metadata fields | 🟡 Major | Limited functionality | - -See [drawbacks.md](./drawbacks.md) for complete list (27 identified issues). - -## Dependencies (Original) - -``` -beets >= 1.0 -fuse-python (Python 2 FUSE bindings) -mutagen (audio metadata library) -``` - -## Usage (Original) - -```bash -# As beets plugin -beet mount /path/to/mountpoint -``` - -## License - -GPLv3 - See COPYING file diff --git a/docs/v1/analysis.md b/docs/v1/analysis.md deleted file mode 100644 index 34771d6..0000000 --- a/docs/v1/analysis.md +++ /dev/null @@ -1,263 +0,0 @@ -# beetfs Performance Analysis - -## Executive Summary - -beetfs has significant performance limitations due to its 2010-era design assumptions. The primary issues are **full file loading into RAM** and **blocking I/O on file open**. - ---- - -## 1. Latency Analysis - -### Operation Latencies - -| Operation | Time Complexity | Typical Latency | Notes | -|-----------|-----------------|-----------------|-------| -| **File Open** | O(file_size) | 50ms - 1s+ | Reads entire file into memory | -| **File Read** | O(1) | <1ms | Pure memory slice | -| **File Write** | O(file_size) | 100ms - 2s+ | Reconstructs + DB write | -| **Directory List** | O(n) | <10ms | In-memory tree traversal | -| **getattr** | O(depth) | <1ms | Tree navigation + stat | - -### File Open Breakdown - -The file open operation is the critical bottleneck: - -``` -Time breakdown for opening 50MB FLAC file: -┌────────────────────────────────────────────────────────────┐ -│ 1. open() syscall │ ~1ms │ -│ 2. file_object.read() - load entire file │ ~100-200ms │ -│ 3. InterpolatedFLAC() - parse FLAC │ ~20-50ms │ -│ 4. Inject DB metadata │ ~1ms │ -│ 5. get_header() - generate new header │ ~10-20ms │ -│ 6. Seek to audio offset │ ~1ms │ -│ 7. Read audio into music_data │ ~100-200ms │ -├────────────────────────────────────────────────────────────┤ -│ TOTAL │ ~230-470ms │ -└────────────────────────────────────────────────────────────┘ -``` - -**Code Evidence** (lines 461-483): -```python -# Step 2-5: Load and parse entire file -self.inf = InterpolatedFLAC(self.file_object.read()) # FULL FILE READ -self.inf["title"] = self.item.title -# ... -self.header = self.inf.get_header(self.real_path) - -# Step 6-7: Cache all audio data -self.file_object.seek(self.music_offset) -self.music_data = self.file_object.read() # ANOTHER FULL READ -``` - -### Read Operation (Post-Open) - -After file is opened, reads are fast: - -```python -def read(self, size, offset): - if offset < self.bound: - return self.header[offset:offset+size] # Memory slice: O(1) - else: - return self.music_data[offset - len(self.header):...] # Memory slice: O(1) -``` - -### Write Operation - -Writes to header area trigger expensive reconstruction: - -``` -Time breakdown for tag write: -┌────────────────────────────────────────────────────────────┐ -│ 1. Reconstruct filedata in memory │ ~10-50ms │ -│ 2. Parse as InterpolatedFLAC │ ~20-50ms │ -│ 3. Extract tag values │ ~1ms │ -│ 4. lib.store() + lib.save() (SQLite) │ ~10-50ms │ -│ 5. Regenerate header │ ~10-20ms │ -├────────────────────────────────────────────────────────────┤ -│ TOTAL │ ~50-170ms │ -└────────────────────────────────────────────────────────────┘ -``` - ---- - -## 2. Memory Footprint - -### Per-File Memory Usage - -``` -┌─────────────────────────────────────────────────────────────────────┐ -│ FileHandler Memory Layout │ -├─────────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ self.music_data (bytes) │ │ -│ │ Size: file_size - original_header_size │ │ -│ │ Typical: 95-99% of file size │ │ -│ │ Example: 48.5 MB for 50 MB file │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ self.header (bytes) │ │ -│ │ Size: Generated FLAC header with DB metadata │ │ -│ │ Typical: 4 KB - 64 KB (depends on metadata + padding) │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ self.inf (InterpolatedFLAC) │ │ -│ │ Size: Parsed metadata blocks + internal state │ │ -│ │ Typical: 10 KB - 100 KB │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────────┐ │ -│ │ Other attributes │ │ -│ │ path, real_path, item reference, format, etc. │ │ -│ │ Typical: ~1 KB │ │ -│ └─────────────────────────────────────────────────────────────┘ │ -│ │ -├─────────────────────────────────────────────────────────────────────┤ -│ TOTAL per file: ~1.0x - 1.1x original file size │ -└─────────────────────────────────────────────────────────────────────┘ -``` - -### Memory Scaling - -| Scenario | Files Open | Avg File Size | RAM Usage | -|----------|------------|---------------|-----------| -| Single track playback | 1 | 30 MB | ~32 MB | -| Album playback (gapless) | 2-3 | 30 MB | ~65-100 MB | -| Album fully opened | 10 | 30 MB | ~320 MB | -| Jellyfin library scan | 50-100 | 30 MB | **1.6 - 3.2 GB** | -| Full library scan | 1000 | 30 MB | **32 GB** (OOM) | - -### Global Memory - -```python -# Directory tree structure -directory_structure = FSNode({}, {}) -# Memory: O(number_of_items) -# Typical: 1-10 MB for libraries with 10,000-100,000 tracks - -# Open file handles -self.files = {} # Dict[str, FileHandler] -# Memory: Sum of all FileHandler instances -# Unbounded - grows with concurrent opens -``` - ---- - -## 3. I/O Patterns - -### Current (Inefficient) - -``` -File Open: - Disk → [Read ALL] → RAM (music_data) - → RAM (inf object) - → RAM (header) - -File Read: - RAM (header or music_data) → Application - -Total I/O: 1x-2x file size on open, 0 on read -``` - -### Optimal (Not Implemented) - -``` -File Open: - Disk → [Read header only] → RAM (small) - -File Read: - If header region: - RAM (header) → Application - If audio region: - Disk → [Seek + Read chunk] → Application - -Total I/O: ~64KB on open, on-demand reads -``` - ---- - -## 4. Concurrency - -### Current Model - -```python -server.multithreaded = 0 # Single-threaded -``` - -**Implications:** -- All FUSE operations serialized -- One slow file open blocks everything -- No benefit from multi-core CPUs - -### Impact on Use Cases - -| Use Case | Impact | -|----------|--------| -| Single player (VLC) | Acceptable - one file at a time | -| Media server scan | Severe - sequential processing | -| Multiple clients | Severe - requests queue up | -| Concurrent reads | Moderate - reads are fast once open | - ---- - -## 5. Benchmarks (Theoretical) - -Based on code analysis, not actual measurements: - -### File Open Time vs Size - -``` -File Size Open Time (HDD) Open Time (SSD) -──────────────────────────────────────────────── - 10 MB 50-100 ms 20-50 ms - 30 MB 150-300 ms 50-100 ms - 50 MB 250-500 ms 100-200 ms - 100 MB 500-1000 ms 200-400 ms - 200 MB 1000-2000 ms 400-800 ms -``` - -### Memory vs Concurrent Opens - -``` -Open Files RAM Usage (30MB avg) -───────────────────────────────────── - 1 ~32 MB - 5 ~160 MB - 10 ~320 MB - 25 ~800 MB - 50 ~1.6 GB - 100 ~3.2 GB -``` - ---- - -## 6. Comparison with Alternatives - -| Metric | beetfs | Direct File | NFS | FUSE passthrough | -|--------|--------|-------------|-----|------------------| -| Open latency | 200-500ms | <10ms | 10-50ms | <10ms | -| Read latency | <1ms | <1ms | 1-10ms | <1ms | -| Memory/file | ~1x size | ~0 | ~0 | ~0 | -| Metadata source | Database | File | File | File | -| Modify original | No | Yes | Yes | Yes | - ---- - -## 7. Recommendations - -### For Current Usage - -1. **Limit concurrent opens** - Don't scan full library -2. **Use SSDs** - Reduces open latency by 2-3x -3. **Increase RAM** - Expect 1x file size per open -4. **Avoid large files** - 24-bit/192kHz FLACs are problematic - -### For Modernization - -1. **Implement lazy loading** - Read audio on demand -2. **Add file handle caching** - Keep headers, release audio -3. **Enable multi-threading** - Parallelize opens -4. **Add memory limits** - Evict old FileHandlers diff --git a/docs/v1/benchmark-plan.md b/docs/v1/benchmark-plan.md deleted file mode 100644 index a360ab3..0000000 --- a/docs/v1/benchmark-plan.md +++ /dev/null @@ -1,403 +0,0 @@ -# beetfs Benchmark Plan - -## Executive Summary - -Benchmark suite to measure beetfs FUSE filesystem performance across mount time, metadata operations, file I/O, and memory usage. Focus on realistic music library workloads. - -## Critical Performance Findings (Pre-Benchmark) - -### Architecture Bottlenecks Identified - -| Bottleneck | Location | Impact | -|------------|----------|--------| -| **Full file load into RAM** | `FileHandler.__init__` line 481 | 50-100MB per open FLAC | -| **Mount-time bulk load** | `mount()` line 143 | O(N) for N library items | -| **GIL serialization** | Python 2.7 | Single-core limit for metadata ops | -| **Per-file DB lookup** | `getattr()`, `access()` | SQLite query per stat call | - -### Expected Performance Characteristics - -| Operation | Expected Performance | Bottleneck | -|-----------|---------------------|------------| -| Mount (10K items) | 5-30 seconds | `lib.items()` + FSNode construction | -| readdir | Fast (in-memory dict) | None | -| getattr (file) | Slow (~1ms) | DB lookup + real file stat | -| open (first) | Very slow | Full file read into RAM | -| read | Fast | Memory-to-memory copy | -| Memory (10 open files) | 500MB-1GB | FileHandler caches entire files | - ---- - -## Benchmark Tools - -### Primary Tools - -| Tool | Purpose | Install | -|------|---------|---------| -| **fio** | I/O throughput, IOPS, latency | `nix-shell -p fio` | -| **mdtest** | Metadata operations (stat, readdir) | `nix-shell -p ior` | -| **hyperfine** | Mount time, command timing | `nix-shell -p hyperfine` | -| **time** | Basic timing | builtin | -| **/usr/bin/time -v** | Memory usage (maxrss) | builtin | - -### Measurement Scripts - -All benchmarks use synthetic FLAC files (5-10MB) to avoid I/O variance from real storage. - ---- - -## Benchmark Categories - -### 1. Mount Time Scaling - -**Goal**: Measure how mount time scales with library size. - -**Method**: -```bash -# Create libraries with N items: 100, 1K, 10K, 50K, 100K -hyperfine --warmup 1 --runs 5 \ - 'beet mount /mnt/beetfs && sleep 1 && fusermount -u /mnt/beetfs' -``` - -**Metrics**: -- Time to mount (seconds) -- Memory usage at mount completion (RSS) - -**Expected scaling**: O(N) - linear with library size - -**Test matrix**: -| Library Size | Expected Mount Time | Expected Memory | -|--------------|--------------------:|----------------:| -| 100 items | <1s | ~50MB | -| 1,000 items | 1-3s | ~60MB | -| 10,000 items | 5-15s | ~100MB | -| 50,000 items | 30-60s | ~300MB | -| 100,000 items | 60-120s | ~500MB | - ---- - -### 2. Metadata Operations (stat/readdir) - -**Goal**: Measure getattr and readdir performance - critical for music players that scan libraries. - -#### 2a. Single stat latency - -```bash -# Measure single stat call latency -hyperfine --warmup 10 --runs 100 \ - 'stat /mnt/beetfs/Artist/Album/01-Track.flac' -``` - -**Target**: <5ms average, <20ms p99 - -#### 2b. Bulk stat (library scan simulation) - -```bash -# Stat all files in library -hyperfine --warmup 1 --runs 5 \ - 'find /mnt/beetfs -type f -exec stat {} + > /dev/null' -``` - -**Metrics**: -- Total time for N files -- stat operations per second -- p50, p95, p99 latency - -**Target**: >500 stat/s (Python FUSE baseline) - -#### 2c. Directory listing - -```bash -# List directory with N entries -hyperfine --warmup 3 --runs 10 \ - 'ls /mnt/beetfs/Artist/Album/' -``` - -**Test matrix**: -| Directory entries | Target time | -|------------------:|------------:| -| 10 | <50ms | -| 100 | <100ms | -| 1,000 | <500ms | - ---- - -### 3. File Open Performance - -**Goal**: Measure file open latency - the critical bottleneck due to full file load. - -#### 3a. First open (cold) - -```bash -# Clear any caches, then open file -echo 3 > /proc/sys/vm/drop_caches -hyperfine --warmup 0 --runs 10 \ - 'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null' -``` - -**Test matrix**: -| File size | Expected open time | -|----------:|-------------------:| -| 5MB | 50-200ms | -| 20MB | 200-500ms | -| 50MB | 500ms-1s | -| 100MB | 1-2s | - -#### 3b. Cached open (warm) - -```bash -# File already opened once -hyperfine --warmup 5 --runs 50 \ - 'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null' -``` - -**Target**: <10ms (should hit FileHandler cache) - ---- - -### 4. Read Throughput - -**Goal**: Measure sequential and random read performance. - -#### 4a. Sequential read - -```bash -fio --name=seq_read \ - --filename=/mnt/beetfs/Artist/Album/01-Track.flac \ - --rw=read --bs=1M --direct=0 \ - --ioengine=sync --numjobs=1 \ - --runtime=30 --time_based -``` - -**Metrics**: MB/s throughput - -**Target**: >100 MB/s (memory-backed after first read) - -#### 4b. Random read (simulates seeking in audio player) - -```bash -fio --name=rand_read \ - --filename=/mnt/beetfs/Artist/Album/01-Track.flac \ - --rw=randread --bs=64k --direct=0 \ - --ioengine=sync --numjobs=1 \ - --runtime=30 --time_based -``` - -**Metrics**: IOPS, latency histogram - ---- - -### 5. Memory Usage - -**Goal**: Measure memory consumption under load. - -#### 5a. Idle memory (mounted, no activity) - -```bash -# Mount and measure RSS -beet mount /mnt/beetfs & -sleep 5 -ps -o rss= -p $(pgrep -f beetfs) -``` - -#### 5b. Memory per open file - -```bash -# Open N files, measure memory growth -for i in 1 5 10 20; do - # Open $i files simultaneously - cat /mnt/beetfs/Artist/Album/0{1..$i}*.flac > /dev/null & - ps -o rss= -p $(pgrep -f beetfs) -done -``` - -**Expected**: ~file_size × open_files (FileHandler caches entire file) - -#### 5c. Memory leak detection - -```bash -# Repeatedly open/close files, check for memory growth -for i in {1..100}; do - cat /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null -done -# Compare RSS before and after -``` - ---- - -### 6. Concurrent Access - -**Goal**: Measure performance under parallel access (multiple processes). - -```bash -# Parallel stat operations -hyperfine --warmup 1 --runs 5 \ - 'seq 1 100 | xargs -P 4 -I {} stat /mnt/beetfs/Artist/Album/0{}-Track.flac' -``` - -**Metrics**: -- Throughput scaling with parallelism (1, 2, 4, 8 workers) -- Latency degradation - -**Expected**: Limited scaling due to Python GIL - ---- - -### 7. Realistic Workloads - -#### 7a. Music player library scan - -Simulates: Rhythmbox/Clementine scanning library at startup - -```bash -# Recursive stat + readdir -time find /mnt/beetfs -type f -name "*.flac" -exec stat {} + | wc -l -``` - -#### 7b. Album playback - -Simulates: Playing 12-track album sequentially - -```bash -# Open each file, read 1MB (simulate buffering), close -for f in /mnt/beetfs/Artist/Album/*.flac; do - dd if="$f" of=/dev/null bs=1M count=1 2>/dev/null -done -``` - -#### 7c. Metadata edit - -Simulates: Editing tags in Picard/Kid3 - -```bash -# Open file, write to header region, close -# (Requires write support to be functional) -``` - ---- - -## Baseline Comparisons - -### Reference Filesystems - -| Filesystem | Purpose | -|------------|---------| -| **ext4 (local)** | Best-case baseline | -| **fuse-passthrough** | FUSE overhead baseline | -| **sshfs** | Network FUSE comparison | - -### Comparison Method - -Run identical benchmarks on: -1. Real music files on ext4 -2. Same files via FUSE passthrough -3. Same files via beetfs - -Calculate overhead: `(beetfs_time - ext4_time) / ext4_time × 100%` - ---- - -## Test Environment - -### Hardware Requirements - -- CPU: 4+ cores (to test GIL impact) -- RAM: 8+ GB (for large library tests) -- Storage: SSD recommended (reduces I/O variance) - -### Software Requirements - -```nix -# Add to flake.nix devShell -buildInputs = [ - fio - hyperfine - # ior # includes mdtest -]; -``` - -### Cache Control - -```bash -# Clear all caches before cold benchmarks -sync -echo 3 > /proc/sys/vm/drop_caches - -# Disable kernel FUSE caching for accurate measurements -mount -o entry_timeout=0,attr_timeout=0,negative_timeout=0 -``` - ---- - -## Success Criteria - -### Minimum Viable Performance - -| Metric | Minimum | Target | Excellent | -|--------|--------:|-------:|----------:| -| Mount time (10K items) | <60s | <15s | <5s | -| stat latency (avg) | <20ms | <5ms | <1ms | -| stat throughput | >100/s | >500/s | >2000/s | -| File open (50MB, cold) | <5s | <1s | <200ms | -| Read throughput | >50 MB/s | >200 MB/s | >500 MB/s | -| Memory (idle, 10K items) | <500MB | <100MB | <50MB | -| Memory per open file | <2× file size | <1.5× | <1.1× | - -### Regression Detection - -Any benchmark result >20% worse than baseline triggers investigation. - ---- - -## Implementation Notes - -### Test Data Generation - -Use existing test infrastructure from `tests/conftest.py`: -- `create_synthetic_flac()` - generates valid FLAC files -- `BeetFSTestCase` - creates isolated beets library - -### Benchmark Script Structure - -``` -beetfs/ -├── benchmarks/ -│ ├── run_all.sh # Master script -│ ├── bench_mount.sh # Mount time tests -│ ├── bench_metadata.sh # stat/readdir tests -│ ├── bench_io.sh # Read/write throughput -│ ├── bench_memory.sh # Memory profiling -│ └── results/ # Output directory -│ ├── mount_scaling.csv -│ ├── stat_latency.csv -│ └── ... -``` - -### Output Format - -```csv -# Example: mount_scaling.csv -library_size,mount_time_ms,memory_rss_kb,timestamp -100,450,52000,2024-01-15T10:30:00 -1000,2100,61000,2024-01-15T10:31:00 -10000,12500,98000,2024-01-15T10:33:00 -``` - ---- - -## Known Limitations - -1. **Python 2.7 GIL**: Cannot achieve true parallelism - expect flat scaling beyond 1 core -2. **FileHandler memory**: Each open file = full file in RAM - will OOM with many large files -3. **No lazy loading**: All library items loaded at mount - slow for large libraries -4. **SQLite single-writer**: Concurrent writes will serialize - -## Optimization Opportunities (Post-Benchmark) - -Based on benchmark results, consider: - -1. **Lazy FSNode construction** - Build tree on first access, not mount -2. **Memory-mapped file access** - mmap instead of full read -3. **LRU cache for FileHandler** - Evict old files instead of holding all -4. **Metadata caching** - Cache getattr results, invalidate on DB change -5. **Batch DB queries** - Prefetch metadata for directory listings diff --git a/docs/v1/benchmark-results.md b/docs/v1/benchmark-results.md deleted file mode 100644 index 969bc8d..0000000 --- a/docs/v1/benchmark-results.md +++ /dev/null @@ -1,101 +0,0 @@ -# beetfs Benchmark Results - -**Date**: 2026-05-12 -**Status**: ❌ ALL BENCHMARKS BLOCKED BY BUGS - -## Executive Summary - -Benchmarks cannot complete due to critical bugs in beetfs. The implementation is non-functional for any library with content. - -## Results - -| Benchmark | Status | Mean | Error | -|-----------|--------|------|-------| -| mount_time | ❌ FAIL | N/A | Directory tree building bug | -| readdir | ❌ FAIL | N/A | Directory tree building bug | -| stat_latency | ❌ FAIL | N/A | Directory tree building bug | -| enoent_lookup | ❌ FAIL | N/A | Directory tree building bug | -| file_open | ❌ FAIL | N/A | Directory tree building bug | -| read_throughput | ❌ FAIL | N/A | Directory tree building bug | -| memory_usage | ❌ FAIL | N/A | Directory tree building bug | - -## Blocking Bugs - -### Bug #1: Nested Methods (Lines 758-1144) - -All FUSE operations (`readdir`, `open`, `read`, `write`, etc.) are indented inside the `access()` method, making them local functions instead of class methods. - -**Impact**: Even if mount succeeds, all file operations return `ENOSYS (Function not implemented)`. - -**Fix Required**: Dedent lines 758-1144 by 8 spaces. - -### Bug #2: Directory Tree Building (Lines 403-414) - -`FSNode.adddir()` calls `getnode()` which assumes parent directories already exist. When building the tree for a new library, parent directories haven't been created yet. - -**Error**: -``` -KeyError: u'Bench Artist' - File "beetFs.py", line 403, in getnode - return self.getnode(elements, root=root.dirs[topdir]) -``` - -**Impact**: Mount crashes when library contains any tracks. - -**Fix Required**: `adddir()` must create parent directories recursively before adding child. - -### Bug #3: Empty Library Only - -The only working configuration is mounting with an empty beets library: -- `test_mount_empty_library`: ✅ PASS -- Any library with tracks: ❌ CRASH - -## Test Environment - -- **Python**: 2.7.15 -- **OS**: Linux (NixOS) -- **Test data**: 10 synthetic FLAC files (5 MB each) -- **Beets**: 1.4.9 - -## Benchmark Configuration - -```python -num_tracks = 10 -track_size_mb = 5 -mount_runs = 3 -stat_runs = 20 -readdir_runs = 10 -``` - -## Raw Results - -See `benchmarks/results/benchmark_results.json` for full JSON output. - -## Next Steps - -1. **Fix Bug #2** (directory tree building) - allows mount with content -2. **Fix Bug #1** (nested methods) - allows FUSE operations to work -3. **Re-run benchmarks** - get actual performance numbers - -## Conclusion - -**beetfs is currently non-functional** for real-world use. Both bugs must be fixed before performance can be measured. The test infrastructure and benchmark suite are ready; only the implementation needs repair. - ---- - -## Appendix: E2E Test Results (For Reference) - -From the e2e test suite (74 tests): - -| Category | Passed | Failed | Errors | -|----------|--------|--------|--------| -| Smoke tests | 4 | 3 | 0 | -| Nested bug detection | 3 (confirmed bug) | 10 | 0 | -| Readdir | 0 | 10 | 0 | -| Stat | 0 | 8 | 0 | -| Read | 0 | 11 | 0 | -| Write | 0 | 7 | 0 | -| Error handling | 0 | 7 | 3 | -| **Total** | **12** | **56** | **3** | - -The 12 passing tests are infrastructure tests and tests that verify the bugs exist. diff --git a/docs/v1/components.md b/docs/v1/components.md deleted file mode 100644 index 1a40fa0..0000000 --- a/docs/v1/components.md +++ /dev/null @@ -1,550 +0,0 @@ -# beetfs Components Deep Dive - -## Component Overview - -``` -┌─────────────────────────────────────────────────────────────────────────┐ -│ beetFs.py │ -│ ┌─────────────────────────────────────────────────────────────────────┐│ -│ │ PLUGIN LAYER ││ -│ │ beetFs (BeetsPlugin) beetFs_command (Subcommand) ││ -│ │ mount() template_mapping() ││ -│ └─────────────────────────────────────────────────────────────────────┘│ -│ ┌─────────────────────────────────────────────────────────────────────┐│ -│ │ VIRTUAL FILESYSTEM ││ -│ │ FSNode beetFileSystem (fuse.Fuse) ││ -│ │ Stat ││ -│ └─────────────────────────────────────────────────────────────────────┘│ -│ ┌─────────────────────────────────────────────────────────────────────┐│ -│ │ METADATA INTERPOLATION ││ -│ │ FileHandler InterpolatedFLAC ││ -│ │ InterpolatedID3 ││ -│ └─────────────────────────────────────────────────────────────────────┘│ -└─────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 1. Plugin Layer - -### 1.1 beetFs (BeetsPlugin) - -**Location**: Lines 188-191 - -```python -class beetFs(BeetsPlugin): - """ The beets plugin hook.""" - def commands(self): - return [beetFs_command] -``` - -**Purpose**: Registers beetfs as a beets plugin, exposing the `mount` subcommand. - -### 1.2 beetFs_command - -**Location**: Lines 47, 185 - -```python -beetFs_command = Subcommand('mount', help='Mount a beets filesystem') -beetFs_command.func = mount -``` - -**Purpose**: CLI subcommand definition for `beet mount`. - -### 1.3 mount() Function - -**Location**: Lines 119-183 - -```python -def mount(lib, config, opts, args): - # 1. Validate arguments - if not args: - raise beets.ui.UserError('no mountpoint specified') - - # 2. Parse path template - global structure_split - structure_split = PATH_FORMAT.split("/") - global structure_depth - structure_depth = len(structure_split) - - # 3. Store library reference - global library - library = lib - - # 4. Build virtual directory tree - global directory_structure - directory_structure = FSNode({}, {}) - - # 5. Iterate all library items - for item in lib.items(): - mapping = template_mapping(lib, item) - # ... build tree ... - directory_structure.addfile(sub_elements, filename, item.id) - - # 6. Create and run FUSE server - server = beetFileSystem(...) - server.main() -``` - -**Key Variables Set**: -| Variable | Type | Purpose | -|----------|------|---------| -| `structure_split` | `List[str]` | Path template components | -| `structure_depth` | `int` | Number of path levels | -| `library` | `Library` | Beets library reference | -| `directory_structure` | `FSNode` | Root of virtual tree | - -### 1.4 template_mapping() Function - -**Location**: Lines 82-116 - -```python -def template_mapping(lib, item): - """Builds a template substitution map from beets item.""" - mapping = {} - for key in METADATA_KEYS: - value = getattr(item, key) - # Sanitize value for filesystem paths - if isinstance(value, basestring): - value = re.sub(r'[\\/:]|^\.', '_', value) - elif key in ('track', 'tracktotal', 'disc', 'disctotal'): - value = '%02i' % value # Zero-pad numbers - mapping[key] = value - - # Add format info - format_ = os.path.splitext(item.path)[1][1:] - mapping['format'] = format_ - mapping['format_upper'] = format_.upper() - - # Default values for missing fields - if mapping['artist'] == '': - mapping['artist'] = 'Unknown Artist' - # ... etc - - return mapping -``` - -**Template Variables Available**: -| Variable | Source | Example | -|----------|--------|---------| -| `$artist` | `item.artist` | "Pink Floyd" | -| `$album` | `item.album` | "The Wall" | -| `$title` | `item.title` | "Comfortably Numb" | -| `$year` | `item.year` | "1979" | -| `$track` | `item.track` | "06" | -| `$format` | file extension | "flac" | -| `$format_upper` | file extension | "FLAC" | - ---- - -## 2. Virtual Filesystem Layer - -### 2.1 FSNode Class - -**Location**: Lines 390-436 - -```python -class FSNode(object): - """A directory node in the virtual filesystem tree.""" - - def __init__(self, dirs, files): - self.dirs = dirs # Dict[str, FSNode] - subdirectories - self.files = files # Dict[str, int] - filename → beets item ID -``` - -**Methods**: - -| Method | Purpose | Signature | -|--------|---------|-----------| -| `getnode()` | Navigate to nested node | `getnode(elements, root=None) → FSNode` | -| `adddir()` | Add a directory | `adddir(elements, directory, root=None)` | -| `addfile()` | Add a file entry | `addfile(elements, filename, id, root=None)` | -| `listdir()` | List contents | `listdir(elements, directories, root=None) → List[str]` | - -**Example Tree Navigation**: -```python -# Path: /Artist/Album/track.flac -# structure_split = ["$artist", "$album ($year) [$format_upper]", "$track - $artist - $title.$format"] - -elements = ["Artist", "Album (2020) [FLAC]"] -node = directory_structure.getnode(elements) -# node.files = {"01 - Artist - Track.flac": 42, ...} - -item_id = node.files["01 - Artist - Track.flac"] -# item_id = 42 -``` - -### 2.2 Stat Class - -**Location**: Lines 568-619 - -```python -class Stat(fuse.Stat): - DIRSIZE = 4096 - - def __init__(self, st_mode, st_size, st_nlink=1, st_uid=None, st_gid=None, - dt_atime=None, dt_mtime=None, dt_ctime=None): - self.st_mode = st_mode - self.st_ino = 0 - self.st_dev = 0 - self.st_nlink = st_nlink - self.st_uid = st_uid or os.getuid() - self.st_gid = st_gid or os.getgid() - self.st_size = st_size - # ... timestamps ... -``` - -**Purpose**: Represents file/directory metadata for FUSE stat operations. - -### 2.3 beetFileSystem Class - -**Location**: Lines 622-1144 - -```python -class beetFileSystem(fuse.Fuse): - """Main FUSE filesystem implementation.""" - - def __init__(self, *args, **kwargs): - logging.basicConfig(filename="LOG", level=logging.INFO) - super(beetFileSystem, self).__init__(*args, **kwargs) - - def fsinit(self): - """Called after filesystem is mounted.""" - self.lib = library - self.files = {} # Dict[path, FileHandler] -``` - -**FUSE Operations Implemented**: - -| Operation | Lines | Purpose | -|-----------|-------|---------| -| `fsinit()` | 630-636 | Post-mount initialization | -| `fsdestroy()` | 638-639 | Pre-unmount cleanup | -| `statfs()` | 641-646 | Filesystem statistics | -| `getattr()` | 648-707 | Get file/dir attributes | -| `access()` | 723-756 | Check permissions | -| `readdir()` | 931-975 | List directory contents | -| `open()` | 988-1021 | Open file | -| `read()` | 1077-1106 | Read file data | -| `write()` | 1108-1135 | Write file data | -| `release()` | 1049-1059 | Close file | - -**Not Implemented (return EOPNOTSUPP)**: -- `mknod()`, `mkdir()`, `unlink()`, `rmdir()` -- `symlink()`, `link()`, `rename()` -- `chmod()`, `chown()`, `truncate()` - ---- - -## 3. Metadata Interpolation Layer - -### 3.1 FileHandler Class - -**Location**: Lines 439-565 - -This is the **core component** that implements metadata overlay. - -```python -class FileHandler(object): - def __init__(self, path, lib): - self.path = path # Virtual path - self.lib = lib # Beets library - - # Resolve virtual path to real file - pathsplit = path[1:].split('/') - self.item = self.lib.get_item(id=directory_structure - .getnode(pathsplit[0:structure_depth-1]) - .files[pathsplit[structure_depth-1]]) - self.real_path = self.item.path - - # Open real file - self.file_object = open(self.real_path, 'r+') - self.instance_count = 1 - - # Determine format - self.format = os.path.splitext(path)[1][1:].lower() - - if self.format == "flac": - # Load file into interpolated FLAC object - self.inf = InterpolatedFLAC(self.file_object.read()) - - # INJECT DATABASE METADATA - self.inf["title"] = self.item.title - self.inf["album"] = self.item.album - self.inf["artist"] = self.item.artist - self.inf["genre"] = self.item.genre - - # Generate new header with DB metadata - self.header = self.inf.get_header(self.real_path) - self.bound = len(self.header) - self.music_offset = self.inf.offset() - - elif self.format == "mp3": - self.bound = 0 # MP3 interpolation disabled - self.music_offset = 0 - - # Cache audio data - self.file_object.seek(self.music_offset) - self.music_data = self.file_object.read() - self.file_object.close() -``` - -**Key Attributes**: - -| Attribute | Type | Purpose | -|-----------|------|---------| -| `path` | `str` | Virtual path (e.g., `/Artist/Album/track.flac`) | -| `real_path` | `str` | Actual file path on disk | -| `item` | `Item` | Beets library item (has DB metadata) | -| `format` | `str` | File format ("flac", "mp3") | -| `inf` | `InterpolatedFLAC` | Mutagen object with injected metadata | -| `header` | `bytes` | Generated header with DB tags | -| `bound` | `int` | Byte offset where header ends | -| `music_offset` | `int` | Byte offset in original file where audio starts | -| `music_data` | `bytes` | Cached audio data | -| `instance_count` | `int` | Reference count for file handles | - -### 3.2 FileHandler.read() Method - -**Location**: Lines 497-517 - -```python -def read(self, size, offset): - # Case 1: Reading within header boundary - if offset < self.bound: - if offset + size < len(self.header): - # Entire read is within header - return self.header[offset:offset+size] - else: - # Read spans header and audio - ret = self.header[offset:len(self.header)] - ret = ret + self.music_data[0:size - (len(self.header) - offset)] - return ret - - # Case 2: Reading audio data only - return self.music_data[offset - len(self.header):offset - len(self.header) + size] -``` - -**Read Logic Diagram**: - -``` -Virtual File Layout: -┌────────────────────────────────────────────────────────────────┐ -│ 0 bound EOF │ -│ ├─────────┼────────────────────────────────────────────────┤ │ -│ │ HEADER │ AUDIO DATA │ │ -│ │ (from │ (from self.music_data) │ │ -│ │ self. │ │ │ -│ │ header) │ │ │ -│ └─────────┴────────────────────────────────────────────────┘ │ -└────────────────────────────────────────────────────────────────┘ - -Read scenarios: -1. offset=0, size=100, bound=500 → Return header[0:100] -2. offset=400, size=200, bound=500 → Return header[400:500] + music[0:100] -3. offset=600, size=100, bound=500 → Return music[100:200] -``` - -### 3.3 FileHandler.write() Method - -**Location**: Lines 519-565 - -```python -def write(self, offset, buf): - # Only handle writes to header area - if offset < self.bound: - # Reconstruct full file in memory - filedata = self.header + self.music_data - - # Patch in new data - filedata = filedata[0:offset] + buf + filedata[offset + len(buf):] - - if self.format == "flac": - # Parse the patched data - self.inf = InterpolatedFLAC(filedata) - - # EXTRACT new tag values and save to DB - self.item.title = str(self.inf["title"][0]).encode('utf-8') - self.item.album = str(self.inf["album"][0]).encode('utf-8') - self.item.artist = str(self.inf["artist"][0]).encode('utf-8') - self.item.genre = str(self.inf["genre"][0]).encode('utf-8') - - # Persist to beets database - self.lib.store(self.item) - self.lib.save() - - # Regenerate header with updated values - self.inf["title"] = self.item.title - self.inf["album"] = self.item.album - self.inf["artist"] = self.item.artist - self.inf["genre"] = self.item.genre - - self.header = self.inf.get_header(self.real_path) - self.bound = len(self.header) - - return len(buf) -``` - -**Write Flow**: -``` -1. App writes new tag data to header region - │ - ▼ -2. Patch header + music_data with new bytes - │ - ▼ -3. Parse patched data as FLAC - │ - ▼ -4. Extract tag values from parsed FLAC - │ - ▼ -5. Update beets Item with new values - │ - ▼ -6. lib.store(item) + lib.save() → SQLite - │ - ▼ -7. Regenerate header for subsequent reads -``` - -### 3.4 InterpolatedFLAC Class - -**Location**: Lines 274-388 - -```python -class InterpolatedFLAC(FLAC): - """Custom FLAC handler that can load from bytes and generate headers.""" - - def load(self, filedata): - """Load FLAC from byte string instead of file.""" - self.metadata_blocks = [] - self.tags = None - self.filedata = filedata - self.fileobj = BytesIO(filedata) - self.__check_header(self.fileobj) - - while self.__read_metadata_block(self.fileobj): - pass - - # Verify audio frame starts correctly - if self.fileobj.read(2) not in ["\xff\xf8", "\xff\xf9"]: - raise FLACNoHeaderError("End of metadata did not start audio") - - def get_header(self, filename=None): - """Generate FLAC header with current metadata.""" - # Add padding block - self.metadata_blocks.append(Padding('\x00' * 1020)) - MetadataBlock.group_padding(self.metadata_blocks) - - # Calculate available space - header = self.__check_header(self.fileobj) - available = self.__find_audio_offset(self.fileobj) - header - data = MetadataBlock.writeblocks(self.metadata_blocks) - - # Adjust padding to match available space - if len(data) > available: - # Reduce padding - padding = self.metadata_blocks[-1] - padding.length -= (len(data) - available) - data = MetadataBlock.writeblocks(self.metadata_blocks) - elif len(data) < available: - # Increase padding - self.metadata_blocks[-1].length += (available - len(data)) - data = MetadataBlock.writeblocks(self.metadata_blocks) - - self.__offset = len("fLaC" + data) - return "fLaC" + data - - def offset(self): - """Return byte offset where audio data starts.""" - return self.__offset -``` - -**FLAC Structure**: -``` -┌──────────────────────────────────────────────────────────────────┐ -│ "fLaC" │ STREAMINFO │ VORBIS_COMMENT │ ... │ PADDING │ AUDIO... │ -│ (4B) │ block │ block │ │ block │ │ -└──────────────────────────────────────────────────────────────────┘ - │◄──────── metadata_blocks ─────────►│ - │ │ - └──── get_header() returns this ─────┘ -``` - -### 3.5 InterpolatedID3 Class - -**Location**: Lines 200-271 - -```python -class InterpolatedID3(ID3): - """Custom ID3 handler for MP3 files.""" - - def save(self, filename=None, v1=0): - """Save ID3 tags to file.""" - # Sort frames by importance - order = ["TIT2", "TPE1", "TRCK", "TALB", "TPOS", "TDRC", "TCON"] - # ... write header ... -``` - -**Note**: MP3 support is **incomplete** in the current implementation. The `FileHandler.__init__` sets `self.bound = 0` for MP3, effectively disabling interpolation. - ---- - -## 4. Supported Metadata Fields - -**Location**: Lines 55-77 - -```python -METADATA_RW_FIELDS = [ - ('title', 'text'), - ('artist', 'text'), - ('album', 'text'), - ('genre', 'text'), - ('composer', 'text'), - ('grouping', 'text'), - ('year', 'int'), - ('month', 'int'), - ('day', 'int'), - ('track', 'int'), - ('tracktotal', 'int'), - ('disc', 'int'), - ('disctotal', 'int'), - ('lyrics', 'text'), - ('comments', 'text'), - ('bpm', 'int'), - ('comp', 'bool'), -] -``` - -**Actually Implemented** (in FileHandler): -| Field | Read | Write | -|-------|------|-------| -| `title` | ✅ | ✅ | -| `artist` | ✅ | ✅ | -| `album` | ✅ | ✅ | -| `genre` | ✅ | ✅ | -| Others | ❌ | ❌ | - ---- - -## 5. Error Handling - -**Error Codes Used**: - -| Code | Constant | Usage | -|------|----------|-------| -| 2 | `ENOENT` | File/directory not found | -| 13 | `EACCES` | Permission denied | -| 1 | `EPERM` | Operation not permitted | -| 95 | `EOPNOTSUPP` | Operation not supported | - -**Exception Handling Pattern**: -```python -def getattr(self, path): - try: - # ... logic ... - except Exception as e: - logging.error(e) - return -errno.ENOENT -``` diff --git a/docs/v1/data-flow.md b/docs/v1/data-flow.md deleted file mode 100644 index 3e0e5c6..0000000 --- a/docs/v1/data-flow.md +++ /dev/null @@ -1,412 +0,0 @@ -# beetfs Data Flow - -## Overview - -This document details the complete data flow for read and write operations in beetfs. - ---- - -## 1. Initialization Flow - -``` -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beet mount /mountpoint │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ mount(lib, config, opts, args) │ -│ │ -│ 1. Parse PATH_FORMAT into structure_split │ -│ PATH_FORMAT = "$artist/$album ($year) [$format_upper]/..." │ -│ structure_split = ["$artist", "$album ($year) [$format_upper]", ...] │ -│ structure_depth = 3 │ -│ │ -│ 2. Store global library reference │ -│ library = lib │ -│ │ -│ 3. Create empty virtual directory tree │ -│ directory_structure = FSNode({}, {}) │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ for item in lib.items(): │ -│ │ -│ For each item in beets library: │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ 1. Build template mapping │ │ -│ │ mapping = { │ │ -│ │ 'artist': 'Pink Floyd', │ │ -│ │ 'album': 'The Wall', │ │ -│ │ 'year': '1979', │ │ -│ │ 'format_upper': 'FLAC', │ │ -│ │ 'track': '01', │ │ -│ │ 'title': 'In The Flesh?', │ │ -│ │ } │ │ -│ │ │ │ -│ │ 2. Substitute template for each level │ │ -│ │ level_subbed[0] = "Pink Floyd" │ │ -│ │ level_subbed[1] = "The Wall (1979) [FLAC]" │ │ -│ │ level_subbed[2] = "01 - Pink Floyd - In The Flesh?.flac" │ │ -│ │ │ │ -│ │ 3. Add directories to tree │ │ -│ │ directory_structure.adddir([], "Pink Floyd") │ │ -│ │ directory_structure.adddir(["Pink Floyd"], "The Wall (1979)...") │ │ -│ │ │ │ -│ │ 4. Add file entry (filename → item.id) │ │ -│ │ directory_structure.addfile( │ │ -│ │ ["Pink Floyd", "The Wall (1979) [FLAC]"], │ │ -│ │ "01 - Pink Floyd - In The Flesh?.flac", │ │ -│ │ item.id # e.g., 42 │ │ -│ │ ) │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem FUSE Server │ -│ │ -│ server = beetFileSystem(...) │ -│ server.multithreaded = 0 │ -│ server.main() ← Enters FUSE event loop │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 2. File Open Flow - -``` -Application: open("/mount/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The Flesh?.flac") - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem.open(path, flags) │ -│ Lines 988-1021 │ -│ │ -│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The..." │ -│ flags = os.O_RDONLY (or O_RDWR) │ -│ │ -│ if path in self.files: │ -│ # File already open - increment reference count │ -│ self.files[path].open() │ -│ return self.files[path] │ -│ else: │ -│ # Create new FileHandler │ -│ self.files[path] = FileHandler(path, self.lib) │ -│ return self.files[path] │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FileHandler.__init__(path, lib) │ -│ Lines 440-483 │ -│ │ -│ Step 1: Resolve virtual path to beets item │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ pathsplit = ["Pink Floyd", "The Wall (1979) [FLAC]", │ │ -│ │ "01 - Pink Floyd - In The Flesh?.flac"] │ │ -│ │ │ │ -│ │ # Navigate to parent directory in virtual tree │ │ -│ │ node = directory_structure.getnode(pathsplit[0:2]) │ │ -│ │ # node.files = {"01 - Pink Floyd - In The Flesh?.flac": 42, ...} │ │ -│ │ │ │ -│ │ # Get beets item by ID │ │ -│ │ item_id = node.files[pathsplit[2]] # 42 │ │ -│ │ self.item = lib.get_item(id=42) │ │ -│ │ self.real_path = self.item.path │ │ -│ │ # e.g., "/mnt/music/torrents/pink_floyd_wall.flac" │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 2: Open real file and detect format │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.file_object = open(self.real_path, 'r+') │ │ -│ │ self.format = "flac" # from file extension │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 3: Create InterpolatedFLAC with database metadata │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.inf = InterpolatedFLAC(self.file_object.read()) │ │ -│ │ │ │ -│ │ # INJECT DATABASE METADATA (this is the key operation!) │ │ -│ │ self.inf["title"] = self.item.title # "In The Flesh?" │ │ -│ │ self.inf["album"] = self.item.album # "The Wall" │ │ -│ │ self.inf["artist"] = self.item.artist # "Pink Floyd" │ │ -│ │ self.inf["genre"] = self.item.genre # "Progressive Rock" │ │ -│ │ │ │ -│ │ # Generate header with injected metadata │ │ -│ │ self.header = self.inf.get_header(self.real_path) │ │ -│ │ self.bound = len(self.header) # e.g., 8192 bytes │ │ -│ │ self.music_offset = self.inf.offset() │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 4: Cache audio data │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.file_object.seek(self.music_offset) │ │ -│ │ self.music_data = self.file_object.read() # All audio data │ │ -│ │ self.file_object.close() │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 3. File Read Flow - -``` -Application: read(fd, buffer, 4096) # offset managed by kernel - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem.read(path, size, offset, fh) │ -│ Lines 1077-1106 │ -│ │ -│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - ..." │ -│ size = 4096 │ -│ offset = 0 (first read) or previous offset + bytes_read │ -│ fh = FileHandler instance │ -│ │ -│ return self.files[path].read(size, offset) │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FileHandler.read(size, offset) │ -│ Lines 497-517 │ -│ │ -│ Variables: │ -│ self.bound = 8192 (header size) │ -│ self.header = bytes (generated FLAC header with DB metadata) │ -│ self.music_data = bytes (original audio frames) │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ┌───────────────────────┼───────────────────────┐ - │ │ │ - ▼ ▼ ▼ -┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐ -│ Case 1: Header Only │ │ Case 2: Span Both │ │ Case 3: Audio Only │ -│ offset < bound │ │ offset < bound │ │ offset >= bound │ -│ offset+size < bound │ │ offset+size >= bound│ │ │ -├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤ -│ Example: │ │ Example: │ │ Example: │ -│ offset=0 │ │ offset=8000 │ │ offset=10000 │ -│ size=4096 │ │ size=4096 │ │ size=4096 │ -│ bound=8192 │ │ bound=8192 │ │ bound=8192 │ -├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤ -│ Return: │ │ Return: │ │ Return: │ -│ header[0:4096] │ │ header[8000:8192] │ │ music_data[ │ -│ │ │ + music_data[0:3904]│ │ 1808:5904] │ -│ (DB metadata!) │ │ │ │ │ -│ │ │ (mixed) │ │ (original audio) │ -└─────────────────────┘ └─────────────────────┘ └─────────────────────┘ - - -Visual representation of virtual file: - - 0 bound (8192) EOF - │ │ │ - ▼ ▼ ▼ - ┌───────────────────────┬────────────────────────────────────────────┐ - │ HEADER │ AUDIO DATA │ - │ (self.header) │ (self.music_data) │ - │ │ │ - │ Contains: │ Contains: │ - │ - "fLaC" magic │ - Original FLAC frames │ - │ - STREAMINFO block │ - Unchanged from disk │ - │ - VORBIS_COMMENT │ │ - │ with DB values: │ │ - │ title, artist, │ │ - │ album, genre │ │ - │ - PADDING block │ │ - └───────────────────────┴────────────────────────────────────────────┘ - ▲ ▲ - │ │ - From InterpolatedFLAC From original file - with injected DB tags (passed through) -``` - ---- - -## 4. File Write Flow - -``` -Application: write(fd, "TITLE=New Title\0", 16) # Hypothetical tag edit - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem.write(path, buf, offset, fh) │ -│ Lines 1108-1135 │ -│ │ -│ return self.files[path].write(offset, buf) │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FileHandler.write(offset, buf) │ -│ Lines 519-565 │ -│ │ -│ if offset >= self.bound: │ -│ # Write is in audio area - DISCARD │ -│ return # Do nothing, audio is read-only │ -│ │ -│ # Write is in header area - process tag update │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ Step 1: Reconstruct full virtual file in memory │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ filedata = self.header + self.music_data │ │ -│ │ │ │ -│ │ # Patch in new data │ │ -│ │ filedata = filedata[0:offset] + buf + filedata[offset + len(buf):] │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 2: Parse patched data as FLAC │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.inf = InterpolatedFLAC(filedata) │ │ -│ │ # This parses the FLAC structure and extracts Vorbis comments │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 3: Extract tag values from parsed FLAC │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.item.title = str(self.inf["title"][0]).encode('utf-8') │ │ -│ │ self.item.album = str(self.inf["album"][0]).encode('utf-8') │ │ -│ │ self.item.artist = str(self.inf["artist"][0]).encode('utf-8') │ │ -│ │ self.item.genre = str(self.inf["genre"][0]).encode('utf-8') │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 4: Save to beets database │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.lib.store(self.item) # Update item in library │ │ -│ │ self.lib.save() # Persist to SQLite │ │ -│ │ │ │ -│ │ # NOTE: Original file on disk is NEVER touched! │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ Step 5: Regenerate header for subsequent reads │ -│ ┌───────────────────────────────────────────────────────────────────────┐ │ -│ │ self.inf["title"] = self.item.title │ │ -│ │ self.inf["album"] = self.item.album │ │ -│ │ self.inf["artist"] = self.item.artist │ │ -│ │ self.inf["genre"] = self.item.genre │ │ -│ │ │ │ -│ │ self.header = self.inf.get_header(self.real_path) │ │ -│ │ self.bound = len(self.header) │ │ -│ └───────────────────────────────────────────────────────────────────────┘ │ -│ │ -│ return len(buf) # Success │ -└─────────────────────────────────────────────────────────────────────────────┘ - - -Write data flow summary: - - ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ - │ Application │ │ beetfs │ │ Beets │ │ Original │ - │ writes │────▶│ parses │────▶│ database │ │ file │ - │ new tags │ │ extracts │ │ updated │ │ UNTOUCHED │ - └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ -``` - ---- - -## 5. File Release Flow - -``` -Application: close(fd) - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem.release(path, flags, fh) │ -│ Lines 1049-1059 │ -│ │ -│ if self.files[path].release(): │ -│ # Reference count reached 0, clean up │ -│ del self.files[path] │ -└─────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ FileHandler.release() │ -│ Lines 489-495 │ -│ │ -│ self.instance_count -= 1 │ -│ │ -│ if self.instance_count == 0: │ -│ return True # OK to delete │ -│ else: │ -│ return False # Still in use │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 6. Directory Listing Flow - -``` -Application: ls /mount/Pink\ Floyd/ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────┐ -│ beetFileSystem.readdir(path, offset, dh) │ -│ Lines 931-975 │ -│ │ -│ path = "/Pink Floyd" │ -│ pathsplit = ["Pink Floyd"] │ -│ │ -│ yield fuse.Direntry(".") │ -│ yield fuse.Direntry("..") │ -│ │ -│ # len(pathsplit) == 1, structure_depth - 1 == 2 │ -│ # So we're listing directories (albums), not files │ -│ │ -│ for dirname in directory_structure.listdir(pathsplit, True): │ -│ yield fuse.Direntry(dirname.encode('utf-8')) │ -│ # "The Wall (1979) [FLAC]" │ -│ # "Animals (1977) [FLAC]" │ -│ # etc. │ -└─────────────────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 7. Complete Request Lifecycle - -``` -┌──────────────────────────────────────────────────────────────────────────────┐ -│ COMPLETE LIFECYCLE │ -│ │ -│ 1. User mounts: beet mount /mnt/music │ -│ ├─ Build virtual tree from beets library │ -│ └─ Start FUSE event loop │ -│ │ -│ 2. Application opens file: open("/mnt/music/Artist/Album/track.flac") │ -│ ├─ Resolve virtual path to beets item ID │ -│ ├─ Load original file into memory │ -│ ├─ Inject database metadata into FLAC structure │ -│ ├─ Generate new header with DB tags │ -│ └─ Cache audio data │ -│ │ -│ 3. Application reads file: read(fd, buf, 4096) │ -│ ├─ If reading header region → return header (DB metadata) │ -│ ├─ If reading audio region → return cached audio (original) │ -│ └─ If spanning both → return combined data │ -│ │ -│ 4. Application writes tags: write(fd, new_tags, offset) │ -│ ├─ If audio region → discard (read-only) │ -│ ├─ If header region: │ -│ │ ├─ Parse new tag values │ -│ │ ├─ Update beets database │ -│ │ └─ Regenerate header │ -│ └─ Original file NEVER modified │ -│ │ -│ 5. Application closes file: close(fd) │ -│ ├─ Decrement reference count │ -│ └─ Clean up if count == 0 │ -│ │ -│ 6. User unmounts: fusermount -u /mnt/music │ -│ └─ fsdestroy() called, cleanup │ -│ │ -└──────────────────────────────────────────────────────────────────────────────┘ -``` diff --git a/docs/v1/drawbacks.md b/docs/v1/drawbacks.md deleted file mode 100644 index 6741aaf..0000000 --- a/docs/v1/drawbacks.md +++ /dev/null @@ -1,479 +0,0 @@ -# beetfs Drawbacks & Limitations - -## Overview - -This document catalogs all identified issues, limitations, and missing features in beetfs. Issues are categorized by severity and type. - ---- - -## Critical Issues (🔴) - -### 1. Full File Loading into Memory - -**Location**: Lines 463, 480-481 - -```python -self.inf = InterpolatedFLAC(self.file_object.read()) # Entire file -# ... -self.music_data = self.file_object.read() # Audio portion again -``` - -**Impact**: -- Memory usage = O(file_size) per open file -- 50MB FLAC = ~50MB RAM -- Library scan of 100 files = 5GB+ RAM -- Out-of-memory crashes on large libraries - -**Fix Required**: Implement lazy loading with seek-based reads. - ---- - -### 2. MP3 Support Disabled - -**Location**: Lines 475-477 - -```python -elif self.format == "mp3": - self.bound = 0 # disable interpolation for now - self.music_offset = 0 # disable interpolation for now -``` - -**Impact**: -- MP3 files return original metadata, not database metadata -- Breaks the core promise of metadata overlay -- MP3 is still one of the most common formats - -**Fix Required**: Implement `InterpolatedID3` header generation. - ---- - -### 3. Python 2 Only - -**Location**: Throughout - -```python -except fuse.FuseError, e: # Python 2 syntax -if isinstance(value, basestring): # Removed in Python 3 -return reduce(lambda a, b: (a << 8) + ord(b), string, 0L) # Long literals -``` - -**Impact**: -- Python 2 EOL was January 2020 -- Security vulnerabilities unfixed -- No modern library support -- Cannot run on Python 3 without migration - -**Fix Required**: Full Python 3 migration (see modernization.md). - ---- - -### 4. Deprecated FUSE Library - -**Location**: Line 25, 51 - -```python -import fuse -fuse.fuse_python_api = (0, 2) -``` - -**Impact**: -- fuse-python is unmaintained -- Missing modern FUSE features (FUSE 3.x) -- Compatibility issues with recent kernels -- No async support - -**Fix Required**: Migrate to pyfuse3 or llfuse. - ---- - -### 5. Single-Threaded Execution - -**Location**: Line 178 - -```python -server.multithreaded = 0 -``` - -**Impact**: -- All operations serialized -- One slow open blocks all other operations -- Cannot utilize multiple CPU cores -- Poor performance under concurrent access - -**Fix Required**: Enable multithreading with proper locking. - ---- - -## Major Issues (🟡) - -### 6. Limited Metadata Fields - -**Location**: Lines 466-469, 540-547 - -```python -# Only these 4 fields are actually used: -self.inf["title"] = self.item.title -self.inf["album"] = self.item.album -self.inf["artist"] = self.item.artist -self.inf["genre"] = self.item.genre -``` - -**Defined but not implemented** (lines 55-77): -- `composer`, `grouping` -- `year`, `month`, `day` -- `track`, `tracktotal` -- `disc`, `disctotal` -- `lyrics`, `comments` -- `bpm`, `comp` -- `albumartist` (not even defined) - -**Impact**: -- Track numbers not from database -- Album artist not supported -- Year/date not interpolated -- Cover art not handled - ---- - -### 7. No File Handle Caching/Eviction - -**Location**: Lines 1004-1018 - -```python -if path in self.files: - self.files[path].open() -else: - self.files[path] = FileHandler(path, self.lib) -``` - -**Missing**: -- No maximum cache size -- No LRU eviction -- No memory pressure handling -- Files stay in memory until explicitly closed - -**Impact**: -- Memory grows unbounded -- No protection against OOM -- Applications that open-then-close still leave data cached - ---- - -### 8. Blocking Database Operations - -**Location**: Lines 549-550 - -```python -self.lib.store(self.item) -self.lib.save() -``` - -**Impact**: -- SQLite operations in FUSE thread -- Write operations block all reads -- No transaction batching -- Potential deadlocks with beets - ---- - -### 9. No Library Hot Reload - -**Issue**: Virtual directory tree built once at mount time. - -**Location**: Lines 142-172 - -```python -for item in lib.items(): - # Build tree... -``` - -**Impact**: -- New files added to beets library not visible -- Deleted files still appear (ENOENT on access) -- Metadata changes in beets not reflected until remount -- Must unmount/remount to see changes - ---- - -### 10. Static Path Format - -**Location**: Lines 44-45 - -```python -PATH_FORMAT = ("$artist/$album ($year) [$format_upper]/" - "$track - $artist - $title.$format") -``` - -**Impact**: -- Cannot customize organization -- Hard-coded template -- No configuration option -- Incompatible with different organizational preferences - ---- - -### 11. No Extended Attribute Support - -**Location**: Not implemented - -**Impact**: -- Cannot store/retrieve xattrs -- Some applications use xattrs for metadata -- macOS Finder metadata lost -- Linux capabilities not supported - ---- - -### 12. No Symlink Support - -**Location**: Lines 758-765 - -```python -def readlink(self, path): - return -errno.EOPNOTSUPP -``` - -**Impact**: -- Cannot create symlinks in mount -- Some applications expect symlink support -- Cannot link to external files - ---- - -### 13. Silent Error Swallowing - -**Location**: Lines 705-707, 1019-1021, 1103-1104 - -```python -except Exception as e: - logging.error(e) - return -errno.ENOENT # Always returns same error -``` - -**Impact**: -- All errors appear as "file not found" -- Hard to debug issues -- No distinction between permission, I/O, parse errors -- Lost stack traces in many cases - ---- - -## Minor Issues (🟢) - -### 14. Global State - -**Location**: Lines 125-140 - -```python -global structure_split -global structure_depth -global library -global directory_structure -``` - -**Impact**: -- Cannot mount multiple instances -- Difficult to unit test -- Tight coupling between components -- No dependency injection - ---- - -### 15. Hard-coded Log File - -**Location**: Lines 624-625 - -```python -LOG_FILENAME = "LOG" -logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO,) -``` - -**Impact**: -- Log file created in current directory -- No log rotation -- No configurable log level -- Fills disk on busy systems - ---- - -### 16. Reference Count Manual Management - -**Location**: Lines 485-495 - -```python -def open(self): - self.instance_count = self.instance_count + 1 - -def release(self): - if self.instance_count > 0: - self.instance_count = self.instance_count - 1 -``` - -**Issues**: -- Race conditions possible if multithreaded -- No context manager support -- Manual counting error-prone -- Off-by-one potential - ---- - -### 17. Inefficient Directory Building - -**Location**: Lines 153-172 - -```python -for level in range(0, structure_depth - 1): - if level-1 in level_subbed: - sub_elements.append(level_subbed[level-1]) - directory_structure.adddir(sub_elements, level_subbed[level]) -``` - -**Issues**: -- Rebuilds path for every item -- O(items × depth) complexity -- String allocations in inner loop -- Could use trie-based insertion - ---- - -### 18. No Cover Art Handling - -**Issue**: Cover art embedded in FLAC not addressed. - -**Impact**: -- Cover art from original file used, not database -- Cannot replace/add cover art through overlay -- PICTURE metadata blocks passed through unchanged - ---- - -### 19. No Cue Sheet Support - -**Issue**: Cue sheets not handled specially. - -**Impact**: -- `.cue` files point to original file paths -- Cannot play cue-referenced tracks correctly -- Split-by-cue not supported - ---- - -### 20. File Size Mismatch Potential - -**Issue**: Virtual file size differs from physical if header size changes. - -**Location**: Lines 675-688 - -```python -statinfo = os.stat(item) -st = Stat(st_mode=statinfo.st_mode, - st_size=statinfo.st_size, # Original size, not virtual! - ...) -``` - -**Impact**: -- `stat()` returns original file size -- If generated header is larger/smaller, size is wrong -- Some applications may fail on size mismatch -- Range requests could break - ---- - -## Missing Features - -### Essential - -| Feature | Status | Notes | -|---------|--------|-------| -| MP3 metadata interpolation | ❌ Disabled | Code exists but disabled | -| OGG/Opus support | ❌ Missing | No implementation | -| AAC/M4A support | ❌ Missing | No implementation | -| Lazy file loading | ❌ Missing | Full file loaded | -| Memory management | ❌ Missing | No limits or eviction | -| Configuration file | ❌ Missing | Hard-coded values | - -### Nice to Have - -| Feature | Status | Notes | -|---------|--------|-------| -| Cover art interpolation | ❌ Missing | Would need PICTURE block handling | -| ReplayGain from database | ❌ Missing | Tags not interpolated | -| Lyrics from database | ❌ Missing | Listed in fields, not implemented | -| Watch mode (hot reload) | ❌ Missing | No inotify integration | -| Multiple mount points | ❌ Missing | Global state prevents | -| Remote database | ❌ Missing | Local beets only | -| Read-only mode | ❌ Missing | Always allows writes | -| Custom path templates | ❌ Missing | Hard-coded PATH_FORMAT | - ---- - -## Security Considerations - -### 1. No Input Validation - -**Location**: Throughout - -```python -pathsplit = path[1:].split('/') -item_id = node.files[pathsplit[structure_depth-1]] # No bounds check -``` - -**Risk**: Path traversal, injection attacks unlikely but possible. - -### 2. Database Credentials Exposed - -**Issue**: Uses beets library directly with stored credentials. - -**Risk**: Low - local access only. - -### 3. No Permission Enforcement - -**Location**: Lines 749-756 - -```python -if flags | os.R_OK: - pass # TODO: actually check the file permissions -if flags | os.W_OK: - pass -``` - -**Risk**: All users can read/write through mount. - ---- - -## Compatibility Issues - -| Component | Issue | -|-----------|-------| -| **Jellyfin** | May scan entire library, causing OOM | -| **Plex** | Same library scan issue | -| **Navidrome** | Expects certain tag fields not implemented | -| **mpd** | Works for playback, database features limited | -| **macOS** | fuse-python macOS support questionable | -| **Docker** | FUSE in containers requires privileged mode | - ---- - -## Summary Table - -| Category | Critical | Major | Minor | -|----------|----------|-------|-------| -| Performance | 2 | 4 | 2 | -| Functionality | 2 | 5 | 4 | -| Code Quality | 2 | 2 | 4 | -| **Total** | **6** | **11** | **10** | - ---- - -## Prioritized Fix List - -1. 🔴 **Memory**: Implement lazy loading (Critical for usability) -2. 🔴 **Python 3**: Migrate to Python 3 (Required for any changes) -3. 🔴 **FUSE lib**: Switch to pyfuse3/llfuse (Required for Python 3) -4. 🔴 **MP3**: Enable MP3 interpolation (Core functionality) -5. 🟡 **Metadata**: Implement all fields (Feature completeness) -6. 🟡 **Threading**: Enable multithreading (Performance) -7. 🟡 **Config**: Add configuration file (Usability) -8. 🟡 **Hot reload**: Watch for library changes (Usability) -9. 🟢 **Globals**: Remove global state (Code quality) -10. 🟢 **Logging**: Configurable logging (Operations) diff --git a/docs/v1/e2e-test-plan.md b/docs/v1/e2e-test-plan.md deleted file mode 100644 index 1c5b2b3..0000000 --- a/docs/v1/e2e-test-plan.md +++ /dev/null @@ -1,493 +0,0 @@ -# beetfs E2E Test Plan - -> **Reviewed by Oracle** - Critical bug discovered, plan updated accordingly - -## Test Results (Latest Run) - -``` -Tests run: 74 -Passed: 12 -Failures: 56 -Errors: 3 -Skipped: 3 -Duration: ~103 seconds -``` - -### Bugs Detected by Tests - -| Bug | Tests Affected | Description | -|-----|----------------|-------------| -| **Nested Methods** | 56 | Lines 758-1144 indented inside `access()` - FUSE operations unreachable | -| **Directory Tree Building** | 3 | `KeyError` in `FSNode.getnode()` when adding files | -| **Unmount** | 1 | Filesystem not unmounting cleanly | - -### Passing Tests (12) - -- `test_fuse_available` - FUSE/fusermount detected -- `test_library_fixture_created` - SQLite DB and music dir created -- `test_temp_directory_created` - Temp dirs set up correctly -- `test_mount_empty_library` - **Mount works with empty library!** -- `test_list_empty_root` - Empty root returns empty list -- `test_list_root_returns_list` - Returns list type -- `test_access_empty_path` - Handles empty path -- Plus 5 nested bug detection tests (confirming bug exists) - -## Executive Summary - -E2E tests for beetfs FUSE filesystem using real music files from qBittorrent container. No mocks - actual filesystem operations against mounted beetfs. - -### Critical Finding - -**BUG DISCOVERED**: Lines 758-1144 in `beetFs.py` are indented inside `access()` method, making these FUSE operations unreachable as class methods: -- `readdir`, `open`, `read`, `write`, `mkdir`, `unlink`, `rmdir`, `symlink`, `link`, `rename`, `chmod`, `chown`, `truncate`, `opendir`, `releasedir`, `fsyncdir`, `create`, `fgetattr`, `release`, `fsync`, `flush`, `ftruncate` - -Tests will expose this immediately - write `test_readdir.py` first. - ---- - -## Test Environment - -| Component | Status | Details | -|-----------|--------|---------| -| Real Music | Available | Metallica "72 Seasons" (12 FLAC, 650MB) at `/home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/` | -| Synthetic Music | Create | 5-10MB FLACs for most tests (avoid RAM explosion) | -| Beets Config | Create | `~/.config/beets/config.yaml` for test isolation | -| Beets Library | Empty | Needs import of test files | -| Python | 2.7.15 | Via Nix flake (nixpkgs-18.09) | -| Test Framework | unittest | stdlib, no external deps for Py2.7 | - ---- - -## Test Architecture - -``` -beetfs/tests/ -├── __init__.py -├── conftest.py # Test fixtures, beets library setup, synthetic FLAC creation -├── test_smoke.py # Mount/unmount lifecycle (run FIRST) -├── test_nested_bug.py # Verify the indentation bug (run SECOND) -├── test_readdir.py # Directory listing operations -├── test_read.py # File reading with metadata overlay (CORE FEATURE) -├── test_stat.py # getattr, fgetattr, statfs -├── test_write.py # Metadata write operations -├── test_error_handling.py # ENOENT, EOPNOTSUPP scenarios -├── test_edge_cases.py # Unicode, concurrent opens, special chars -├── test_integration.py # Real 650MB files (skip by default) -└── fixtures/ - ├── synthetic/ # Generated 5-10MB test FLACs - └── real -> /home/fujin/.local/share/docker/volumes/containers_downloads/_data/ -``` - ---- - -## Test Tiers - -### Tier 1: Unit-ish (Synthetic FLACs, ~500KB each) -- Fast execution -- No memory issues (FileHandler loads entire file to RAM) -- Run on every commit - -### Tier 2: Integration (Subset of real files, 1-2 tracks) -- Uses real Metallica FLACs -- Tests real-world metadata -- Run before merge - -### Tier 3: E2E (All 12 tracks, 650MB) -- Full album processing -- Memory stress testing -- Run via `E2E=1 python -m unittest discover` -- Skip by default - ---- - -## Test Isolation Strategy - -| Resource | Strategy | Rationale | -|----------|----------|-----------| -| Audio Files | **Symlinks** for reads | beetfs NEVER writes to source files, only to beets DB | -| Beets DB | **Copy per test** | Writes mutate DB; need isolation | -| Mount Point | **Fresh tempdir** | Each test gets clean mount | -| Global State | **Fresh subprocess** | `library`, `directory_structure` are module globals | - ---- - -## Implementation Order - -> Reordered per Oracle recommendation: smoke → nested-bug → read → write → errors → edge - -### Phase 1: Infrastructure (Day 1 AM) - -1. Create `tests/` directory structure -2. Implement `BeetFSTestCase` base class with: - - Subprocess timeout via `threading.Timer` (Py2.7 compatible) - - Mount wait polling (`os.path.ismount()`) - - Proper cleanup (`fusermount -u`) -3. Create synthetic FLAC generator using ffmpeg + flac CLI -4. Setup isolated beets config and library - -### Phase 2: Bug Detection (Day 1 PM) - -5. `test_smoke.py` - Mount/unmount lifecycle -6. `test_nested_bug.py` - Verify `readdir`, `open` are callable (will fail, exposing bug) - -### Phase 3: Core Tests (Day 2) - -7. `test_readdir.py` - Directory listing -8. `test_read.py` - **Metadata overlay verification** (critical) -9. `test_stat.py` - File/directory attributes - -### Phase 4: Write & Errors (Day 3) - -10. `test_write.py` - Metadata modification, DB persistence -11. `test_error_handling.py` - ENOENT, EOPNOTSUPP - -### Phase 5: Edge Cases (Day 3-4) - -12. `test_edge_cases.py` - Unicode, concurrent opens, special chars -13. `test_integration.py` - Real 650MB files (optional tier) - ---- - -## Test Categories - -### 1. Smoke Tests (`test_smoke.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_mount_success` | Mount beetfs | `os.path.ismount()` returns True | -| `test_unmount_clean` | Unmount | Process exits 0, dir accessible | -| `test_mount_empty_library` | Mount with 0 items | Mounts successfully, root empty | -| `test_mount_invalid_path` | Mount to non-existent | Fails gracefully | -| `test_fsinit_called` | Check initialization | No crash on mount | - -### 2. Nested Methods Bug (`test_nested_bug.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_readdir_exists` | `hasattr(beetFileSystem, 'readdir')` | True (currently False!) | -| `test_open_exists` | `hasattr(beetFileSystem, 'open')` | True (currently False!) | -| `test_read_exists` | `hasattr(beetFileSystem, 'read')` | True (currently False!) | -| `test_readdir_callable` | `os.listdir(mount)` | Returns list (currently fails!) | - -### 3. Directory Operations (`test_readdir.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_list_root` | `os.listdir(mount)` | Returns artist directories | -| `test_list_artist` | `os.listdir(mount/artist)` | Returns album directories | -| `test_list_album` | `os.listdir(mount/artist/album)` | Returns track files | -| `test_path_format` | Check structure | Matches `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format` | -| `test_unicode_paths` | Non-ASCII chars | Handles "Lux Aeterna" correctly | - -### 4. Read Operations (`test_read.py`) - CORE FEATURE - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_read_header_overlay` | Read + parse with mutagen | Tags match DB, not file | -| `test_read_audio_passthrough` | Compare audio bytes | Identical to original after header | -| `test_read_full_file` | Read entire file | Header from DB + audio from file | -| `test_metadata_artist` | Check artist tag | DB value, not file value | -| `test_metadata_title` | Check title tag | DB value, not file value | -| `test_metadata_album` | Check album tag | DB value, not file value | -| `test_metadata_genre` | Check genre tag | DB value, not file value | -| `test_original_unchanged` | Read original file | Original metadata intact | - -#### Metadata Overlay Verification Pattern - -```python -import mutagen.flac -from io import BytesIO - -def test_read_header_overlay(self): - # Setup: Import file, modify DB metadata - # beet import /path/to/file - # beet modify artist="DB Artist" # File has "Original Artist" - - # Read mounted file as bytes - with open(os.path.join(self.mount_dir, 'DB Artist/...'), 'rb') as f: - mounted_data = f.read() - - # Parse with mutagen - flac = mutagen.flac.FLAC(BytesIO(mounted_data)) - - # Verify overlay worked - self.assertEqual(flac['artist'][0], 'DB Artist') # From DB - self.assertNotEqual(flac['artist'][0], 'Original Artist') # Not from file -``` - -### 5. Stat Operations (`test_stat.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_stat_file` | `os.stat(file)` | Valid stat with size, mtime | -| `test_stat_directory` | `os.stat(dir)` | Directory mode (S_IFDIR) | -| `test_statfs` | `os.statvfs(mount)` | Valid filesystem stats | -| `test_access_read` | `os.access(file, R_OK)` | True | -| `test_access_write` | `os.access(file, W_OK)` | True (header writable) | - -### 6. Write Operations (`test_write.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_write_title` | Modify title in header | DB updated, file unchanged | -| `test_write_artist` | Modify artist | DB updated | -| `test_write_album` | Modify album | DB updated | -| `test_write_genre` | Modify genre | DB updated | -| `test_write_audio_discarded` | Write at offset > bound | Silently discarded | -| `test_write_persistence` | Write -> unmount -> remount | Changes persisted in DB | -| `test_write_mp3_noop` | Write to MP3 header | No error, but no effect (bound=0) | - -### 7. Error Handling (`test_error_handling.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_enoent_file` | Read non-existent | `OSError(ENOENT)` | -| `test_enoent_dir` | List non-existent | `OSError(ENOENT)` | -| `test_eopnotsupp_mkdir` | `os.mkdir()` | `OSError(EOPNOTSUPP)` | -| `test_eopnotsupp_unlink` | `os.unlink()` | `OSError(EOPNOTSUPP)` | -| `test_eopnotsupp_rename` | `os.rename()` | `OSError(EOPNOTSUPP)` | -| `test_eopnotsupp_symlink` | `os.symlink()` | `OSError(EOPNOTSUPP)` | - -### 8. Edge Cases (`test_edge_cases.py`) - -| Test | Operation | Expected | -|------|-----------|----------| -| `test_special_chars_sanitized` | Path with `?/` | Sanitized via `sanitize()` | -| `test_concurrent_opens` | Open same file twice | `instance_count` increments | -| `test_concurrent_release` | Release after double open | File stays cached until count=0 | -| `test_unicode_metadata` | Non-ASCII in artist/title | Handled correctly | -| `test_empty_metadata` | None/empty fields | Doesn't crash | -| `test_mp3_no_interpolation` | Read MP3 | Returns original file (no overlay) | - -### 9. Integration (`test_integration.py`) - -| Test | Env Var | Expected | -|------|---------|----------| -| `test_real_album_listing` | `E2E=1` | Lists all 12 Metallica tracks | -| `test_real_file_read` | `E2E=1` | Reads 67MB file successfully | -| `test_memory_usage` | `E2E=1` | Documents but doesn't fail on high RAM | - ---- - -## Test Infrastructure Code - -### Base Test Class (Python 2.7 Compatible) - -```python -# tests/conftest.py -import unittest -import subprocess -import tempfile -import shutil -import os -import time -import threading - -class BeetFSTestCase(unittest.TestCase): - """Base class for beetfs e2e tests - Python 2.7 compatible""" - - MOUNT_TIMEOUT = 30 # seconds - - @classmethod - def setUpClass(cls): - """Check FUSE availability""" - try: - with open(os.devnull, 'w') as devnull: - subprocess.check_call(['which', 'fusermount'], - stdout=devnull, stderr=devnull) - except subprocess.CalledProcessError: - raise unittest.SkipTest("fusermount not available") - - def setUp(self): - self.mount_dir = tempfile.mkdtemp(prefix='beetfs_test_') - self.fs_process = None - - def mount_beetfs(self, library_path=None): - """Mount beetfs in background with timeout""" - cmd = ['python', '-c', - 'from beetsplug.beetFs import mount; mount()'] - # Add mount point and other args as needed - - self.fs_process = subprocess.Popen( - cmd, - stdout=open(os.devnull, 'w'), - stderr=subprocess.STDOUT - ) - - # Python 2.7 timeout workaround - timer = threading.Timer(self.MOUNT_TIMEOUT, self._timeout_kill) - timer.start() - - try: - self._wait_for_mount() - finally: - timer.cancel() - - def _timeout_kill(self): - if self.fs_process and self.fs_process.poll() is None: - self.fs_process.kill() - - def _wait_for_mount(self): - """Wait for filesystem to be mounted""" - start = time.time() - while time.time() - start < self.MOUNT_TIMEOUT: - if os.path.ismount(self.mount_dir): - return - if self.fs_process.poll() is not None: - self.fail("Filesystem process terminated prematurely") - time.sleep(0.1) - self.fail("Mount timeout after {} seconds".format(self.MOUNT_TIMEOUT)) - - def tearDown(self): - """Cleanup: unmount and kill process""" - if self.fs_process: - with open(os.devnull, 'w') as devnull: - subprocess.call(['fusermount', '-z', '-u', self.mount_dir], - stdout=devnull, stderr=devnull) - - self.fs_process.terminate() - - # Wait for termination (Py2.7 compatible) - start = time.time() - while time.time() - start < 5: - if self.fs_process.poll() is not None: - break - time.sleep(0.1) - else: - self.fs_process.kill() - - shutil.rmtree(self.mount_dir, ignore_errors=True) -``` - -### Synthetic FLAC Generator - -```python -# tests/conftest.py (continued) -import subprocess -import tempfile -import os - -def create_synthetic_flac(duration_sec=5, artist="Test Artist", - title="Test Track", album="Test Album"): - """Create minimal FLAC with known metadata (~500KB for 5s silence)""" - wav_fd, wav_path = tempfile.mkstemp(suffix='.wav') - os.close(wav_fd) - flac_path = wav_path.replace('.wav', '.flac') - - try: - # Generate silence WAV - subprocess.check_call([ - 'ffmpeg', '-f', 'lavfi', '-i', - 'anullsrc=r=44100:cl=stereo', '-t', str(duration_sec), - '-y', wav_path - ], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT) - - # Convert to FLAC with metadata - subprocess.check_call([ - 'flac', '--best', - '-T', 'ARTIST={}'.format(artist), - '-T', 'TITLE={}'.format(title), - '-T', 'ALBUM={}'.format(album), - '-o', flac_path, wav_path - ], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT) - - return flac_path - finally: - if os.path.exists(wav_path): - os.unlink(wav_path) -``` - ---- - -## Dependencies to Add to flake.nix - -```nix -# In devShell buildInputs, add: -pkgs.ffmpeg # For synthetic FLAC generation -pkgs.flac # For FLAC encoding - -# pythonEnv already has mutagen for verification -``` - ---- - -## Risks & Mitigations - -| Risk | Impact | Mitigation | -|------|--------|------------| -| Memory explosion | High | Use 5-10MB synthetic FLACs, skip 650MB tests by default | -| Nested methods bug | Critical | Tests will expose; fix required before other tests pass | -| Python 2.7 EOL | Medium | Nix provides isolated environment | -| Global state pollution | Medium | Fresh subprocess per test | -| FUSE permissions | Low | Run as regular user, skip privileged tests | -| Concurrent access | Low | Single-threaded mode, sequential tests | - ---- - -## Success Criteria - -1. **All smoke tests pass** - beetfs mounts and unmounts cleanly -2. **Nested bug exposed and fixed** - All FUSE methods callable -3. **Metadata overlay verified** - Reads return DB metadata, not file metadata -4. **Writes update DB** - Metadata changes persist -5. **Errors handled gracefully** - Correct errno for unsupported ops -6. **No crashes on edge cases** - Unicode, special chars, concurrent access - ---- - -## Findings from Test Execution - -### Bug #1: Nested Methods (CRITICAL) - -**Location**: `beetFs.py` lines 758-1144 - -**Problem**: All FUSE operation methods are indented inside the `access()` method, making them local functions instead of class methods. - -**Evidence**: -```python -def access(self, path, flags): # Line 723 - correct class method - ... - return 0 - - def readdir(self, path, ...): # Line 931 - WRONG! Nested inside access() - ... - def open(self, path, flags): # Line 988 - Also nested - ... - def read(self, path, ...): # Line 1077 - Also nested - ... -``` - -**Symptom**: `os.listdir()` returns `OSError: [Errno 38] Function not implemented` - -**Fix Required**: Dedent lines 758-1144 by 8 spaces to make them class methods. - -### Bug #2: Directory Tree Building - -**Location**: `beetFs.py` lines 403-414 (`FSNode.getnode()` and `FSNode.adddir()`) - -**Problem**: When adding files to the directory structure, the code assumes parent directories already exist. - -**Evidence**: -``` -KeyError: u'Test Artist' -File "beetFs.py", line 403, in getnode - return self.getnode(elements, root=root.dirs[topdir]) -``` - -**Symptom**: Mount fails when library contains tracks. - -### Bug #3: Unmount Not Clean - -**Problem**: After unmounting, `os.path.ismount()` still returns `True`. - -**Likely Cause**: FUSE process not terminating properly, or lazy unmount not completing. - ---- - -## Notes from Oracle Review - -1. **MP3 is not "readonly"** - metadata overlay is disabled (`bound=0`), but reads still work -2. **Write returns None for MP3** - no explicit return in MP3 path (falls through) -3. **Path format is hardcoded** - tests must match `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format` -4. **basestring vs str** - use `isinstance(x, basestring)` for Py2.7 string checks -5. **Global variables** - `library`, `directory_structure` must be reset between tests (use subprocesses) diff --git a/docs/v1/features.md b/docs/v1/features.md deleted file mode 100644 index 3f145a8..0000000 --- a/docs/v1/features.md +++ /dev/null @@ -1,249 +0,0 @@ -# beetfs Feature Set - -## Overview - -beetfs is a FUSE filesystem plugin for [beets](https://beets.io/) that presents your music library as a virtual filesystem organized by metadata. Files appear with paths derived from their database metadata, and reading file headers returns metadata from the beets database rather than the actual file tags. - -**Author**: Martin Eve (2010) -**License**: GPLv3 -**Python**: 2.7 (uses fuse-python) - -## Core Features - -### 1. Virtual Metadata-Based Directory Structure - -Files are presented in a configurable path format based on beets database fields: - -``` -$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format -``` - -**Example**: -``` -/mnt/beetfs/ -├── Metallica/ -│ └── 72 Seasons (2023) [FLAC]/ -│ ├── 01 - Metallica - 72 Seasons.flac -│ ├── 02 - Metallica - Shadows Follow.flac -│ └── ... -├── Pink Floyd/ -│ └── The Dark Side of the Moon (1973) [FLAC]/ -│ └── ... -``` - -**Available template variables**: -- `$artist`, `$album`, `$title`, `$genre`, `$composer`, `$grouping` -- `$year`, `$month`, `$day` -- `$track`, `$tracktotal`, `$disc`, `$disctotal` -- `$format`, `$format_upper` (file extension) -- `$lyrics`, `$comments`, `$bpm`, `$comp` - -### 2. Metadata Overlay (Read) - -When you read a file through beetfs, the **metadata header is synthesized from the beets database**, not read from the actual file on disk. - -**How it works**: -1. Open file → beetfs reads the real file from disk -2. Parse the audio format header (FLAC/MP3) -3. Replace metadata fields with values from beets database -4. Return synthesized header + original audio data - -**Supported fields for overlay**: -- `title`, `artist`, `album`, `genre` (FLAC only currently) - -**Use case**: Your files may have inconsistent or wrong tags, but beetfs presents them with the corrected metadata from your beets library. - -### 3. Metadata Passthrough (Write) - -When you write to file headers through beetfs, the **changes are saved to the beets database**, not to the actual file. - -**How it works**: -1. Application writes new metadata to file header region -2. beetfs intercepts the write -3. Parses the new metadata values -4. Updates the beets database (`lib.store()`, `lib.save()`) -5. Regenerates the synthesized header - -**Result**: Tag editors (Picard, Kid3, etc.) can edit metadata through beetfs, and changes persist in the beets database without modifying the original files. - -### 4. Format Support - -| Format | Read | Metadata Overlay | Write to DB | -|--------|------|------------------|-------------| -| FLAC | ✅ | ✅ Full | ✅ | -| MP3 | ✅ | ❌ Disabled | ❌ | -| Other | ❌ | ❌ | ❌ | - -**FLAC Implementation**: -- Uses `InterpolatedFLAC` class extending mutagen -- Reconstructs Vorbis comment block with DB values -- Preserves audio data and other metadata blocks - -**MP3 Implementation**: -- Passthrough only (no interpolation) -- `self.bound = 0` disables header replacement - -### 5. File Caching - -Open files are cached in `FileHandler` objects: - -- First open: Load entire file into memory, parse headers -- Subsequent opens: Reuse cached `FileHandler` -- Reference counting for multiple opens -- Release when reference count reaches zero - -**Memory impact**: Each open file consumes ~filesize RAM. - -## FUSE Operations - -### Implemented (Functional) - -| Operation | Description | -|-----------|-------------| -| `getattr` | File/directory stat (size, mode, timestamps) | -| `access` | Permission checking | -| `opendir` | Open directory for listing | -| `readdir` | List directory contents | -| `releasedir` | Close directory | -| `open` | Open file for reading/writing | -| `read` | Read file contents | -| `write` | Write to file (header region only) | -| `release` | Close file | -| `fgetattr` | Stat with file handle | -| `statfs` | Filesystem statistics | - -### Not Implemented (Return EOPNOTSUPP) - -| Operation | Reason | -|-----------|--------| -| `create` | Read-only structure | -| `mknod` | Read-only structure | -| `mkdir` | Read-only structure | -| `unlink` | Read-only structure | -| `rmdir` | Read-only structure | -| `symlink` | Not needed | -| `link` | Not needed | -| `rename` | Would break DB consistency | -| `chmod` | Metadata-only FS | -| `chown` | Metadata-only FS | -| `truncate` | Would corrupt audio | -| `utime` | Metadata-only FS | - -## Usage - -### Mount - -```bash -beet mount /mnt/beetfs -``` - -### Unmount - -```bash -fusermount -u /mnt/beetfs -``` - -### Example Session - -```bash -# Mount the filesystem -beet mount /mnt/music - -# Browse by artist -ls /mnt/music/ -# Metallica/ Pink Floyd/ The Beatles/ ... - -# List an album -ls "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/" -# 01 - Metallica - 72 Seasons.flac -# 02 - Metallica - Shadows Follow.flac -# ... - -# Play through any music player -mpv "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/01 - Metallica - 72 Seasons.flac" - -# Edit tags (changes go to beets DB) -kid3 "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/" - -# Unmount -fusermount -u /mnt/music -``` - -## Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ User Applications │ -│ (mpv, Rhythmbox, Kid3, etc.) │ -└─────────────────────────┬───────────────────────────────────┘ - │ POSIX calls (open, read, write) - ▼ -┌─────────────────────────────────────────────────────────────┐ -│ Linux Kernel │ -│ FUSE module │ -└─────────────────────────┬───────────────────────────────────┘ - │ /dev/fuse - ▼ -┌─────────────────────────────────────────────────────────────┐ -│ beetfs │ -│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │ -│ │ FSNode Tree │ │ FileHandler │ │ InterpolatedFLAC │ │ -│ │ (in-memory) │ │ (cache) │ │ (header synth) │ │ -│ └─────────────┘ └──────────────┘ └───────────────────┘ │ -└────────┬────────────────┬───────────────────┬───────────────┘ - │ │ │ - ▼ ▼ ▼ -┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ Beets DB │ │ Real Files │ │ Mutagen │ -│ (SQLite) │ │ (on disk) │ │ (parsing) │ -└─────────────┘ └─────────────────┘ └─────────────────┘ -``` - -## Limitations - -### Current Bugs (Non-Functional) - -1. **Nested Methods Bug**: Lines 758-1144 are indented inside `access()`, making FUSE operations unreachable -2. **Directory Tree Bug**: `FSNode.adddir()` crashes when building tree for non-empty library - -### Design Limitations - -1. **Memory Usage**: Entire file loaded into RAM on open -2. **Mount Time**: O(N) - loads all library items at mount -3. **No Lazy Loading**: Full directory tree built upfront -4. **Single Format**: Only FLAC has full metadata overlay -5. **No Real File Modification**: Writes only update DB, not actual files -6. **Python 2.7 GIL**: Single-threaded performance - -### Not Supported - -- Creating/deleting files or directories -- Moving/renaming files -- Modifying audio content -- Album art / embedded images -- Multi-value tags -- Non-ASCII in some edge cases - -## Configuration - -Currently hardcoded. Potential configuration points: - -| Setting | Current Value | Description | -|---------|---------------|-------------| -| `PATH_FORMAT` | `$artist/$album ($year)...` | Directory structure template | -| `METADATA_RW_FIELDS` | 17 fields | Fields available for read/write | -| Caching | Always on | FileHandler caching behavior | -| Threading | Disabled | `multithreaded = 0` | - -## Dependencies - -- Python 2.7 -- fuse-python -- beets 1.4.x -- mutagen (FLAC/MP3 parsing) - -## See Also - -- [e2e-test-plan.md](e2e-test-plan.md) - Test strategy and bug documentation -- [benchmark-plan.md](benchmark-plan.md) - Performance measurement methodology -- [benchmark-results.md](benchmark-results.md) - Current benchmark status diff --git a/docs/v1/modernization.md b/docs/v1/modernization.md deleted file mode 100644 index 52f481a..0000000 --- a/docs/v1/modernization.md +++ /dev/null @@ -1,459 +0,0 @@ -# beetfs Modernization Guide - -## Current State Analysis - -### Technical Debt - -| Issue | Severity | Location | -|-------|----------|----------| -| Python 2 syntax | 🔴 Critical | Throughout | -| fuse-python (deprecated) | 🔴 Critical | Lines 25, 51 | -| `basestring` usage | 🔴 Critical | Line 89 | -| `reduce` without import | 🟡 Medium | Line 197 | -| `0755` octal syntax | 🟡 Medium | Lines 654, 700 | -| `print` as statement | 🟡 Medium | N/A (not used) | -| `except Exception, e` | 🔴 Critical | Line 181 | -| Long integers (`0L`) | 🟡 Medium | Line 197 | -| Global state | 🟡 Medium | Lines 125-140 | -| Memory-heavy design | 🟡 Medium | Line 481 | - -### Dependencies to Update - -| Original | Replacement | Notes | -|----------|-------------|-------| -| `fuse-python` | `pyfuse3` or `llfuse` | Modern FUSE bindings | -| `beets` (old API) | `beets >= 1.6` | Check API compatibility | -| `mutagen` | `mutagen >= 1.45` | Mostly compatible | -| Python 2.7 | Python 3.9+ | Full migration needed | - ---- - -## Migration Steps - -### Phase 1: Python 3 Compatibility - -#### 1.1 Fix Syntax Issues - -```python -# BEFORE (Python 2) -except fuse.FuseError, e: - log.error(str(e)) - -# AFTER (Python 3) -except fuse.FuseError as e: - log.error(str(e)) -``` - -```python -# BEFORE -if isinstance(value, basestring): - -# AFTER -if isinstance(value, str): -``` - -```python -# BEFORE -return reduce(lambda a, b: (a << 8) + ord(b), string, 0L) - -# AFTER -from functools import reduce -return reduce(lambda a, b: (a << 8) + b, string, 0) -``` - -```python -# BEFORE -mode = stat.S_IFDIR | 0755 - -# AFTER -mode = stat.S_IFDIR | 0o755 -``` - -#### 1.2 Fix String/Bytes Handling - -```python -# BEFORE - implicit string/bytes mixing -self.header = self.inf.get_header(self.real_path) -return self.header[offset:offset+size] - -# AFTER - explicit bytes handling -self.header: bytes = self.inf.get_header(self.real_path) -return self.header[offset:offset+size] -``` - -```python -# BEFORE -self.item.title = str(self.inf["title"][0]).encode('utf-8') - -# AFTER -self.item.title = self.inf["title"][0] # Already str in Python 3 -``` - -#### 1.3 Fix Dictionary Methods - -```python -# BEFORE -return node.dirs.keys() - -# AFTER -return list(node.dirs.keys()) # If list is needed -# or just -return node.dirs.keys() # If iteration is sufficient -``` - ---- - -### Phase 2: FUSE Library Migration - -#### Option A: pyfuse3 (Recommended) - -Modern, async-capable FUSE bindings. - -```python -# BEFORE (fuse-python) -import fuse -fuse.fuse_python_api = (0, 2) - -class beetFileSystem(fuse.Fuse): - def read(self, path, size, offset): - return data - -# AFTER (pyfuse3) -import pyfuse3 -import trio - -class BeetFS(pyfuse3.Operations): - async def read(self, fh, offset, size): - return data - -async def main(): - fs = BeetFS() - fuse_options = set(pyfuse3.default_options) - fuse_options.add('fsname=beetfs') - pyfuse3.init(fs, mountpoint, fuse_options) - try: - await pyfuse3.main() - finally: - pyfuse3.close() - -trio.run(main) -``` - -**Key Differences**: -| fuse-python | pyfuse3 | -|-------------|---------| -| `read(path, size, offset)` | `read(fh, offset, size)` | -| Synchronous | Async (trio) | -| Return data directly | Return bytes | -| Path-based | File handle based | - -#### Option B: llfuse (Alternative) - -Lower-level, synchronous. - -```python -import llfuse - -class BeetFS(llfuse.Operations): - def read(self, fh, offset, size): - return data - -def main(): - fs = BeetFS() - llfuse.init(fs, mountpoint, options) - try: - llfuse.main() - finally: - llfuse.close() -``` - -#### Option C: fusepy (Simple) - -Simple wrapper, but less maintained. - -```python -from fuse import FUSE, Operations - -class BeetFS(Operations): - def read(self, path, size, offset, fh): - return data - -FUSE(BeetFS(), mountpoint, foreground=True) -``` - ---- - -### Phase 3: Architecture Improvements - -#### 3.1 Remove Global State - -```python -# BEFORE - Global variables -global structure_split -global structure_depth -global library -global directory_structure - -# AFTER - Instance variables -class BeetFS: - def __init__(self, lib: Library, path_format: str): - self.lib = lib - self.path_format = path_format - self.structure_split = path_format.split("/") - self.structure_depth = len(self.structure_split) - self.directory_structure = FSNode({}, {}) - self._build_tree() -``` - -#### 3.2 Reduce Memory Usage - -```python -# BEFORE - Load entire audio into memory -self.music_data = self.file_object.read() # Could be 100MB+ - -# AFTER - Lazy loading with mmap or seek -class FileHandler: - def __init__(self, path, lib): - self.real_path = self._resolve_path(path) - self.file_object = open(self.real_path, 'rb') - self._header = None # Lazy load - self._music_offset = None - - @property - def header(self) -> bytes: - if self._header is None: - self._header = self._generate_header() - return self._header - - def read(self, size: int, offset: int) -> bytes: - if offset < len(self.header): - # Header region - return from generated header - if offset + size <= len(self.header): - return self.header[offset:offset+size] - else: - # Span header and audio - header_part = self.header[offset:] - audio_offset = 0 - audio_size = size - len(header_part) - audio_part = self._read_audio(audio_offset, audio_size) - return header_part + audio_part - else: - # Audio region - read directly from file - audio_offset = offset - len(self.header) - return self._read_audio(audio_offset, size) - - def _read_audio(self, offset: int, size: int) -> bytes: - self.file_object.seek(self._music_offset + offset) - return self.file_object.read(size) -``` - -#### 3.3 Add Type Hints - -```python -from typing import Dict, List, Optional, Tuple -from pathlib import Path - -class FSNode: - def __init__(self, dirs: Dict[str, 'FSNode'], files: Dict[str, int]): - self.dirs: Dict[str, FSNode] = dirs - self.files: Dict[str, int] = files - - def getnode(self, elements: List[str], root: Optional['FSNode'] = None) -> 'FSNode': - ... - - def addfile(self, elements: List[str], filename: str, item_id: int) -> None: - ... -``` - -#### 3.4 Add MP3 Support - -```python -class FileHandler: - def __init__(self, path: str, lib: Library): - self.format = Path(path).suffix[1:].lower() - - if self.format == "flac": - self._handler = FLACHandler(self.real_path, self.item) - elif self.format == "mp3": - self._handler = MP3Handler(self.real_path, self.item) - elif self.format in ("ogg", "opus"): - self._handler = OggHandler(self.real_path, self.item) - else: - raise UnsupportedFormatError(f"Format {self.format} not supported") - -class FLACHandler: - def generate_header(self, item: Item) -> bytes: - inf = InterpolatedFLAC(self.file_data) - inf["title"] = item.title - inf["album"] = item.album - inf["artist"] = item.artist - inf["genre"] = item.genre - return inf.get_header() - -class MP3Handler: - def generate_header(self, item: Item) -> bytes: - # Implement ID3v2 header generation - id3 = InterpolatedID3() - id3.add(TIT2(encoding=3, text=item.title)) - id3.add(TPE1(encoding=3, text=item.artist)) - id3.add(TALB(encoding=3, text=item.album)) - id3.add(TCON(encoding=3, text=item.genre)) - - # Calculate padding to match original header size - ... - return id3.render() -``` - ---- - -### Phase 4: Testing - -#### 4.1 Unit Tests - -```python -import pytest -from beetfs import FSNode, FileHandler - -class TestFSNode: - def test_adddir(self): - root = FSNode({}, {}) - root.adddir([], "Artist") - assert "Artist" in root.dirs - - def test_addfile(self): - root = FSNode({}, {}) - root.adddir([], "Artist") - root.addfile(["Artist"], "track.flac", 42) - assert root.dirs["Artist"].files["track.flac"] == 42 - - def test_getnode(self): - root = FSNode({}, {}) - root.adddir([], "Artist") - root.adddir(["Artist"], "Album") - node = root.getnode(["Artist", "Album"]) - assert node is not None - -class TestFileHandler: - def test_read_header(self, mock_flac_file, mock_beets_item): - handler = FileHandler("/Artist/Album/track.flac", mock_lib) - data = handler.read(100, 0) - assert data.startswith(b"fLaC") - - def test_read_audio(self, mock_flac_file, mock_beets_item): - handler = FileHandler("/Artist/Album/track.flac", mock_lib) - data = handler.read(100, handler.bound + 100) - # Should be audio data from original file - assert data == mock_flac_file.audio_data[100:200] -``` - -#### 4.2 Integration Tests - -```python -import subprocess -import tempfile -import os - -class TestFUSEMount: - def test_mount_unmount(self, beets_library): - with tempfile.TemporaryDirectory() as mountpoint: - # Mount - proc = subprocess.Popen( - ["beet", "mount", mountpoint], - stdout=subprocess.PIPE - ) - time.sleep(1) - - # Verify mount - assert os.path.ismount(mountpoint) - - # List files - files = os.listdir(mountpoint) - assert len(files) > 0 - - # Unmount - subprocess.run(["fusermount", "-u", mountpoint]) - proc.wait() -``` - ---- - -### Phase 5: Standalone Mode (Optional) - -Remove beets dependency for use as standalone metadata overlay. - -```python -class StandaloneFS: - """Metadata overlay without beets dependency.""" - - def __init__(self, - source_dir: Path, - metadata_db: Path, - path_format: str): - self.source_dir = source_dir - self.db = sqlite3.connect(metadata_db) - self.path_format = path_format - self._build_tree() - - def _build_tree(self): - """Build virtual tree from source directory and metadata DB.""" - for audio_file in self.source_dir.rglob("*.flac"): - # Get metadata from DB or scan file - metadata = self._get_metadata(audio_file) - # Build virtual path from template - virtual_path = self._format_path(metadata) - # Add to tree - self.directory_structure.addfile( - virtual_path.parent.parts, - virtual_path.name, - str(audio_file) # Store actual path instead of ID - ) -``` - ---- - -## Recommended Migration Order - -``` -1. [ ] Fork and set up development environment -2. [ ] Add type hints throughout (helps catch issues) -3. [ ] Fix Python 3 syntax issues -4. [ ] Replace fuse-python with pyfuse3/llfuse -5. [ ] Add unit tests for FSNode and FileHandler -6. [ ] Refactor global state to instance variables -7. [ ] Implement lazy loading for audio data -8. [ ] Add MP3 support -9. [ ] Add integration tests -10. [ ] Optional: Create standalone mode -``` - ---- - -## Estimated Effort - -| Phase | Effort | Risk | -|-------|--------|------| -| Phase 1 (Python 3) | 2-3 days | Low | -| Phase 2 (FUSE migration) | 3-5 days | Medium | -| Phase 3 (Architecture) | 3-5 days | Medium | -| Phase 4 (Testing) | 2-3 days | Low | -| Phase 5 (Standalone) | 3-5 days | Medium | -| **Total** | **13-21 days** | | - ---- - -## Alternative: Rewrite from Scratch - -Given the age of the codebase, a rewrite might be more efficient: - -**Pros of Rewrite**: -- Clean architecture from start -- Modern async design -- Better memory management -- Easier to test - -**Cons of Rewrite**: -- More initial effort -- Risk of missing edge cases -- Need to re-discover FLAC/ID3 intricacies - -**Recommended Approach**: Start with Phase 1-2 to understand the code deeply, then decide whether to continue refactoring or rewrite. diff --git a/docs/v1/rust-migration.md b/docs/v1/rust-migration.md deleted file mode 100644 index e83f2e3..0000000 --- a/docs/v1/rust-migration.md +++ /dev/null @@ -1,451 +0,0 @@ -# Rust Migration Analysis for beetfs - -## Executive Summary - -Migrating beetfs from Python to Rust is **strongly recommended** based on research findings. Expected improvements: - -| Metric | Python (Current) | Rust (Expected) | Improvement | -|--------|------------------|-----------------|-------------| -| **Memory per file** | ~280 bytes overhead | ~60 bytes | **4-5x reduction** | -| **File open latency** | 200-500ms | 20-50ms | **10x faster** | -| **Read latency** | 5-10ms | 0.5-2ms | **5-10x faster** | -| **Concurrent opens** | ~1,000 (threading) | ~100,000+ (Tokio) | **100x more** | -| **GC pauses** | 50-2200ms | 0ms | **Eliminated** | - ---- - -## 1. Rust FUSE Ecosystem - -### Recommended: **fuser** - -| Attribute | Value | -|-----------|-------| -| **Downloads** | 3.2M+ | -| **Maturity** | Production-ready | -| **Platforms** | Linux, macOS, FreeBSD | -| **Async** | Experimental (stable sync API) | -| **Used by** | AWS Mountpoint for S3 | - -**API Example:** -```rust -use fuser::{Filesystem, Request, ReplyData}; - -impl Filesystem for BeetFS { - fn read(&self, _req: &Request, ino: u64, _fh: u64, - offset: i64, size: u32, _flags: i32, - _lock: Option, reply: ReplyData) { - - let file = self.get_file(ino); - - if offset < file.header_len { - // Return metadata from database (interpolated) - reply.data(&file.header[offset as usize..]); - } else { - // Return audio from original file (zero-copy via mmap) - let audio_offset = offset - file.header_len; - reply.data(&file.mmap[audio_offset as usize..]); - } - } -} -``` - -### Alternatives - -| Library | Async | Maturity | Best For | -|---------|-------|----------|----------| -| **fuser** | Experimental | ⭐⭐⭐⭐⭐ | General purpose | -| **fuse3** | Native | ⭐⭐⭐⭐ | Async-heavy, Linux-only | -| **polyfuse** | Native | ⭐⭐⭐ | Custom control flow | - ---- - -## 2. Rust Audio Metadata: **lofty** - -Full feature parity with Python's mutagen: - -| Feature | mutagen (Python) | lofty (Rust) | -|---------|------------------|--------------| -| FLAC Vorbis Comments | ✅ | ✅ | -| MP3 ID3v2 (all versions) | ✅ | ✅ | -| OGG Vorbis Comments | ✅ | ✅ | -| Opus metadata | ✅ | ✅ | -| In-memory manipulation | ✅ | ✅ | -| Header generation | ✅ | ✅ `dump_to()` | -| Picture/artwork | ✅ | ✅ | - -**API Comparison:** -```python -# Python mutagen -audio = mutagen.File("song.flac") -audio['artist'] = 'New Artist' -audio['title'] = 'New Title' -audio.save() -``` - -```rust -// Rust lofty -let mut file = lofty::read_from_path("song.flac")?; -let tag = file.primary_tag_mut().unwrap(); -tag.set_artist("New Artist".to_string()); -tag.set_title("New Title".to_string()); -tag.save_to_path("song.flac", WriteOptions::default())?; -``` - -**Header Generation (Critical for beetfs):** -```rust -// Generate FLAC header with modified tags WITHOUT writing to file -let mut buffer = Vec::new(); -tag.dump_to(&mut buffer, WriteOptions::default())?; -// `buffer` contains serialized metadata header -``` - ---- - -## 3. Memory Benefits - -### Python Object Overhead - -| Python Type | Size | Notes | -|-------------|------|-------| -| Empty dict | 232 bytes | Base overhead | -| Dict entry | +184 bytes | Per key-value | -| Empty string | 49 bytes | Base overhead | -| Empty list | 56 bytes | Base overhead | -| Small int | 28 bytes | Even for `0` | - -**Current beetfs FileHandler (Python):** -``` -self.path → str → 49 + len(path) bytes -self.real_path → str → 49 + len(path) bytes -self.item → dict → 232 + entries -self.header → bytes → 33 + len(header) -self.music_data → bytes → 33 + len(audio) ← CRITICAL: full file! -self.inf → object → 100+ bytes -───────────────────────────────────────── -TOTAL: ~500 bytes + entire file in RAM -``` - -### Rust Struct Efficiency - -```rust -struct FileHandler { - path: PathBuf, // 24 bytes (ptr+len+cap) - real_path: PathBuf, // 24 bytes - item_id: u64, // 8 bytes - header: Vec, // 24 bytes (ptr+len+cap) + header data - mmap: Mmap, // 24 bytes (NO file data in RAM!) - header_len: u64, // 8 bytes - audio_offset: u64, // 8 bytes -} -// TOTAL: ~120 bytes + header only (audio via mmap) -``` - -### Memory Comparison - -| Scenario | Python | Rust | Savings | -|----------|--------|------|---------| -| 1 file (50MB) | ~50 MB | ~64 KB | **780x** | -| 10 files (50MB each) | ~500 MB | ~640 KB | **780x** | -| 100 files (50MB each) | ~5 GB | ~6.4 MB | **780x** | -| Library scan (1000 files) | **OOM** | ~64 MB | ∞ | - -**Key insight**: Rust can use memory-mapped files (`mmap`) to serve audio data with zero copies, eliminating the need to load files into RAM. - ---- - -## 4. Latency Benefits - -### Python FUSE Bottlenecks - -1. **Dict-to-struct conversion**: Every FUSE callback requires converting Python dicts to C structs -2. **GIL contention**: Single-threaded execution despite multi-core CPUs -3. **GC pauses**: Stop-the-world pauses of 50-2200ms under load -4. **Object allocation**: Creating Python objects for every I/O operation - -### Rust FUSE Advantages - -1. **Zero-cost abstractions**: No runtime overhead for type conversions -2. **No GIL**: True parallelism across all cores -3. **No GC**: Deterministic memory management, no pauses -4. **Stack allocation**: Small objects allocated on stack, not heap - -### Benchmark Data - -| Operation | Python FUSE | Rust FUSE | Improvement | -|-----------|-------------|-----------|-------------| -| File stat | 5-10ms | 0.5-1ms | **10x** | -| Small read | 5-10ms | 0.5-2ms | **5-10x** | -| Large read | 115 MB/s | 260+ MB/s | **2-3x** | -| Metadata lookup | 10ms | <1ms | **10x** | - -### GC Pause Elimination - -``` -Python GC Pauses (measured): -├── P50: ~10ms -├── P95: ~50ms -├── P99: ~320ms -└── Max: ~2200ms (!) - -Rust (no GC): -├── P50: ~0.5ms -├── P95: ~1ms -├── P99: ~2ms -└── Max: ~5ms (deterministic) -``` - ---- - -## 5. Concurrency Benefits - -### Python Threading Limitations - -```python -# Python (current beetfs) -server.multithreaded = 0 # Single-threaded! - -# Even with threading enabled: -# - GIL prevents true parallelism -# - ~8MB per thread -# - OS limits: ~1000-2000 threads max -# - Context switch: 1-10μs (kernel) -``` - -### Rust Async (Tokio) - -```rust -// Rust with Tokio -#[tokio::main] -async fn main() { - // Can handle 100K+ concurrent operations - // - ~2KB per task (4000x less than thread) - // - Work-stealing scheduler - // - Context switch: ~10ns (userspace) -} -``` - -| Metric | Python Threading | Rust Tokio | -|--------|------------------|------------| -| Memory per task | 8 MB | 2 KB | -| Max concurrent | ~1,000 | ~100,000+ | -| Context switch | 1-10μs | ~10ns | -| Parallelism | Blocked by GIL | True multi-core | - ---- - -## 6. Zero-Copy I/O - -### Python (Current) - -```python -# Every read copies data through Python: -self.file_object.read() # syscall → kernel buffer - # kernel buffer → Python bytes object - # Python bytes → FUSE reply buffer -# = 2-3 copies per read -``` - -### Rust (Proposed) - -```rust -// Memory-mapped file + zero-copy reply: -let mmap = unsafe { MmapOptions::new().map(&file)? }; - -fn read(&self, ..., reply: ReplyData) { - // Direct slice from mmap → FUSE kernel - reply.data(&self.mmap[offset..offset+size]); - // = 0 copies (kernel reads directly from mapped pages) -} -``` - -### I/O Comparison - -| Scenario | Python | Rust | Benefit | -|----------|--------|------|---------| -| Serve 50MB file | 50MB copied to RAM | 0 bytes copied | **50MB saved** | -| 100 concurrent reads | 5GB buffers | ~0 (shared mmap) | **5GB saved** | -| Throughput | 115 MB/s | 260+ MB/s | **2.3x faster** | - ---- - -## 7. Real-World Migration Results - -### Case Studies - -| Project | Metric | Python | Rust | Improvement | -|---------|--------|--------|------|-------------| -| API Service | Response time | 200ms | 8ms | **96% faster** | -| Data Pipeline | Processing | 3 hours | 4.5 min | **40x faster** | -| Web Backend | Memory | 1.2 GB | 180 MB | **85% less** | -| Trajectory Lib | Compute | baseline | 10x faster | **10x** | - -### AWS Mountpoint for S3 - -- Built on **fuser** (Rust FUSE) -- Handles **terabits/sec** aggregate throughput -- Production-ready since 2024 -- Validates Rust FUSE at scale - ---- - -## 8. Migration Architecture - -### Proposed Rust beetfs Structure - -``` -beetfs-rs/ -├── Cargo.toml -├── src/ -│ ├── main.rs # Entry point, mount logic -│ ├── lib.rs # Library root -│ ├── fs/ -│ │ ├── mod.rs # FUSE filesystem impl -│ │ ├── tree.rs # Virtual directory tree (FSNode equivalent) -│ │ ├── file.rs # File handler with mmap -│ │ └── stat.rs # File attributes -│ ├── metadata/ -│ │ ├── mod.rs # Metadata overlay logic -│ │ ├── flac.rs # FLAC header generation (using lofty) -│ │ ├── mp3.rs # MP3 ID3 header generation -│ │ └── db.rs # Database interface (SQLite or custom) -│ └── config.rs # Configuration (path templates, etc.) -└── tests/ - ├── fs_tests.rs - └── metadata_tests.rs -``` - -### Key Components - -```rust -// Virtual directory tree (equivalent to FSNode) -pub struct VirtualTree { - root: Arc>, -} - -pub struct DirNode { - dirs: HashMap>>, - files: HashMap, -} - -pub struct FileEntry { - inode: u64, - real_path: PathBuf, - metadata_id: i64, // Database reference -} - -// File handler with memory-mapped audio -pub struct OpenFile { - header: Vec, // Generated header with DB metadata - header_len: usize, - mmap: Mmap, // Memory-mapped original file - audio_offset: usize, // Where audio starts in original -} - -impl OpenFile { - pub fn read(&self, offset: usize, size: usize) -> &[u8] { - if offset < self.header_len { - // Return from generated header (DB metadata) - &self.header[offset..min(offset + size, self.header_len)] - } else { - // Return from mmap (original audio, zero-copy) - let audio_off = offset - self.header_len + self.audio_offset; - &self.mmap[audio_off..audio_off + size] - } - } -} -``` - ---- - -## 9. Migration Effort Estimate - -### Timeline - -| Phase | Duration | Deliverable | -|-------|----------|-------------| -| **1. Prototype** | 1-2 weeks | Basic FUSE mount, read-only | -| **2. Core features** | 2-3 weeks | Metadata overlay, FLAC support | -| **3. Full parity** | 2-3 weeks | MP3, write support, all fields | -| **4. Testing** | 1-2 weeks | Unit tests, integration tests | -| **5. Optimization** | 1-2 weeks | mmap, async, benchmarking | - -**Total: 7-12 weeks** - -### Skill Requirements - -- Rust fundamentals (ownership, borrowing, lifetimes) -- FUSE protocol knowledge (from Python experience) -- Audio metadata formats (FLAC, ID3) -- Async Rust (Tokio) - optional for Phase 5 - ---- - -## 10. Risk Assessment - -### Low Risk ✅ - -| Factor | Why Low Risk | -|--------|--------------| -| FUSE library | fuser is production-proven (AWS) | -| Metadata library | lofty has full mutagen parity | -| Core algorithm | Same logic, different language | -| File format support | FLAC/MP3/OGG all supported | - -### Medium Risk ⚠️ - -| Factor | Mitigation | -|--------|------------| -| Learning curve | Existing Rust experience helps | -| Edge cases | Port Python tests to Rust | -| Async complexity | Start with sync API, add async later | - -### Benefits vs Effort - -``` -Current Python Issues: -├── Memory: OOM on library scan → Fixed by mmap -├── Latency: 200-500ms file open → Fixed by zero-copy -├── GC pauses: 50-2200ms → Eliminated -├── Concurrency: single-threaded → Fixed by async -└── MP3 support: disabled → Implemented properly - -Migration Effort: 7-12 weeks -Expected Lifetime: 5+ years -ROI: Highly positive -``` - ---- - -## 11. Recommendation - -### ✅ **Proceed with Rust Migration** - -**Justification:** -1. **10x memory reduction** via mmap (eliminates OOM) -2. **5-10x latency improvement** (eliminates blocking reads) -3. **GC pauses eliminated** (deterministic performance) -4. **100x concurrency** improvement (Tokio async) -5. **Production-proven** ecosystem (fuser + lofty) -6. **Reasonable effort** (7-12 weeks) - -### Next Steps - -1. **Set up Rust project** with fuser and lofty dependencies -2. **Port FSNode** to Rust VirtualTree -3. **Implement basic FUSE** operations (read, getattr, readdir) -4. **Add metadata overlay** with lofty for FLAC -5. **Add mmap** for zero-copy audio serving -6. **Benchmark** against Python implementation -7. **Add MP3/OGG** support -8. **Add async** with Tokio (optional) - -### Dependencies - -```toml -[dependencies] -fuser = "0.17" -lofty = "0.21" -memmap2 = "0.9" -tokio = { version = "1", features = ["full"], optional = true } -rusqlite = "0.31" # For beets DB compatibility -``` diff --git a/docs/v2/architecture.md b/docs/v2/architecture.md deleted file mode 100644 index 56f6767..0000000 --- a/docs/v2/architecture.md +++ /dev/null @@ -1,1151 +0,0 @@ -# MusicFS: Design Doc - -**Authors:** [TBD] -**Status:** Draft -**Last Updated:** 2026-05-12 -**Reviewers:** [TBD] -**Approvers:** [TBD] -**Requirements:** [requirements.md](requirements.md) - ---- - -[TOC] - ---- - -## 1. Abstract - -MusicFS is a read-only FUSE filesystem that presents music libraries organized -by metadata (artist/album/track) rather than physical file paths. It supports -multiple origin storage backends (local, NFS, S3, SFTP), provides intelligent -caching with delta synchronization, and exposes a plugin architecture for -extensibility. - -The system addresses limitations of the existing beetfs implementation: -- O(N) mount time → O(1) lazy loading -- Full file in RAM → streaming with content-addressable chunks -- Single origin → federated multi-origin with failover -- No offline support → cache-first with graceful degradation - -Target users are media enthusiasts with large music collections (100K-10M+ -tracks) distributed across multiple storage systems who want a unified, -metadata-organized view without modifying original files. - ---- - -## 2. Background - -### 2.1 Current State - -The existing beetfs implementation is a Python 2.7 FUSE plugin for beets that: -- Presents a virtual filesystem organized by metadata templates -- Overlays metadata from beets database onto file headers -- Supports metadata writes back to the beets database - -### 2.2 Pain Points - -| Problem | Impact | -|---------|--------| -| O(N) mount time (5-120s for large libraries) | Unusable for large collections | -| Loads entire file into RAM on open | OOM risk, 50-100MB per file | -| Python GIL limits concurrency | Poor performance under load | -| No caching between sessions | Repeated work on every mount | -| Single local origin only | Can't federate across storage | -| No offline support | Unusable without origin access | -| Critical bugs (nested methods, tree building) | Non-functional | - -### 2.3 Related Systems - -| System | Relationship | -|--------|--------------| -| [beets](https://beets.io/) | Source of inspiration; potential import source | -| [rclone mount](https://rclone.org/commands/rclone_mount/) | Similar FUSE + remote storage; no metadata organization | -| [Plex/Jellyfin](https://jellyfin.org/) | Media servers with metadata; not filesystem-based | - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -| ID | Goal | Success Metric | -|----|------|----------------| -| G1 | O(1) mount time | <500ms regardless of library size | -| G2 | Minimal memory footprint | <50MB idle, <500MB peak | -| G3 | Support multiple origins | ≥2 origins with automatic failover | -| G4 | Offline-first operation | Serve cached data when origin unavailable | -| G5 | Delta synchronization | >90% bandwidth reduction vs full sync | -| G6 | Plugin extensibility | Support custom origins, formats, metadata sources | -| G7 | Full-text search | Sub-second search across 1M+ tracks | - -### 3.2 Design Requirements - -The following quantitative requirements drive architectural decisions. Full -specification in [requirements.md](requirements.md). - -#### 3.2.1 Latency Requirements - -| Operation | Target | Maximum | Requirement | -|-----------|--------|---------|-------------| -| `stat()` cached | <1ms | 5ms | NFR-1.1 | -| `readdir()` cached | <10ms | 50ms | NFR-1.2 | -| `open()` cached | <5ms | 20ms | NFR-1.3 | -| `read()` cached | <1ms | 5ms | NFR-1.4 | -| `read()` cache miss (local) | <50ms | 200ms | NFR-1.5 | -| `read()` cache miss (remote) | <200ms | 1000ms | NFR-1.6 | -| Mount completion | <100ms | 500ms | NFR-1.7 | -| Search query (1M files) | <500ms | 1000ms | FR-14 | - -**Design Response:** -- Lazy loading eliminates mount-time I/O → O(1) mount -- In-memory LRU cache for hot metadata → <1ms stat -- SQLite with indexes → O(log n) lookups -- Async I/O via tokio → non-blocking operations - -#### 3.2.2 Throughput Requirements - -| Metric | Target | Requirement | -|--------|--------|-------------| -| Sequential read (cached) | >500 MB/s | NFR-2.1 | -| Sequential read (local origin) | >200 MB/s | NFR-2.2 | -| Metadata ops/sec | >1000 | NFR-2.3 | -| Concurrent file handles | >1000 | NFR-2.4 | - -**Design Response:** -- Memory-mapped chunk files → kernel-optimized reads -- No GIL (Rust) → true parallelism -- Async FUSE ops → handle many concurrent requests - -#### 3.2.3 Scalability Requirements - -| Metric | Target | Stretch | Requirement | -|--------|--------|---------|-------------| -| Library size | 1M files | 10M files | NFR-3.1, NFR-3.5 | -| Directory entries | 100K | 1M | NFR-3.2 | -| Concurrent clients | 10 | 100+ | NFR-3.6 | -| Mount time scaling | O(1) | O(1) | NFR-3.3 | - -**Design Response:** -- Lazy tree loading → mount time independent of size -- SQLite indexes → O(log n) regardless of scale -- Streaming readdir → handle large directories -- Connection pooling → support many clients - -#### 3.2.4 Resource Requirements - -| Resource | Idle | Active (1K files) | Peak | Requirement | -|----------|------|-------------------|------|-------------| -| Memory | <50 MB | <200 MB | <500 MB | NFR-4.1-4.3 | -| Per-file overhead | - | <1 KB | - | NFR-4.4 | -| Metadata cache | - | 100 MB default | configurable | NFR-5.1 | -| Content cache | - | 10 GB default | configurable | NFR-5.2 | - -**Design Response:** -- Streaming reads → never load full file in memory -- Content-addressed chunks → bounded cache with LRU eviction -- Metadata in SQLite → minimal per-file RAM overhead - -#### 3.2.5 Efficiency Requirements - -| Metric | Target | Requirement | -|--------|--------|-------------| -| Delta sync bandwidth reduction | >90% | NFR-6.4 | -| Cache hit rate (warm) | >95% | Derived | -| Deduplication ratio | >10% typical | FR-20 | - -**Design Response:** -- CDC chunking → stable boundaries, minimal re-transfer -- Content-addressable storage → automatic deduplication -- Prefetch engine → anticipate access patterns - -#### 3.2.6 Reliability Requirements - -| Scenario | Behavior | Requirement | -|----------|----------|-------------| -| Origin offline | Serve cached data | NFR-7.1 | -| Network failure | Graceful degradation, no crash | NFR-7.2 | -| Failed operation | Retry with backoff (100ms, 500ms, 2s) | NFR-7.3 | -| Malformed audio | Skip file, log error, don't crash | NFR-7.4 | -| Chunk corruption | Detect via checksum, re-fetch | NFR-8.1, NFR-8.4 | -| Interrupted sync | Resume from last good state | NFR-8.3 | -| Unclean unmount | Recover on next mount | NFR-8.2 | - -**Design Response:** -- Cache-first architecture → offline operation by default -- Origin federation with health checks → survive single origin failure -- xxHash checksums on all chunks → detect corruption -- WAL mode SQLite → ACID transactions, crash recovery - -#### 3.2.7 Concurrent Access Requirements - -| Scenario | Limit | Latency Impact | Requirement | -|----------|-------|----------------|-------------| -| Simultaneous open files | >1000 handles | None | NFR-2.4 | -| Parallel read ops | >100 concurrent | <2x p99 latency | Derived | -| Multiple clients | >10 (target 100+) | Linear degradation | NFR-3.6 | -| Readdir during sync | No blocking | Serve stale if needed | FR-9.2 | - -**Design Response:** -- Async I/O (tokio) → non-blocking operations -- No GIL → true parallelism across cores -- Read-write locks on cache → readers don't block readers -- Stale-while-revalidate → serve cached during refresh - -### 3.3 Non-Goals - -| ID | Non-Goal | Rationale | -|----|----------|-----------| -| NG1 | Write to origin files | Read-only by design; preserves originals | -| NG2 | Transcoding | Out of scope for MVP; plugin possible later | -| NG3 | Video file support | Focus on audio; deferred to future | -| NG4 | Distributed/clustered mode | Single-node for MVP; architecture supports later | -| NG5 | Mobile app | CLI/daemon only; filesystem interface | - ---- - -## 4. Proposed Design - -### 4.1 High-Level Architecture - -```plantuml -@startuml -!theme plain -skinparam componentStyle rectangle - -package "User Space" { - [Media Players\n(mpv, VLC, Plex)] as Apps - - package "MusicFS Daemon" { - [FUSE Interface] as FUSE - [Control API] as Control - [Metrics] as Metrics - - package "Core Services" { - [Virtual Path\nResolver] as VPR - [Event Bus] as Events - [Search Engine\n(tantivy)] as Search - } - - package "Plugin Host" { - [Origin\nPlugins] as OriginPlugins - [Metadata\nPlugins] as MetaPlugins - [Format\nPlugins] as FormatPlugins - } - - package "Storage Layer" { - [Content-Addressable\nStore (CAS)] as CAS - database "SQLite\n(metadata)" as SQLite - database "sled\n(chunks)" as Sled - } - - [Origin\nFederation] as Federation - } -} - -package "Origins (Read-Only)" { - [Local FS] as Local - [NFS] as NFS - [S3] as S3 - [SFTP] as SFTP -} - -Apps --> FUSE : POSIX -FUSE --> VPR -VPR --> Events -VPR --> Search -VPR --> CAS -CAS --> SQLite -CAS --> Sled -VPR --> Federation -Federation --> OriginPlugins -OriginPlugins --> Local -OriginPlugins --> NFS -OriginPlugins --> S3 -OriginPlugins --> SFTP -Control --> Events -Metrics --> Events - -@enduml -``` - -### 4.2 Component Overview - -| Component | Responsibility | Technology | -|-----------|---------------|------------| -| FUSE Interface | Translate POSIX ops to internal calls | fuser (Rust) | -| Virtual Path Resolver | Map virtual ↔ real paths | Custom | -| Event Bus | Decouple components, enable observability | tokio broadcast | -| Search Engine | Full-text metadata search | tantivy | -| Plugin Host | Load/manage plugins | Native + WASM | -| CAS | Content-addressed chunk storage | Custom + sled | -| Origin Federation | Multi-origin routing with failover | Custom | - -### 4.3 Detailed Design - -#### 4.3.1 Virtual Path Resolution - -The resolver maps metadata-based virtual paths to real origin paths. - -```plantuml -@startuml -!theme plain - -participant "FUSE" as F -participant "VirtualPathResolver" as VPR -participant "MetadataIndex" as MI -participant "TreeCache" as TC -participant "OriginFederation" as OF - -F -> VPR : lookup("/Metallica/72 Seasons/01.flac") -VPR -> TC : get_cached(path) -alt cache hit - TC --> VPR : CachedEntry -else cache miss - VPR -> MI : query(artist="Metallica", album="72 Seasons", track=1) - MI --> VPR : FileRecord { origin_id, real_path, metadata } - VPR -> TC : store(path, entry) -end -VPR -> OF : resolve_origin(origin_id) -OF --> VPR : OriginHandle -VPR --> F : ResolvedPath { origin, real_path, inode } - -@enduml -``` - -**Path Template Grammar:** -``` -template = segment ("/" segment)* -segment = (literal | variable)+ -variable = "$" identifier -identifier = "artist" | "album" | "title" | "track" | "year" | "genre" - | "format" | "format_upper" | "disc" -``` - -**Default Template:** -``` -$artist/$album ($year) [$format_upper]/$track - $title.$format -``` - -#### 4.3.2 Content-Addressable Store (CAS) - -All file content is stored as content-addressed chunks, enabling deduplication -and efficient delta sync. - -```plantuml -@startuml -!theme plain - -package "Content-Addressable Store" { - component "Chunk Manager" as CM - component "CDC Chunker\n(FastCDC)" as CDC - component "Hash Index\n(xxHash64)" as Hash - - database "Chunk Files\n~/.cache/musicfs/chunks/" as Chunks - database "Index DB\n(sled)" as Index - - CM --> CDC : chunk data - CDC --> Hash : compute hash - Hash --> Index : store hash → location - CM --> Chunks : write chunk file -} - -note right of CDC - Avg chunk: 64KB - Min: 16KB, Max: 256KB - Stable boundaries for delta sync -end note - -@enduml -``` - -**Chunk Storage Layout:** -``` -~/.cache/musicfs/ -├── chunks/ -│ ├── aa/ -│ │ ├── aa1b2c3d4e5f6789... (64KB chunk) -│ │ └── aa9f8e7d6c5b4a32... -│ ├── ab/ -│ └── ... (256 subdirs for distribution) -├── metadata.db (SQLite: file metadata, tree cache) -├── search.idx/ (tantivy: full-text index) -└── chunks.sled/ (sled: hash → chunk location) -``` - -#### 4.3.3 Origin Federation - -Multiple origins are managed with priority-based routing and health tracking. - -```plantuml -@startuml -!theme plain - -participant "VirtualPathResolver" as VPR -participant "OriginFederation" as OF -participant "HealthChecker" as HC -participant "Origin[Local]" as O1 -participant "Origin[NFS]" as O2 -participant "Origin[S3]" as O3 - -VPR -> OF : read(real_path, offset, size) -OF -> OF : select_origin(priority, health) - -alt Origin[Local] healthy (pri=1) - OF -> O1 : read() - O1 --> OF : data -else Origin[Local] unhealthy, try NFS (pri=2) - OF -> O2 : read() - alt success - O2 --> OF : data - else failure - OF -> O3 : read() - O3 --> OF : data - end -end - -OF --> VPR : data - -note over HC - Background health checks - every 30s per origin -end note - -@enduml -``` - -**Origin Configuration:** -```toml -[[origins]] -id = "local" -type = "local" -path = "/mnt/nas/music" -priority = 1 - -[[origins]] -id = "backup" -type = "s3" -bucket = "music-backup" -priority = 2 -``` - -#### 4.3.4 Plugin System - -Plugins extend functionality without modifying core code. - -```plantuml -@startuml -!theme plain - -interface "Plugin" { - +name(): String - +version(): Version - +init(config) - +shutdown() -} - -interface "OriginPlugin" { - +list_dir(path): Vec - +read(path, offset, size): Vec - +stat(path): FileStat - +watch(path, callback): WatchHandle -} - -interface "MetadataPlugin" { - +extract(data, format): Metadata - +can_handle(format): bool -} - -interface "FormatPlugin" { - +extensions(): Vec - +parse_header(data): AudioHeader - +synthesize_header(metadata): Vec -} - -Plugin <|-- OriginPlugin -Plugin <|-- MetadataPlugin -Plugin <|-- FormatPlugin - -class "LocalFSPlugin" implements OriginPlugin -class "S3Plugin" implements OriginPlugin -class "SymphoniaPlugin" implements MetadataPlugin -class "FlacPlugin" implements FormatPlugin -class "Mp3Plugin" implements FormatPlugin - -@enduml -``` - -**Plugin Loading:** -1. **Built-in:** Compiled into binary (Local, S3, SFTP, symphonia) -2. **Native:** Dynamic libraries (`.so`/`.dylib`) loaded at runtime -3. **WASM:** Sandboxed plugins via wasmtime (future) - -#### 4.3.5 Data Flow: Read Operation - -```plantuml -@startuml -!theme plain - -|FUSE| -start -:receive read(path, offset, size); - -|VirtualPathResolver| -:resolve virtual path to real path; -:lookup file metadata; - -|CAS| -:compute chunk range for [offset, offset+size]; -if (all chunks cached?) then (yes) - :read from local chunk files; -else (no) - |OriginFederation| - :select healthy origin by priority; - :fetch missing byte range; - |CAS| - :chunk fetched data (CDC); - :store chunks by hash; - :update chunk manifest; -endif - -|EventBus| -:emit FileAccessed event; - -|FUSE| -:return data to application; -stop - -@enduml -``` - -#### 4.3.6 Data Schema - -**Metadata Index (SQLite):** -```sql -CREATE TABLE files ( - id INTEGER PRIMARY KEY, - origin_id TEXT NOT NULL, - real_path TEXT NOT NULL, - virtual_path TEXT NOT NULL, - - -- Metadata (see FR-6 in requirements.md) - title TEXT, - artist TEXT, - album TEXT, - album_artist TEXT, - genre TEXT, - year INTEGER, - track INTEGER, - disc INTEGER, - duration_ms INTEGER, - bitrate INTEGER, - sample_rate INTEGER, - format TEXT, - - -- Sync state - origin_mtime INTEGER, - origin_size INTEGER, - content_hash TEXT, - chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)] - last_sync INTEGER, - - UNIQUE(origin_id, real_path) -); - -CREATE INDEX idx_virtual ON files(virtual_path); -CREATE INDEX idx_artist_album ON files(artist, album); -CREATE INDEX idx_content_hash ON files(content_hash); - -CREATE TABLE artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER REFERENCES files(id), - art_type TEXT, -- 'front', 'back' - chunk_hash TEXT, -- reference to CAS - width INTEGER, - height INTEGER, - UNIQUE(file_id, art_type) -); - -CREATE TABLE collections ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE, - query_json TEXT, -- smart collection query - created_at INTEGER -); -``` - -#### 4.3.7 Control API - -**Protocol Choice: gRPC over Unix Socket** - -| Criterion | JSON-RPC | gRPC | Winner | -|-----------|----------|------|--------| -| Type safety | Runtime validation | Compile-time (protobuf) | gRPC | -| Schema evolution | Ad-hoc versioning | Built-in field numbering | gRPC | -| Streaming | Requires WebSocket/polling | Native bidirectional | gRPC | -| Client generation | Manual per language | Auto-gen 10+ languages | gRPC | -| Performance | JSON parse overhead | Binary, zero-copy | gRPC | -| Debugging | Human-readable | Needs tooling (grpcurl) | JSON-RPC | -| Simplicity | Lower barrier | Requires protoc | JSON-RPC | - -**Decision:** gRPC for primary API. Human-readable debugging via `grpcurl` and CLI wrapper. - -**Rationale:** -1. **Event streaming** - Native server-streaming for real-time sync/cache events without polling -2. **Multi-language clients** - Auto-generated clients for Python (beets integration), Go, Node.js -3. **Schema evolution** - Protobuf field numbering allows backward-compatible API changes -4. **Performance** - Binary encoding avoids JSON serialization overhead on high-frequency stat() calls - ---- - -**Protocol Buffer Definitions:** - -```protobuf -syntax = "proto3"; -package musicfs.v1; - -// ============================================================================ -// Core Services -// ============================================================================ - -service MusicFS { - // Daemon lifecycle - rpc GetStatus(Empty) returns (StatusResponse); - rpc Shutdown(ShutdownRequest) returns (Empty); - - // Cache management - rpc GetCacheStats(Empty) returns (CacheStats); - rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse); - rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress); - - // Origin management - rpc ListOrigins(Empty) returns (OriginsResponse); - rpc GetOriginHealth(OriginRequest) returns (OriginHealth); - rpc RescanOrigin(OriginRequest) returns (stream SyncProgress); - - // Search - rpc Search(SearchRequest) returns (SearchResponse); - rpc SearchStream(SearchRequest) returns (stream SearchResult); - - // Events (server-streaming) - rpc SubscribeEvents(EventFilter) returns (stream Event); -} - -// ============================================================================ -// Messages: Daemon -// ============================================================================ - -message Empty {} - -message StatusResponse { - string version = 1; - uint64 uptime_seconds = 2; - string mount_point = 3; - MountState state = 4; - uint32 open_file_handles = 5; - uint64 fuse_ops_total = 6; -} - -enum MountState { - MOUNT_STATE_UNKNOWN = 0; - MOUNT_STATE_MOUNTING = 1; - MOUNT_STATE_READY = 2; - MOUNT_STATE_SYNCING = 3; - MOUNT_STATE_DEGRADED = 4; // Some origins unavailable - MOUNT_STATE_UNMOUNTING = 5; -} - -message ShutdownRequest { - bool force = 1; // Skip graceful drain - uint32 drain_timeout_ms = 2; // Max wait for in-flight ops (default: 5000) -} - -// ============================================================================ -// Messages: Cache -// ============================================================================ - -message CacheStats { - // Hit/miss counters - uint64 hits = 1; - uint64 misses = 2; - double hit_rate = 3; - - // Storage - uint64 chunks_stored = 4; - uint64 chunks_unique = 5; // After deduplication - double dedup_ratio = 6; // Space saved by dedup - uint64 size_bytes = 7; - uint64 size_limit_bytes = 8; - - // Metadata cache - uint64 metadata_entries = 9; - uint64 metadata_bytes = 10; - - // Per-tier breakdown - TierStats l1_metadata = 11; - TierStats l2_headers = 12; - TierStats l3_chunks = 13; -} - -message TierStats { - uint64 entries = 1; - uint64 bytes = 2; - uint64 evictions = 3; -} - -message ClearCacheRequest { - optional string origin_id = 1; // Empty = all origins - CacheTier tier = 2; // Which tier to clear - bool dry_run = 3; // Report what would be cleared -} - -enum CacheTier { - CACHE_TIER_ALL = 0; - CACHE_TIER_METADATA = 1; - CACHE_TIER_HEADERS = 2; - CACHE_TIER_CHUNKS = 3; -} - -message ClearCacheResponse { - uint64 entries_cleared = 1; - uint64 bytes_freed = 2; -} - -message PrefetchRequest { - repeated string paths = 1; // Virtual paths to prefetch - optional string query = 2; // Or search query - PrefetchStrategy strategy = 3; -} - -enum PrefetchStrategy { - PREFETCH_METADATA_ONLY = 0; // Just stat info - PREFETCH_HEADERS = 1; // Metadata + audio headers - PREFETCH_FULL = 2; // Complete file content -} - -message PrefetchProgress { - string path = 1; - uint64 bytes_fetched = 2; - uint64 bytes_total = 3; - bool complete = 4; - optional string error = 5; -} - -// ============================================================================ -// Messages: Origins -// ============================================================================ - -message OriginRequest { - string origin_id = 1; -} - -message OriginsResponse { - repeated OriginInfo origins = 1; -} - -message OriginInfo { - string id = 1; - string origin_type = 2; // "local", "sftp", "s3", "smb" - string display_name = 3; - OriginHealth health = 4; - uint64 file_count = 5; - uint64 total_bytes = 6; - int64 last_sync_unix = 7; -} - -message OriginHealth { - HealthStatus status = 1; - uint32 latency_ms = 2; - optional string error_message = 3; - int64 last_check_unix = 4; -} - -enum HealthStatus { - HEALTH_UNKNOWN = 0; - HEALTH_HEALTHY = 1; - HEALTH_DEGRADED = 2; // Slow but working - HEALTH_UNHEALTHY = 3; // Connection failed -} - -message SyncProgress { - string origin_id = 1; - SyncPhase phase = 2; - uint64 files_scanned = 3; - uint64 files_changed = 4; - uint64 files_total = 5; - uint64 bytes_transferred = 6; - optional string current_file = 7; - bool complete = 8; - optional string error = 9; -} - -enum SyncPhase { - SYNC_PHASE_SCANNING = 0; - SYNC_PHASE_COMPARING = 1; - SYNC_PHASE_FETCHING = 2; - SYNC_PHASE_INDEXING = 3; - SYNC_PHASE_COMPLETE = 4; -} - -// ============================================================================ -// Messages: Search -// ============================================================================ - -message SearchRequest { - string query = 1; // Full-text query - uint32 limit = 2; // Max results (default: 100) - uint32 offset = 3; // Pagination - repeated string fields = 4; // Restrict to fields: artist, album, title - optional string origin_id = 5; // Filter by origin -} - -message SearchResponse { - repeated SearchResult results = 1; - uint64 total_matches = 2; - uint32 query_time_ms = 3; -} - -message SearchResult { - string virtual_path = 1; - string title = 2; - string artist = 3; - string album = 4; - float score = 5; // Relevance score - map highlights = 6; // Field -> highlighted snippet -} - -// ============================================================================ -// Messages: Events -// ============================================================================ - -message EventFilter { - repeated EventType types = 1; // Empty = all events - optional string origin_id = 2; // Filter by origin -} - -enum EventType { - EVENT_TYPE_ALL = 0; - EVENT_TYPE_FILE_ADDED = 1; - EVENT_TYPE_FILE_REMOVED = 2; - EVENT_TYPE_FILE_MODIFIED = 3; - EVENT_TYPE_ORIGIN_CONNECTED = 4; - EVENT_TYPE_ORIGIN_DISCONNECTED = 5; - EVENT_TYPE_SYNC_STARTED = 6; - EVENT_TYPE_SYNC_COMPLETED = 7; - EVENT_TYPE_CACHE_EVICTION = 8; -} - -message Event { - EventType type = 1; - int64 timestamp_unix = 2; - string origin_id = 3; - optional string path = 4; - map metadata = 5; -} -``` - ---- - -**CLI Interface** (wraps gRPC client): - -```bash -musicfs mount /mnt/music # Mount filesystem -musicfs status # GetStatus() -musicfs cache stats # GetCacheStats() -musicfs cache clear --origin=local # ClearCache(origin_id="local") -musicfs search "metallica heavy" # Search(query="metallica heavy") -musicfs origin list # ListOrigins() -musicfs origin rescan local # RescanOrigin() with progress -musicfs events --type=file_added # SubscribeEvents() stream -``` - -**Debugging:** -```bash -# Direct gRPC inspection via grpcurl -grpcurl -unix /run/musicfs.sock musicfs.v1.MusicFS/GetStatus -grpcurl -unix /run/musicfs.sock -d '{"query":"metallica"}' musicfs.v1.MusicFS/Search -``` - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -| Concern | Mitigation | -|---------|------------| -| Credential storage | Use system keyring (secret-service) or env vars; never in config file | -| Credential exposure | Redact from logs; exclude from `/proc/cmdline` | -| Cache at rest | Optional encryption via age/libsodium (P3 requirement) | -| Plugin sandboxing | WASM plugins run in wasmtime sandbox; native plugins require trust | -| Access control | Respect origin permissions; run as unprivileged user | -| No PII handling | Filesystem metadata only; no user data collected | - -### 5.2 Observability - -**Metrics (Prometheus format):** -``` -musicfs_fuse_ops_total{op="read"} 152341 -musicfs_fuse_ops_total{op="readdir"} 8234 -musicfs_fuse_latency_seconds{op="read",quantile="0.99"} 0.004 -musicfs_cache_hits_total 142107 -musicfs_cache_misses_total 10234 -musicfs_cache_size_bytes 5368709120 -musicfs_origin_health{origin="local"} 1 -musicfs_origin_health{origin="s3"} 0 -musicfs_sync_files_changed{origin="local"} 15 -``` - -**Logging Levels:** -| Level | Content | -|-------|---------| -| ERROR | Unrecoverable failures, data corruption | -| WARN | Recoverable failures, origin timeouts | -| INFO | Mount/unmount, sync completion, config reload | -| DEBUG | Cache hits/misses, origin selection | -| TRACE | Individual FUSE operations, chunk I/O | - -**Golden Signals Dashboard:** -1. **Latency:** p50/p95/p99 for read, stat, readdir -2. **Traffic:** FUSE ops/sec, bytes read/sec -3. **Errors:** Origin failures, cache corruption -4. **Saturation:** Cache fullness, open file handles - -### 5.3 Scalability & Performance - -**Expected Load:** -| Metric | Target | Maximum | -|--------|--------|---------| -| Library size | 1M files | 10M files | -| Concurrent clients | 10 | 100+ | -| FUSE ops/sec | 1,000 | 10,000 | -| Read throughput | 500 MB/s | 1 GB/s | - -**Scaling Strategy:** -- **Horizontal:** Not supported (single daemon per mountpoint) -- **Vertical:** Increase cache size, add origins - -**Resource Requirements:** -| Resource | Minimum | Recommended | -|----------|---------|-------------| -| CPU | 1 core | 4 cores | -| RAM | 256 MB | 2 GB | -| Disk (cache) | 1 GB | 50 GB | -| Network | 10 Mbps | 1 Gbps | - -### 5.4 Testing Plan - -| Test Type | Scope | Tools | -|-----------|-------|-------| -| Unit | Individual components | cargo test | -| Integration | Component interaction | cargo test --features integration | -| E2E | Full FUSE operations | pytest + real mount | -| Performance | Latency, throughput | criterion.rs, custom benchmarks | -| Stress | High load, large libraries | locust, custom generators | -| Chaos | Origin failures, network issues | toxiproxy | - -**Test Matrix:** -``` -Origins: [local, s3, sftp] × [healthy, degraded, offline] -Cache: [cold, warm, full] -Library: [100, 10K, 1M, 10M] files -Operations: [mount, readdir, stat, read, search] -``` - ---- - -## 6. Alternatives Considered - -### 6.1 Alternative A: Extend beetfs (Python) - -**Description:** Fix bugs in existing beetfs, add features incrementally. - -**Rejected Because:** -- Python GIL fundamentally limits concurrency -- Python 2.7 EOL; migration to Python 3 substantial -- Architecture (full file in RAM) requires rewrite anyway -- No async I/O support in fuse-python - -### 6.2 Alternative B: Use rclone mount - -**Description:** Use rclone's FUSE mount with VFS caching. - -**Rejected Because:** -- No metadata-based virtual path organization -- No metadata overlay functionality -- Limited plugin extensibility -- Would require forking and heavy modification - -### 6.3 Alternative C: Build as Plex/Jellyfin Plugin - -**Description:** Extend existing media server with virtual filesystem view. - -**Rejected Because:** -- Tied to specific media server -- Not a true filesystem (no POSIX interface) -- Heavy runtime dependency -- Different use case (streaming vs filesystem) - -### 6.4 Alternative D: Go Implementation - -**Description:** Implement in Go using go-fuse. - -**Considered Trade-offs:** -| Aspect | Rust | Go | -|--------|------|-----| -| Memory safety | Compile-time | GC pauses | -| Concurrency | async/await, no GC | goroutines, GC | -| FUSE library | fuser (mature) | go-fuse (mature) | -| Learning curve | Steeper | Gentler | -| Binary size | Smaller | Larger | - -**Decision:** Rust chosen for zero-cost abstractions, no GC pauses during I/O, -and better fit for systems programming. - ---- - -## 7. Implementation Plan - -### 7.1 Phase 1: MVP (4.5 weeks) - -**Goal:** Basic functional filesystem with single origin. - -| Week | Deliverables | -|------|--------------| -| 1 | Project setup, FUSE skeleton, local origin plugin | -| 2 | Metadata extraction (symphonia), SQLite schema | -| 3 | Virtual path resolver, tree cache, basic readdir/stat/read | -| 4 | CAS implementation, chunk caching, LRU eviction | -| 4b | Origin→CAS connector (ContentFetcher), cache-miss handling | - -**Exit Criteria:** -- Mount and browse local music library -- Play audio files through mounted filesystem -- Cache persists across restarts - -### 7.2 Phase 2: Delta Sync & Multi-Origin (3 weeks) - -**Goal:** Efficient synchronization and origin federation. - -| Week | Deliverables | -|------|--------------| -| 5 | CDC chunking (FastCDC), delta detection | -| 6 | Origin federation, priority routing, health checks | -| 7 | S3 origin plugin, SFTP origin plugin | - -**Exit Criteria:** -- Delta sync achieves >90% bandwidth reduction -- Automatic failover between origins -- Remote origins functional - -### 7.3 Phase 3: Search & Smart Features (2 weeks) - -**Goal:** Full-text search and intelligent caching. - -| Week | Deliverables | -|------|--------------| -| 8 | tantivy integration, search indexing, `/.search/` virtual dir | -| 9 | Smart collections, prefetch engine, access pattern learning | - -**Exit Criteria:** -- Search returns results in <1s for 1M tracks -- Prefetch reduces cache misses by >50% - -### 7.4 Phase 4: Plugin System & Polish (2 weeks) - -**Goal:** Extensibility and production readiness. - -| Week | Deliverables | -|------|--------------| -| 10 | Plugin host, plugin API stabilization, example plugins | -| 11 | Control API, metrics, documentation, packaging | - -**Exit Criteria:** -- Custom origin plugin loadable at runtime -- Prometheus metrics exported -- systemd service functional - -### 7.5 Rollout Strategy - -```plantuml -@startuml -!theme plain - -[*] --> Alpha -Alpha --> Beta : Internal testing complete -Beta --> GA : Community testing complete - -state Alpha { - [*] --> DevTesting - DevTesting --> DogFood : Core features work -} - -state Beta { - [*] --> LimitedRelease - LimitedRelease --> PublicBeta : No critical bugs -} - -state GA { - [*] --> Stable -} - -note right of Alpha : 2-4 weeks\nDevelopers only -note right of Beta : 4-8 weeks\nEarly adopters -note right of GA : Stable releases - -@enduml -``` - -**Feature Flags:** -```toml -[features] -search_enabled = true -smart_collections = false # Beta -wasm_plugins = false # Experimental -``` - -**Rollback:** Binary replacement + cache clear; no data migration needed. - ---- - -## 8. Glossary & References - -### 8.1 Glossary - -| Term | Definition | -|------|------------| -| **CAS** | Content-Addressable Store; data stored/retrieved by hash | -| **CDC** | Content-Defined Chunking; chunking with stable boundaries | -| **FUSE** | Filesystem in Userspace; kernel interface for user-space filesystems | -| **Origin** | Source storage backend (local, S3, NFS, etc.) | -| **Virtual Path** | Metadata-derived path shown to users | -| **Real Path** | Actual path on origin storage | - -### 8.2 References - -| Document | Link | -|----------|------| -| Requirements Specification | [requirements.md](requirements.md) | -| beetfs (Original) | [beetsplug/beetFs.py](../../beetsplug/beetFs.py) | -| beetfs Features | [v1/features.md](../v1/features.md) | -| fuser (Rust FUSE) | https://github.com/cberner/fuser | -| tantivy (Search) | https://github.com/quickwit-oss/tantivy | -| symphonia (Audio) | https://github.com/pdrat/symphonia | -| FastCDC | https://github.com/nlfiedler/fastcdc-rs | -| wasmtime | https://wasmtime.dev/ | - -### 8.3 Dependencies - -| Crate | Version | Purpose | -|-------|---------|---------| -| fuser | 0.14+ | FUSE interface | -| tokio | 1.x | Async runtime | -| rusqlite | 0.31+ | SQLite bindings | -| sled | 0.34+ | Embedded key-value store | -| tantivy | 0.21+ | Full-text search | -| symphonia | 0.5+ | Audio metadata extraction | -| fastcdc | 3.x | Content-defined chunking | -| xxhash-rust | 0.8+ | Fast hashing | -| serde | 1.x | Serialization | -| toml | 0.8+ | Configuration | -| tracing | 0.1+ | Logging/instrumentation | -| metrics | 0.22+ | Prometheus metrics | diff --git a/docs/v2/development-plan.md b/docs/v2/development-plan.md deleted file mode 100644 index a10beb6..0000000 --- a/docs/v2/development-plan.md +++ /dev/null @@ -1,1387 +0,0 @@ -# MusicFS Development Plan - -**Version**: 1.0 -**Date**: 2026-05-12 -**Status**: Draft -**Prerequisites**: [requirements.md](requirements.md), [architecture.md](architecture.md) - ---- - -## 1. Overview - -This plan breaks down the 11-week implementation into specific deliverables with Rust modules, test requirements, and acceptance criteria mapped to requirements. - -### 1.1 Project Structure - -``` -musicfs/ -├── Cargo.toml # Workspace root -├── proto/ -│ └── musicfs.proto # gRPC definitions -├── crates/ -│ ├── musicfs-core/ # Core types, traits, errors -│ ├── musicfs-fuse/ # FUSE filesystem implementation -│ ├── musicfs-cache/ # Three-tier caching (L1/L2/L3) -│ ├── musicfs-cas/ # Content-addressable storage -│ ├── musicfs-sync/ # Delta sync, CDC chunking -│ ├── musicfs-origins/ # Origin plugins (local, s3, sftp) -│ ├── musicfs-metadata/ # Audio metadata extraction -│ ├── musicfs-search/ # Full-text search (tantivy) -│ ├── musicfs-plugins/ # Plugin host (native + WASM) -│ ├── musicfs-grpc/ # gRPC control API -│ └── musicfs-cli/ # CLI binary -├── tests/ -│ ├── integration/ # Cross-crate integration tests -│ └── e2e/ # End-to-end FUSE tests -└── benches/ # Criterion benchmarks -``` - -### 1.2 Dependency Graph - -``` - musicfs-cli - │ - ┌──────────┼──────────┐ - │ │ │ - ▼ ▼ ▼ - musicfs-grpc musicfs-fuse musicfs-search - │ │ │ - └────┬─────┴───────────────┘ - │ - ▼ - musicfs-core - / | \ - / | \ - ▼ ▼ ▼ - musicfs-cache musicfs-origins musicfs-metadata - │ │ - ▼ │ - musicfs-cas ◄───────┘ - │ - ▼ - musicfs-sync -``` - ---- - -## 2. Phase 1: MVP (Weeks 1-4b) - -**Goal**: Basic functional filesystem with single local origin. - -**Requirements Covered**: FR-1, FR-2, FR-3, FR-4, FR-5, FR-6, FR-7, FR-8, FR-9, FR-18, NFR-1.1-1.7 - -**Note**: Week 4b bridges Origin→CAS data flow (cache-miss handling) required for actual file reads. - ---- - -### Week 1: Foundation - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Workspace setup | root | `Cargo.toml`, `.cargo/config.toml` | - | -| Core types | musicfs-core | `lib.rs`, `error.rs`, `types.rs` | - | -| Event Bus | musicfs-core | `events.rs` | FR-18.1-18.4 | -| FUSE skeleton | musicfs-fuse | `lib.rs`, `filesystem.rs` | FR-1.1, FR-1.2 | -| Local origin | musicfs-origins | `lib.rs`, `local.rs`, `traits.rs` | FR-12.1 | -| Nix flake | root | `flake.nix` | - | - -#### Core Types (`musicfs-core/src/types.rs`) - -```rust -// Virtual path in metadata-organized tree -pub struct VirtualPath(PathBuf); - -// Real path on origin -pub struct RealPath { - pub origin_id: OriginId, - pub path: PathBuf, -} - -// File metadata -pub struct FileMeta { - pub id: FileId, - pub virtual_path: VirtualPath, - pub real_path: RealPath, - pub size: u64, - pub mtime: SystemTime, - pub content_hash: ContentHash, - pub audio: Option, -} - -pub struct AudioMeta { - pub artist: Option, - pub album: Option, - pub title: Option, - pub track_number: Option, - pub duration_ms: Option, - pub format: AudioFormat, -} - -// Content-addressable hash (per architecture 8.3) -pub struct ContentHash([u8; 8]); // xxHash64 for file-level dedup -pub struct ChunkHash([u8; 8]); // xxHash64 for chunk-level dedup -``` - -#### Event Bus (`musicfs-core/src/events.rs`) - -```rust -use tokio::sync::broadcast; - -/// Central event bus for system-wide notifications (per architecture 4.2) -pub struct EventBus { - sender: broadcast::Sender, -} - -impl EventBus { - pub fn new(capacity: usize) -> Self { - let (sender, _) = broadcast::channel(capacity); - Self { sender } - } - - pub fn publish(&self, event: Event) { - let _ = self.sender.send(event); // Ignore if no receivers - } - - pub fn subscribe(&self) -> broadcast::Receiver { - self.sender.subscribe() - } -} - -#[derive(Clone, Debug)] -pub enum Event { - FileAdded { path: VirtualPath, origin_id: OriginId }, - FileRemoved { path: VirtualPath }, - FileModified { path: VirtualPath }, - OriginConnected { origin_id: OriginId }, - OriginDisconnected { origin_id: OriginId }, - SyncStarted { origin_id: OriginId }, - SyncCompleted { origin_id: OriginId, files_changed: u64 }, - CacheEviction { bytes_freed: u64 }, -} -``` - -#### Origin Trait (`musicfs-origins/src/traits.rs`) - -```rust -#[async_trait] -pub trait Origin: Send + Sync { - fn id(&self) -> &OriginId; - fn origin_type(&self) -> OriginType; - - /// List entries in directory - async fn readdir(&self, path: &Path) -> Result>; - - /// Get file metadata - async fn stat(&self, path: &Path) -> Result; - - /// Read file content - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result; - - /// Check if path exists - async fn exists(&self, path: &Path) -> Result; - - /// Health check - async fn health(&self) -> HealthStatus; -} -``` - -#### FUSE Skeleton (`musicfs-fuse/src/filesystem.rs`) - -```rust -pub struct MusicFs { - origins: Arc, - cache: Arc, - tree: Arc>, -} - -impl fuser::Filesystem for MusicFs { - fn lookup(&mut self, _req: &Request, parent: u64, name: &OsStr, reply: ReplyEntry); - fn getattr(&mut self, _req: &Request, ino: u64, reply: ReplyAttr); - fn readdir(&mut self, _req: &Request, ino: u64, fh: u64, offset: i64, reply: ReplyDirectory); - fn open(&mut self, _req: &Request, ino: u64, flags: i32, reply: ReplyOpen); - fn read(&mut self, _req: &Request, ino: u64, fh: u64, offset: i64, size: u32, ...); - fn release(&mut self, _req: &Request, ino: u64, fh: u64, ...); -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_workspace_builds` | Unit | All crates compile | -| `test_local_origin_readdir` | Unit | Local origin lists files | -| `test_fuse_mount_unmount` | Integration | Mount/unmount works (FR-1.1, FR-1.2) | - -#### Exit Criteria - -- [ ] `cargo build` succeeds for all crates -- [ ] `cargo test` passes -- [ ] FUSE mount creates mount point -- [ ] FUSE unmount removes mount point - ---- - -### Week 2: Metadata Extraction - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Audio parsing | musicfs-metadata | `lib.rs`, `parser.rs`, `formats/` | FR-6.1-6.5 | -| Format detection | musicfs-metadata | `formats/flac.rs`, `formats/mp3.rs`, etc. | FR-24.1 | -| SQLite schema | musicfs-cache | `schema.sql`, `db.rs` | FR-7.1-7.4 | -| Metadata cache | musicfs-cache | `metadata.rs` | FR-7.1 | - -#### Metadata Parser (`musicfs-metadata/src/parser.rs`) - -```rust -pub struct MetadataParser { - symphonia: SymphoniaParser, -} - -impl MetadataParser { - /// Extract metadata from audio file - pub fn parse(&self, reader: impl Read + Seek) -> Result { - // Uses symphonia for format-agnostic parsing - } - - /// Extract just the header (first N KB for metadata overlay) - pub fn parse_header(&self, reader: impl Read + Seek) -> Result<(AudioMeta, Bytes)> { - // Returns metadata + raw header bytes for caching - } -} -``` - -#### SQLite Schema (`musicfs-cache/src/schema.sql`) - -```sql --- As defined in architecture.md section 4.3.6 --- NOTE: Chunk index stored in sled (chunks.sled/), NOT SQLite - -CREATE TABLE files ( - id INTEGER PRIMARY KEY, - origin_id TEXT NOT NULL, -- Origin identifier - real_path TEXT NOT NULL, -- Path on origin - virtual_path TEXT NOT NULL, -- Metadata-derived path - - -- Audio metadata - title TEXT, - artist TEXT, - album TEXT, - track_number INTEGER, - duration_ms INTEGER, - bitrate INTEGER, - sample_rate INTEGER, - format TEXT, - - -- Sync state - origin_mtime INTEGER, - origin_size INTEGER, - content_hash TEXT, - chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)] - last_sync INTEGER, - - UNIQUE(origin_id, real_path) -); - -CREATE TABLE artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER REFERENCES files(id), - art_type TEXT, -- 'front', 'back' - chunk_hash TEXT, -- reference to CAS - width INTEGER, - height INTEGER, - UNIQUE(file_id, art_type) -); - -CREATE TABLE collections ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE, - query_json TEXT, -- smart collection query - created_at INTEGER -); - --- Indexes for performance (NFR-1.1, NFR-1.2) -CREATE INDEX idx_virtual ON files(virtual_path); -CREATE INDEX idx_artist_album ON files(artist, album); -CREATE INDEX idx_content_hash ON files(content_hash); -``` - -#### Sled Chunk Index (`musicfs-cas/chunks.sled/`) - -```rust -// Chunk hash → storage location (NOT in SQLite per architecture 4.3.2) -// Key: ChunkHash (8 bytes, xxHash64) -// Value: ChunkLocation { path: PathBuf, offset: u64, size: u32 } -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_parse_flac` | Unit | FLAC metadata extraction (FR-6.1-6.5) | -| `test_parse_mp3_id3v2` | Unit | MP3 ID3v2 tags | -| `test_parse_mp3_id3v1` | Unit | MP3 ID3v1 fallback | -| `test_parse_opus` | Unit | Opus/Vorbis comments | -| `test_metadata_cache_insert` | Unit | SQLite insert/query | -| `test_metadata_cache_persistence` | Integration | Data survives restart (FR-7.4) | - -#### Exit Criteria - -- [ ] Parse FLAC, MP3, Opus, M4A metadata -- [ ] Extract: artist, album, title, track, duration, format -- [ ] SQLite schema created and migrated -- [ ] Metadata persists across restarts - ---- - -### Week 3: Virtual Tree & Basic Ops - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Virtual path resolver | musicfs-core | `resolver.rs` | FR-5.1, FR-5.2 | -| Tree cache | musicfs-cache | `tree.rs` | FR-9.1-9.4 | -| readdir impl | musicfs-fuse | `ops/readdir.rs` | FR-2.1-2.3 | -| stat impl | musicfs-fuse | `ops/stat.rs` | FR-2.1 | -| open/read impl | musicfs-fuse | `ops/read.rs` | FR-3.1-3.5 | - -#### Virtual Path Resolver (`musicfs-core/src/resolver.rs`) - -```rust -pub struct PathResolver { - templates: Vec, -} - -impl PathResolver { - /// Map real path + metadata → virtual path - pub fn resolve(&self, real: &RealPath, meta: &AudioMeta) -> VirtualPath { - // Template: "/{artist}/{album}/{track:02} - {title}.{ext}" - } - - /// Reverse lookup: virtual path → file ID - pub fn lookup(&self, virtual_path: &VirtualPath, tree: &VirtualTree) -> Option { - // O(1) hash lookup - } -} - -pub struct PathTemplate { - pub pattern: String, - pub fallback_artist: String, // "Unknown Artist" - pub fallback_album: String, // "Unknown Album" -} -``` - -#### Tree Cache (`musicfs-cache/src/tree.rs`) - -```rust -pub struct VirtualTree { - root: DirNode, - inode_map: HashMap, - path_map: HashMap, - next_inode: AtomicU64, -} - -pub enum VirtualNode { - Dir(DirNode), - File(FileNode), -} - -pub struct DirNode { - pub inode: u64, - pub name: OsString, - pub children: BTreeMap, - pub mtime: SystemTime, -} - -pub struct FileNode { - pub inode: u64, - pub name: OsString, - pub file_id: FileId, - pub size: u64, - pub mtime: SystemTime, -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_resolve_complete_metadata` | Unit | Full path resolution (FR-5.1) | -| `test_resolve_missing_album` | Unit | Fallback handling (FR-5.2) | -| `test_tree_readdir` | Unit | Directory listing (FR-2.1) | -| `test_tree_stat` | Unit | File attributes (FR-2.1) | -| `test_fuse_readdir` | Integration | FUSE readdir (FR-2.2, NFR-1.2) | -| `test_fuse_stat` | Integration | FUSE stat (NFR-1.1) | -| `test_fuse_read` | Integration | FUSE read (FR-3.1) | -| `test_read_only_enforcement` | Integration | Write ops fail (FR-4.1-4.4) | - -#### Benchmark - -```rust -// benches/tree_ops.rs -fn bench_stat_cached(c: &mut Criterion) { - // Target: <1ms p99 (NFR-1.1) -} - -fn bench_readdir_1000_entries(c: &mut Criterion) { - // Target: <10ms p99 (NFR-1.2) -} - -fn bench_mount_time(c: &mut Criterion) { - // Target: <100ms, Max: <500ms (NFR-1.7) - // Mount must be O(1), not O(files) -} -``` - -#### Exit Criteria - -- [ ] Virtual tree built from metadata -- [ ] `ls /mnt/music` shows Artist directories -- [ ] `ls /mnt/music/Artist/Album` shows tracks -- [ ] `stat` returns correct size, mtime -- [ ] `cat` reads file content -- [ ] Write operations return EROFS (FR-4.1) -- [ ] Mount completes in <500ms (NFR-1.7) - ---- - -### Week 4: CAS & Chunk Caching - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| CAS implementation | musicfs-cas | `lib.rs`, `store.rs` | FR-20.1-20.4 | -| Chunk storage | musicfs-cas | `chunks.rs` | FR-8.1-8.4 | -| Cache eviction | musicfs-cache | `eviction.rs` | FR-8.2 | -| Integration tests | tests/integration | `basic_mount.rs` | FR-1, FR-2, FR-3 | - -#### CAS Store (`musicfs-cas/src/store.rs`) - -```rust -pub struct CasStore { - chunks_dir: PathBuf, - index: sled::Db, // hash → chunk location -} - -impl CasStore { - /// Store chunk, returns hash - pub async fn put(&self, data: &[u8]) -> Result { - let hash = xxhash64(data); - let path = self.chunk_path(&hash); - if !path.exists() { - tokio::fs::write(&path, data).await?; - } - Ok(hash) - } - - /// Retrieve chunk by hash - pub async fn get(&self, hash: &ChunkHash) -> Result { - let path = self.chunk_path(hash); - Ok(tokio::fs::read(&path).await?.into()) - } - - /// Check existence (for dedup) - pub fn exists(&self, hash: &ChunkHash) -> bool { - self.chunk_path(hash).exists() - } -} -``` - -#### Cache Eviction (`musicfs-cache/src/eviction.rs`) - -```rust -pub struct LruEviction { - max_size: u64, - current_size: AtomicU64, - access_log: RwLock>, -} - -impl LruEviction { - /// Evict chunks until under limit - pub async fn evict_to_target(&self, store: &CasStore, target: u64) -> Result { - // LRU eviction based on access time - // Returns bytes freed - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_cas_put_get` | Unit | Basic store/retrieve (FR-20.1) | -| `test_cas_dedup` | Unit | Same content → same hash (FR-20.2) | -| `test_cas_integrity` | Unit | Verify chunk hash (FR-20.4) | -| `test_cache_eviction` | Unit | LRU eviction works (FR-8.2) | -| `test_cache_persistence` | Integration | Survives restart (FR-8.4) | -| `test_mount_play_audio` | E2E | mpv can play file through FUSE | - -#### Exit Criteria - -- [ ] Chunks stored in CAS with deduplication -- [ ] Cache size limit enforced via eviction -- [ ] Cache persists across daemon restarts - -**Note**: Audio playback requires Week 4b (Origin→CAS connector). - ---- - -### Week 4b: Origin-CAS Connector - -**Detailed plan**: See `plans/week-04b-origin-connector.md` - -#### Summary - -Bridges the gap between Origin (source files) and CAS (chunk cache). Without this, FUSE read() cannot return actual file content. - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| ContentFetcher | musicfs-cas | `fetcher.rs` | FR-3.2 | -| Cache-miss handling | musicfs-cas | `reader.rs` | FR-3.2 | -| FUSE integration | musicfs-fuse | `filesystem.rs` | FR-3.1-3.2 | - -#### Key Components - -```rust -pub struct ContentFetcher { - store: Arc, - origins: HashMap>, - file_meta: HashMap, -} - -impl ContentFetcher { - /// Fetch file from origin, store in CAS, return manifest - pub async fn fetch_file(&self, file_id: FileId) -> Result; -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_fetch_file` | Unit | Origin → CAS works | -| `test_reader_cache_miss` | Unit | Fetcher called on miss | -| `test_e2e_cat_file` | E2E | `cat` returns content | - -#### Exit Criteria - -- [ ] `cat /mnt/musicfs/Artist/Album/track.flac` returns actual data -- [ ] Audio playback works through mounted filesystem -- [ ] All Phase 1 requirements pass acceptance tests - ---- - -## 3. Phase 2: Delta Sync & Multi-Origin (Weeks 5-7) - -**Goal**: Efficient synchronization and origin federation. - -**Requirements Covered**: FR-10, FR-11, FR-12, FR-13, NFR-4, NFR-5 - ---- - -### Week 5: CDC & Delta Detection - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| FastCDC integration | musicfs-sync | `cdc.rs` | FR-11.2 | -| Manifest storage | musicfs-sync | `manifest.rs` | FR-11.3 | -| Delta detection | musicfs-sync | `delta.rs` | FR-10.1-10.4, FR-11.1 | -| Change watcher | musicfs-sync | `watcher.rs` | FR-10.3 | - -#### CDC Chunking (`musicfs-sync/src/cdc.rs`) - -```rust -pub struct CdcChunker { - min_size: usize, // 16 KB - avg_size: usize, // 64 KB - max_size: usize, // 256 KB -} - -impl CdcChunker { - /// Chunk file content using FastCDC - pub fn chunk(&self, data: &[u8]) -> Vec { - let chunker = fastcdc::v2020::FastCDC::new( - data, self.min_size, self.avg_size, self.max_size - ); - chunker.map(|c| Chunk { - offset: c.offset, - length: c.length, - hash: xxhash64(&data[c.offset..c.offset + c.length]), - }).collect() - } -} - -pub struct ChunkManifest { - pub content_hash: ContentHash, - pub chunks: Vec<(ChunkHash, u64, u32)>, // (hash, offset, size) - pub total_size: u64, -} -``` - -#### Delta Detection (`musicfs-sync/src/delta.rs`) - -```rust -pub struct DeltaDetector { - db: Arc, -} - -impl DeltaDetector { - /// Compare origin state to cached state - pub async fn detect_changes(&self, origin: &dyn Origin) -> Result { - let origin_files = origin.list_recursive().await?; - let cached_files = self.db.list_files(origin.id()).await?; - - ChangeSet { - added: self.find_added(&origin_files, &cached_files), - removed: self.find_removed(&origin_files, &cached_files), - modified: self.find_modified(&origin_files, &cached_files).await?, - } - } - - /// Compute chunk delta for modified file - pub async fn compute_delta( - &self, - old_manifest: &ChunkManifest, - new_chunks: &[Chunk], - ) -> ChunkDelta { - let old_hashes: HashSet<_> = old_manifest.chunks.iter().map(|c| c.0).collect(); - ChunkDelta { - reuse: new_chunks.iter().filter(|c| old_hashes.contains(&c.hash)).cloned().collect(), - fetch: new_chunks.iter().filter(|c| !old_hashes.contains(&c.hash)).cloned().collect(), - } - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_cdc_stable_boundaries` | Unit | Insertions don't shift all chunks | -| `test_delta_detect_added` | Unit | New files detected (FR-10.1) | -| `test_delta_detect_removed` | Unit | Deleted files detected (FR-10.2) | -| `test_delta_detect_modified` | Unit | Changed files detected (FR-10.4) | -| `test_delta_chunk_reuse` | Unit | Unchanged chunks reused (FR-11.1) | -| `test_bandwidth_reduction` | Integration | >90% reduction on metadata edit (NFR-5.1) | - -#### Exit Criteria - -- [ ] CDC produces stable chunk boundaries -- [ ] Delta sync detects add/remove/modify -- [ ] Unchanged chunks reused (not re-fetched) -- [ ] Metadata-only edits achieve >90% bandwidth reduction - ---- - -### Week 6: Origin Federation - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Origin registry | musicfs-origins | `registry.rs` | FR-13.1-13.3 | -| Priority routing | musicfs-origins | `router.rs` | FR-13.4 | -| Health checks | musicfs-origins | `health.rs` | FR-13.6 | -| Failover logic | musicfs-origins | `failover.rs` | FR-13.5 | - -#### Origin Registry (`musicfs-origins/src/registry.rs`) - -```rust -pub struct OriginRegistry { - origins: RwLock>>, - router: Router, - health: HealthMonitor, -} - -impl OriginRegistry { - /// Register new origin - pub async fn register(&self, config: OriginConfig) -> Result; - - /// Get best origin for path - pub async fn route(&self, path: &RealPath) -> Result>; - - /// Get all origins for path (for redundancy) - pub async fn route_all(&self, path: &RealPath) -> Vec>; -} -``` - -#### Router (`musicfs-origins/src/router.rs`) - -```rust -pub struct Router { - priority_map: RwLock>, - latency_stats: DashMap, -} - -impl Router { - /// Select best origin based on priority + health + latency - pub fn select(&self, candidates: &[OriginId], health: &HealthSnapshot) -> Option { - candidates - .iter() - .filter(|id| health.is_healthy(id)) - .min_by_key(|id| { - let priority = self.priority_map.read().get(id).copied().unwrap_or(100); - let latency = self.latency_stats.get(id).map(|s| s.p50_ms).unwrap_or(1000); - (priority, latency) - }) - .copied() - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_register_multiple_origins` | Unit | Multi-origin support (FR-13.1) | -| `test_route_by_priority` | Unit | Priority routing (FR-13.4) | -| `test_failover_on_unhealthy` | Unit | Automatic failover (FR-13.5) | -| `test_health_check_interval` | Integration | Periodic health checks (FR-13.6) | -| `test_origin_offline_graceful` | Integration | Serve cached when offline (NFR-7.1) | - -#### Exit Criteria - -- [ ] Multiple origins configurable -- [ ] Requests route to healthiest origin -- [ ] Failover when primary fails -- [ ] Offline origin doesn't crash daemon - ---- - -### Week 7: Remote Origins - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| NFS origin | musicfs-origins | `nfs.rs` | FR-12.2 | -| SMB origin | musicfs-origins | `smb.rs` | FR-12.3 | -| S3 origin | musicfs-origins | `s3.rs` | FR-12.4 | -| SFTP origin | musicfs-origins | `sftp.rs` | FR-12.5 | -| Credential handling | musicfs-core | `credentials.rs` | Security | - -#### NFS Origin (`musicfs-origins/src/nfs.rs`) - -```rust -/// NFS origin treats mounted NFS as local filesystem -/// User mounts NFS externally; we just read from mount point -pub struct NfsOrigin { - mount_point: PathBuf, - id: OriginId, -} - -#[async_trait] -impl Origin for NfsOrigin { - // Delegates to LocalOrigin implementation - // NFS-specific: handle stale file handles, network timeouts - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result { - // Retry on ESTALE (stale NFS file handle) - for attempt in 0..3 { - match self.do_read(path, offset, size).await { - Ok(data) => return Ok(data), - Err(e) if e.raw_os_error() == Some(libc::ESTALE) => { - tokio::time::sleep(Duration::from_millis(100 * (1 << attempt))).await; - } - Err(e) => return Err(e.into()), - } - } - Err(Error::NfsStaleHandle) - } -} -``` - -#### S3 Origin (`musicfs-origins/src/s3.rs`) - -```rust -pub struct S3Origin { - client: aws_sdk_s3::Client, - bucket: String, - prefix: String, - id: OriginId, -} - -#[async_trait] -impl Origin for S3Origin { - async fn readdir(&self, path: &Path) -> Result> { - let prefix = self.prefix.join(path); - let resp = self.client.list_objects_v2() - .bucket(&self.bucket) - .prefix(&prefix) - .delimiter("/") - .send().await?; - // Convert S3 response to DirEntry - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result { - let range = format!("bytes={}-{}", offset, offset + size as u64 - 1); - let resp = self.client.get_object() - .bucket(&self.bucket) - .key(&self.prefix.join(path)) - .range(range) - .send().await?; - Ok(resp.body.collect().await?.into_bytes()) - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_nfs_origin_stale_handle` | Unit | NFS ESTALE retry (FR-12.2) | -| `test_smb_origin_mock` | Unit | SMB protocol (FR-12.3) | -| `test_s3_origin_mock` | Unit | S3 protocol (FR-12.4) | -| `test_sftp_origin_mock` | Unit | SFTP protocol (FR-12.5) | -| `test_s3_origin_real` | Integration | Real S3 (requires creds) | -| `test_mixed_origins` | Integration | Local + remote together | - -#### Exit Criteria - -- [ ] NFS origin functional (treats NFS mount as local) -- [ ] SMB origin functional -- [ ] S3 origin functional -- [ ] SFTP origin functional -- [ ] Mixed local + remote works -- [ ] All Phase 2 requirements pass acceptance tests - ---- - -## 4. Phase 3: Search & Smart Features (Weeks 8-9) - -**Goal**: Full-text search and intelligent caching. - -**Requirements Covered**: FR-14, FR-15, FR-16, FR-19 - ---- - -### Week 8: Search Index - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| tantivy integration | musicfs-search | `index.rs` | FR-14.1-14.4 | -| Search virtual dir | musicfs-fuse | `ops/search.rs` | FR-14.3 | -| Query parser | musicfs-search | `query.rs` | FR-14.2 | -| Incremental indexing | musicfs-search | `indexer.rs` | FR-14.4 | - -#### Search Index (`musicfs-search/src/index.rs`) - -```rust -pub struct SearchIndex { - index: tantivy::Index, - reader: IndexReader, - writer: Mutex, -} - -impl SearchIndex { - pub fn search(&self, query: &str, limit: usize) -> Result> { - let searcher = self.reader.searcher(); - let query_parser = QueryParser::for_index( - &self.index, - vec![ - self.schema.get_field("artist")?, - self.schema.get_field("album")?, - self.schema.get_field("title")?, - ], - ); - let query = query_parser.parse_query(query)?; - let top_docs = searcher.search(&query, &TopDocs::with_limit(limit))?; - // Convert to SearchHit - } - - pub fn index_file(&self, file: &FileMeta) -> Result<()> { - let mut writer = self.writer.lock(); - writer.add_document(doc!( - self.schema.get_field("path")? => file.virtual_path.as_str(), - self.schema.get_field("artist")? => file.audio.as_ref().and_then(|a| a.artist.as_deref()).unwrap_or(""), - self.schema.get_field("album")? => file.audio.as_ref().and_then(|a| a.album.as_deref()).unwrap_or(""), - self.schema.get_field("title")? => file.audio.as_ref().and_then(|a| a.title.as_deref()).unwrap_or(""), - ))?; - Ok(()) - } -} -``` - -#### Search Virtual Directory (`musicfs-fuse/src/ops/search.rs`) - -```rust -// /.search/metallica/ → symlinks to matching files -impl MusicFs { - fn handle_search_readdir(&self, query: &str, reply: ReplyDirectory) { - let results = self.search.search(query, 1000)?; - for (i, hit) in results.iter().enumerate() { - reply.add( - hit.inode, - (i + 1) as i64, - FileType::Symlink, - &hit.display_name(), - ); - } - } - - fn handle_search_readlink(&self, inode: u64, reply: ReplyData) { - // Return path to actual file - let target = self.search.resolve_inode(inode)?; - reply.data(target.as_bytes()); - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_search_artist` | Unit | Artist search (FR-14.1) | -| `test_search_album` | Unit | Album search (FR-14.1) | -| `test_search_multi_field` | Unit | Cross-field search (FR-14.2) | -| `test_search_performance_1m` | Benchmark | <1s for 1M tracks (NFR-2.5) | -| `test_search_virtual_dir` | Integration | /.search/query works (FR-14.3) | -| `test_search_incremental` | Integration | New files indexed (FR-14.4) | - -#### Exit Criteria - -- [ ] Full-text search works -- [ ] `/.search/{query}/` returns symlinks -- [ ] Search <1s for 1M tracks -- [ ] New files automatically indexed - ---- - -### Week 9: Smart Features - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Smart collections | musicfs-search | `collections.rs` | FR-15.1-15.5 | -| Cover art cache | musicfs-cache | `artwork.rs` | FR-16.1-16.4 | -| Prefetch engine | musicfs-cache | `prefetch.rs` | FR-19.1-19.4 | -| Access patterns | musicfs-cache | `patterns.rs` | FR-19.3 | - -#### Smart Collections (`musicfs-search/src/collections.rs`) - -```rust -pub struct SmartCollection { - pub name: String, - pub query: CollectionQuery, -} - -pub enum CollectionQuery { - DateRange { field: String, start: NaiveDate, end: NaiveDate }, - Match { field: String, pattern: String }, - Rating { min: u8 }, - RecentlyAdded { days: u32 }, - Compound { op: BoolOp, children: Vec }, -} - -impl SmartCollection { - /// Materialize as virtual directory - pub fn resolve(&self, index: &SearchIndex) -> Vec { - index.query(&self.query.to_tantivy_query()) - } -} -``` - -#### Prefetch Engine (`musicfs-cache/src/prefetch.rs`) - -```rust -pub struct PrefetchEngine { - pattern_db: PatternStore, - queue: PriorityQueue, -} - -impl PrefetchEngine { - /// Record file access - pub fn record_access(&self, file_id: FileId, context: AccessContext) { - self.pattern_db.record(file_id, context); - } - - /// Predict next likely accesses - pub fn predict(&self, current: FileId) -> Vec { - // Album-aware: if track N accessed, prefetch N+1, N+2 - // Artist-aware: other albums by same artist - // Time-based: tracks often played together - } - - /// Background prefetch worker - pub async fn run(&self, cache: Arc) { - loop { - if let Some(task) = self.queue.pop() { - cache.prefetch(&task.file_id).await; - } - tokio::time::sleep(Duration::from_millis(10)).await; - } - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_collection_date_range` | Unit | Date-based collections (FR-15.2) | -| `test_collection_recently_added` | Unit | Recent additions (FR-15.2) | -| `test_artwork_extraction` | Unit | Cover art from audio (FR-16.1) | -| `test_artwork_thumbnail` | Unit | Thumbnail generation (FR-16.3) | -| `test_prefetch_album_sequence` | Unit | Sequential track prefetch (FR-19.1) | -| `test_prefetch_reduces_misses` | Integration | >50% miss reduction (FR-19.4) | - -#### Exit Criteria - -- [ ] Smart collections as virtual directories -- [ ] Album art extracted and thumbnailed -- [ ] Prefetch reduces cache misses >50% -- [ ] All Phase 3 requirements pass acceptance tests - ---- - -## 5. Phase 4: Plugin System & Polish (Weeks 10-11) - -**Goal**: Extensibility and production readiness. - -**Requirements Covered**: FR-17, FR-18, FR-23, FR-24, NFR-6 - ---- - -### Week 10: Plugin System - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Plugin traits | musicfs-plugins | `traits.rs` | FR-23.1-23.4 | -| Native host | musicfs-plugins | `native.rs` | FR-23.2 | -| WASM host | musicfs-plugins | `wasm.rs` | FR-23.3 | -| Example plugins | plugins/ | `local-origin/`, `flac-format/` | FR-23.5 | - -#### Plugin Traits (`musicfs-plugins/src/traits.rs`) - -```rust -/// Origin plugin interface -pub trait OriginPlugin: Send + Sync { - fn id(&self) -> &str; - fn origin_type(&self) -> &str; - fn create(&self, config: Value) -> Result>; -} - -/// Metadata source plugin -pub trait MetadataPlugin: Send + Sync { - fn id(&self) -> &str; - fn lookup(&self, query: &MetadataQuery) -> Result>; -} - -/// Format plugin (for custom audio formats) -pub trait FormatPlugin: Send + Sync { - fn id(&self) -> &str; - fn extensions(&self) -> &[&str]; - fn parse(&self, reader: &mut dyn Read) -> Result; -} -``` - -#### WASM Host (`musicfs-plugins/src/wasm.rs`) - -```rust -pub struct WasmPluginHost { - engine: wasmtime::Engine, - linker: wasmtime::Linker, -} - -impl WasmPluginHost { - pub fn load(&self, wasm_bytes: &[u8]) -> Result { - let module = wasmtime::Module::new(&self.engine, wasm_bytes)?; - let instance = self.linker.instantiate(&mut store, &module)?; - // Extract plugin interface - } -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_native_plugin_load` | Unit | Native plugin loading (FR-23.2) | -| `test_wasm_plugin_sandbox` | Unit | WASM isolation (FR-23.3) | -| `test_plugin_hot_reload` | Integration | Reload without restart (FR-23.4) | -| `test_example_origin_plugin` | Integration | Custom origin works | - -#### Exit Criteria - -- [ ] Native plugins loadable at runtime -- [ ] WASM plugins sandboxed -- [ ] Example plugins functional -- [ ] Plugins hot-reloadable - ---- - -### Week 11: Control API & Production - -#### Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| gRPC server | musicfs-grpc | `server.rs` | FR-17.1-17.5 | -| Proto codegen | proto/ | `musicfs.proto`, `build.rs` | FR-17.2 | -| Event streaming | musicfs-grpc | `events.rs` | FR-18.1-18.4 | -| Metrics export | musicfs-core | `metrics.rs` | NFR-6.1-6.4 | -| CLI completion | musicfs-cli | `main.rs` | FR-17 | -| systemd unit | dist/ | `musicfs.service` | Production | -| Packaging | dist/ | `PKGBUILD`, `musicfs.spec` | Production | - -#### Proto Definitions (`proto/musicfs.proto`) - -**Source**: Copy verbatim from [architecture.md section 4.3.7](architecture.md#437-control-api) - -```protobuf -syntax = "proto3"; -package musicfs.v1; - -service MusicFS { - // Daemon lifecycle - rpc GetStatus(Empty) returns (StatusResponse); - rpc Shutdown(ShutdownRequest) returns (Empty); - - // Cache management - rpc GetCacheStats(Empty) returns (CacheStats); - rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse); - rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress); - - // Origin management - rpc ListOrigins(Empty) returns (OriginsResponse); - rpc GetOriginHealth(OriginRequest) returns (OriginHealth); - rpc RescanOrigin(OriginRequest) returns (stream SyncProgress); - - // Search - rpc Search(SearchRequest) returns (SearchResponse); - rpc SearchStream(SearchRequest) returns (stream SearchResult); - - // Events (server-streaming) - rpc SubscribeEvents(EventFilter) returns (stream Event); -} - -// Full message definitions in architecture.md section 4.3.7 -// Including: StatusResponse, CacheStats, TierStats, OriginInfo, -// OriginHealth, SyncProgress, SearchRequest, SearchResponse, -// EventFilter, Event, and all enums -``` - -#### gRPC Server (`musicfs-grpc/src/server.rs`) - -```rust -pub struct MusicFsService { - core: Arc, - events: broadcast::Sender, -} - -#[tonic::async_trait] -impl musicfs::v1::music_fs_server::MusicFs for MusicFsService { - async fn get_status(&self, _: Request) -> Result, Status> { - let status = self.core.status().await; - Ok(Response::new(status.into())) - } - - type SubscribeEventsStream = ReceiverStream>; - - async fn subscribe_events( - &self, - request: Request, - ) -> Result, Status> { - let filter = request.into_inner(); - let rx = self.events.subscribe(); - let stream = BroadcastStream::new(rx) - .filter(|e| filter.matches(e)) - .map(|e| Ok(e.into())); - Ok(Response::new(ReceiverStream::new(stream))) - } -} -``` - -#### Metrics (`musicfs-core/src/metrics.rs`) - -```rust -lazy_static! { - pub static ref FUSE_OPS: IntCounterVec = register_int_counter_vec!( - "musicfs_fuse_ops_total", - "Total FUSE operations", - &["op"] - ).unwrap(); - - pub static ref FUSE_LATENCY: HistogramVec = register_histogram_vec!( - "musicfs_fuse_latency_seconds", - "FUSE operation latency", - &["op"], - vec![0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0] - ).unwrap(); - - pub static ref CACHE_HITS: IntCounter = register_int_counter!( - "musicfs_cache_hits_total", - "Cache hits" - ).unwrap(); -} -``` - -#### Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_grpc_status` | Unit | GetStatus RPC (FR-17.1) | -| `test_grpc_cache_clear` | Unit | ClearCache RPC (FR-17.3) | -| `test_grpc_events_stream` | Integration | Event streaming (FR-18.1) | -| `test_metrics_prometheus` | Unit | Prometheus format (NFR-6.1) | -| `test_cli_commands` | Integration | CLI works | -| `test_systemd_service` | E2E | Service lifecycle | - -#### Exit Criteria - -- [ ] gRPC API fully functional -- [ ] Event streaming works -- [ ] Prometheus metrics exported -- [ ] CLI feature-complete -- [ ] systemd service works -- [ ] All acceptance tests pass - ---- - -## 6. Acceptance Test Matrix - -| Test Suite | Requirements | Phase | -|------------|--------------|-------| -| `test_basic_mount` | FR-1.1-1.5, NFR-1.4-1.5, NFR-1.7 | 1 | -| `test_directory_ops` | FR-2.1-2.3, NFR-1.2 | 1 | -| `test_file_read` | FR-3.1-3.5, NFR-1.3 | 1 | -| `test_read_only` | FR-4.1-4.4 | 1 | -| `test_metadata_extraction` | FR-6.1-6.5 | 1 | -| `test_cache_persistence` | FR-7.1-7.4, FR-8.1-8.4 | 1 | -| `test_delta_sync` | FR-10.1-10.4, FR-11.1-11.4, NFR-5.1 | 2 | -| `test_multi_origin` | FR-13.1-13.6 | 2 | -| `test_remote_origins` | FR-12.1-12.5 | 2 | -| `test_search` | FR-14.1-14.4, NFR-2.5 | 3 | -| `test_smart_collections` | FR-15.1-15.5 | 3 | -| `test_album_art` | FR-16.1-16.4 | 3 | -| `test_prefetch` | FR-19.1-19.4 | 3 | -| `test_plugins` | FR-23.1-23.5 | 4 | -| `test_control_api` | FR-17.1-17.5 | 4 | -| `test_events` | FR-18.1-18.4 | 4 | -| `test_performance` | NFR-1.1-1.5, NFR-2.1-2.4 | All | -| `test_scalability` | NFR-3.1-3.4 | All | - ---- - -## 7. Risk Mitigation - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| FUSE performance ceiling | Medium | High | Early benchmarking in Week 3; consider io_uring bypass | -| symphonia format gaps | Low | Medium | Document unsupported formats; allow format plugins | -| WASM plugin overhead | Medium | Low | Benchmark; native fallback for perf-critical plugins | -| S3 latency variability | High | Medium | Aggressive prefetch; health-based routing | -| SQLite contention | Low | Medium | WAL mode; read-only connections; consider sled fallback | - ---- - -## 8. Deferred Requirements (Post-MVP) - -The following P1 requirements are explicitly deferred from the 11-week plan: - -### FR-21: External Metadata Sources [P1] → Phase 5 - -| ID | Requirement | Rationale for Deferral | -|----|-------------|------------------------| -| FR-21.1 | MusicBrainz integration | Requires rate limiting, caching, API key management | -| FR-21.2 | Discogs integration | OAuth flow complexity; lower priority than core function | -| FR-21.3 | Last.fm integration | Scrobbling is write operation; conflicts with read-only model | -| FR-21.4 | AcoustID fingerprinting | CPU-intensive; requires chromaprint dependency | -| FR-21.5 | Custom metadata plugins | Covered by FR-23 plugin system; metadata plugin type needed | - -**Recommended Phase 5 scope (Weeks 12-13):** -- MusicBrainz lookup plugin (native) -- Metadata plugin trait in musicfs-plugins -- Example WASM metadata plugin - -### FR-22: Import & Migration [P1] → Phase 5 - -| ID | Requirement | Rationale for Deferral | -|----|-------------|------------------------| -| FR-22.1 | Import from beets database | Requires beets schema knowledge; optional for fresh installs | -| FR-22.2 | Import from iTunes library | XML parsing; Apple ecosystem complexity | -| FR-22.3 | Export library metadata | Nice-to-have; users can query SQLite directly | - -**Recommended Phase 5 scope (Week 14):** -- `musicfs import beets /path/to/beets.db` CLI command -- Export to JSON/CSV via CLI - -### Why Defer? - -1. **Core functionality first**: MVP must prove FUSE + caching + multi-origin works -2. **Plugin system prerequisite**: FR-21.5 depends on FR-23 being stable -3. **Limited impact**: Users can populate library from origins without import -4. **Scope control**: 11 weeks is aggressive; buffer for unknowns - ---- - -## 9. Definition of Done - -A feature is complete when: - -1. [ ] Implementation matches architecture.md specification -2. [ ] Unit tests pass with >80% coverage -3. [ ] Integration tests pass -4. [ ] Performance meets NFR targets (benchmarked) -5. [ ] Documentation updated -6. [ ] No new `unsafe` without justification -7. [ ] Clippy clean (deny warnings) -8. [ ] Code reviewed - ---- - -## 9. Getting Started (Week 1, Day 1) - -```bash -# Create workspace -mkdir -p musicfs && cd musicfs -cargo init --name musicfs - -# Create crate structure -mkdir -p crates/{musicfs-core,musicfs-fuse,musicfs-cache,musicfs-cas,musicfs-sync,musicfs-origins,musicfs-metadata,musicfs-search,musicfs-plugins,musicfs-grpc,musicfs-cli} -mkdir -p proto tests/{integration,e2e} benches - -# Initialize each crate -for crate in crates/*; do - cargo init --lib "$crate" -done -cargo init crates/musicfs-cli - -# Add to workspace Cargo.toml -cat > Cargo.toml << 'EOF' -[workspace] -resolver = "2" -members = ["crates/*"] - -[workspace.package] -version = "0.1.0" -edition = "2021" -license = "MIT OR Apache-2.0" -rust-version = "1.75" - -[workspace.dependencies] -tokio = { version = "1", features = ["full"] } -async-trait = "0.1" -thiserror = "1" -tracing = "0.1" -serde = { version = "1", features = ["derive"] } -EOF - -# Create flake.nix for reproducible builds -# ... (nix flake with rust-overlay) -``` - -Ready to begin implementation? diff --git a/docs/v2/features/metadata-enrichment-standalone.md b/docs/v2/features/metadata-enrichment-standalone.md deleted file mode 100644 index 837871e..0000000 --- a/docs/v2/features/metadata-enrichment-standalone.md +++ /dev/null @@ -1,579 +0,0 @@ -# Metadata Enrichment (Standalone Mode): Design Doc - -**Authors:** Sisyphus -**Status:** Draft -**Last Updated:** 2026-05-18 -**Reviewers:** — -**Approvers:** — -**Document Link:** `docs/v2/plans/metadata-enrichment-standalone.md` -**Prerequisites:** [architecture.md](../architecture.md), [week-12-external-metadata.md](week-12-external-metadata.md) - ---- - -## 1. Abstract - -When musicfs operates without the music-agregator orchestrator, it should -still be able to enrich file metadata (genres, label, artwork URL, album -type) by querying the metadata-agregator service directly. This document -describes a **built-in metadata provider** compiled into musicfs that -queries metadata-agregator's gRPC `SearchAlbums` endpoint using -artist + album names extracted from file tags. Enrichment is lazy and -non-blocking — file access always returns immediately using embedded -tags, while a background worker enriches metadata asynchronously. - -This plan **supersedes** the week-12 plan's approach of embedding -MusicBrainz/Discogs/Last.fm HTTP clients directly into musicfs. Instead, -musicfs delegates all external metadata resolution to metadata-agregator, -which already handles provider APIs, rate limiting, and caching. - -## 2. Background - -### 2.1. Current State - -musicfs extracts audio metadata via symphonia (FLAC, MP3, AAC, OGG, -Opus) and stores it in `AudioMeta`. This metadata is whatever the file -tags contain — typically title, artist, album, year, track number. - -The existing plugin system (`musicfs-plugins`) defines a `MetadataPlugin` -trait for external metadata lookup, but: - -- No plugins have been implemented yet. -- The plugin system only supports native `.so` and WASM plugins. -- A gRPC client to metadata-agregator would require bundling an async - runtime and tonic inside a `.so` — an awkward fit. - -Meanwhile, metadata-agregator is a Go gRPC service that: - -- Searches MusicBrainz by artist + album name (`SearchAlbums` RPC). -- Caches results in PostgreSQL. -- Returns rich metadata: genres, cover URL, label, release date, album - type, artist credits. - -### 2.2. Pain Points - -- musicfs files lack genres, artwork URLs, and label info unless the - original files were meticulously tagged. -- The week-12 plan proposed embedding 4 separate HTTP API clients - (MusicBrainz, Discogs, Last.fm, AcoustID) directly into musicfs, - duplicating what metadata-agregator already does. -- The `MetadataPlugin` trait is designed for `.so`/WASM plugins, which - is wrong for a core infrastructure gRPC client. - -## 3. Goals & Non-Goals - -### 3.1. Goals - -- **G1:** Enrich file metadata with genres, label, album type, and cover - URL by querying metadata-agregator via gRPC. -- **G2:** Never block file access — enrichment happens in background. -- **G3:** Make the provider entirely optional — disabled by default, - musicfs works identically without it. -- **G4:** Respect enrichment source priority so orchestrator pushes - (from the full-system mode) are not overwritten. - -### 3.2. Non-Goals - -- **NG1:** Embedding MusicBrainz/Discogs/Last.fm HTTP clients directly - into musicfs (metadata-agregator handles this). -- **NG2:** Audio fingerprinting (AcoustID) — deferred to future work. -- **NG3:** Modifying the existing `MetadataPlugin` trait — the built-in - provider is separate from the plugin system. -- **NG4:** Bidirectional communication — musicfs only queries - metadata-agregator, never the reverse. - -## 4. Proposed Design - -### 4.1. High-Level Architecture - -```plantuml -@startuml -!theme plain -skinparam componentStyle rectangle - -package "musicfs" as mfs { - component "FUSE Layer\n(readdir/open/read)" as fuse - component "MetadataCache / DB" as db - component "OverlayReader\n(synthesize headers)" as overlay - component "EnrichmentQueue\n(bounded, async)" as queue - component "EnrichmentWorker\n(background)" as worker -} - -component "metadata-agregator\nSearchAlbums(query, artist)" as meta - -fuse -right-> db : lookup metadata -db -right-> overlay : serve with overlay - -fuse -down-> queue : enriched_at NULL?\npush request -queue -down-> worker : dequeue -worker -down-> meta : gRPC:\nSearchAlbums(\n query=album,\n artist=artist) -meta -up-> worker : Album (genres,\nlabel, cover_url) -worker -up-> db : write enriched\nmetadata to overlay - -note bottom of meta - metadata-agregator handles: - • MusicBrainz API - • rate limiting - • PostgreSQL cache -end note - -note right of fuse - File access is never blocked. - Returns embedded tags immediately. - Enrichment happens async. -end note -@enduml -``` - -### 4.2. Enrichment Flow - -```plantuml -@startuml -!theme plain -skinparam sequenceMessageAlign center - -participant "Media Player" as mp -participant "FUSE Layer" as fuse -participant "MetadataCache\n(SQLite)" as db -participant "EnrichmentQueue" as queue -participant "EnrichmentWorker" as worker -participant "metadata-agregator" as meta - -== File Access (non-blocking) == - -mp -> fuse : open("/Pink Floyd/The Wall/01 - In the Flesh.flac") -fuse -> db : lookup(virtual_path) -db --> fuse : AudioMeta(artist, album, title, ...)\nenriched_at = NULL - -fuse -> queue : try_push(file_id, artist="Pink Floyd", album="The Wall") -note right of queue : non-blocking,\nbounded queue - -fuse --> mp : return file handle\n(with embedded tags only) - -== Background Enrichment (async) == - -queue -> worker : dequeue(file_id, artist, album) - -worker -> worker : check enrichment_source\n(skip if 'orchestrator' or 'provider') - -worker -> worker : dedup check:\nalready enriched same album?\n(reuse cached result) - -worker -> meta : SearchAlbums(\n query="The Wall",\n artist="Pink Floyd",\n limit=1) -meta --> worker : Album(\n genres=["Progressive Rock", "Art Rock"],\n label="Harvest",\n cover_url="https://...",\n album_type="album") - -worker -> db : update_metadata(\n file_id,\n genres, label, cover_url,\n enrichment_source='provider',\n enriched_at=now()) - -worker -> worker : publish EventBus::FileModified - -note over mp : next access sees\nenriched metadata -@enduml -``` - -### 4.3. Detailed Design - -#### 4.3.1. Configuration - -Add `[metadata_provider]` section to `config.toml`: - -```toml -[metadata_provider] -enabled = false # disabled by default -endpoint = "http://localhost:50051" # metadata-agregator gRPC -timeout_ms = 5000 # per-request timeout -retry_max = 3 # max retries on failure -retry_backoff_ms = 1000 # initial backoff between retries -queue_size = 256 # enrichment queue capacity -``` - -Config struct addition in `musicfs-core/src/config.rs`: - -```rust -#[derive(Debug, Clone, Serialize, Deserialize, Default)] -pub struct MetadataProviderConfig { - #[serde(default)] - pub enabled: bool, - #[serde(default = "default_provider_endpoint")] - pub endpoint: String, - #[serde(default = "default_provider_timeout_ms")] - pub timeout_ms: u64, - #[serde(default = "default_retry_max")] - pub retry_max: u32, - #[serde(default = "default_retry_backoff_ms")] - pub retry_backoff_ms: u64, - #[serde(default = "default_queue_size")] - pub queue_size: usize, -} -``` - -#### 4.3.2. Built-in Metadata Provider - -New module in `musicfs-metadata` (not a plugin, compiled in): - -```rust -// musicfs-metadata/src/provider.rs - -pub struct MetadataAgregatorProvider { - client: MetadataServiceClient, - config: MetadataProviderConfig, -} - -impl MetadataAgregatorProvider { - pub async fn connect(config: &MetadataProviderConfig) - -> Result; - - /// Query metadata-agregator by artist + album names. - /// Returns enriched metadata if a match is found. - pub async fn lookup( - &self, - artist: &str, - album: &str, - ) -> Result>; -} -``` - -The `lookup` method calls `SearchAlbums(query=album, artist=artist, -limit=1)` on metadata-agregator. If a result is returned, it maps -the response to `EnrichedMetadata`: - -```rust -pub struct EnrichedMetadata { - pub genres: Vec, - pub label: Option, - pub album_type: Option, - pub cover_url: Option, - pub release_date: Option, - pub total_tracks: Option, - pub total_discs: Option, -} -``` - -#### 4.3.3. ExternalMetadata Extension - -Extend the existing `ExternalMetadata` in `musicfs-plugins/src/traits.rs` -to carry richer data: - -```rust -pub struct ExternalMetadata { - // existing fields... - pub title: Option, - pub artist: Option, - pub album: Option, - pub album_artist: Option, - pub genre: Option, // kept for backward compat - pub year: Option, - pub track: Option, - pub disc: Option, - pub musicbrainz_id: Option, - pub artwork_url: Option, - - // new fields - pub genres: Vec, - pub label: Option, - pub album_type: Option, - pub cover_url: Option, -} -``` - -#### 4.3.4. Database Schema Changes - -Add columns to `file_metadata` table in -`musicfs-cache/src/schema.sql`: - -```sql -ALTER TABLE file_metadata ADD COLUMN enrichment_source TEXT; - -- 'embedded' | 'provider' | 'orchestrator' -ALTER TABLE file_metadata ADD COLUMN enriched_at INTEGER; - -- unix timestamp, NULL = not enriched -ALTER TABLE file_metadata ADD COLUMN enrichment_attempts INTEGER DEFAULT 0; - -- number of failed enrichment attempts -ALTER TABLE file_metadata ADD COLUMN last_enrichment_error TEXT; - -- last error message, NULL if no error -ALTER TABLE file_metadata ADD COLUMN genres_json TEXT; - -- JSON array: '["Progressive Rock","Art Rock"]' - -- separate from existing `genre` (singular) for backward compat -ALTER TABLE file_metadata ADD COLUMN label TEXT; -ALTER TABLE file_metadata ADD COLUMN album_type TEXT; -ALTER TABLE file_metadata ADD COLUMN cover_url TEXT; -``` - -> **Note:** The existing `genre TEXT` column (singular) is preserved -> for backward compatibility. `genres_json` stores the full list. -> The singular `genre` field is set to the first genre in the array -> when enriched. - -#### 4.3.5. Background Enrichment Queue + Worker - -```rust -// musicfs-metadata/src/enrichment.rs - -pub struct EnrichmentQueue { - tx: mpsc::Sender, - /// Tracks in-flight (artist, album) pairs to prevent duplicate - /// API calls when multiple tracks from the same album are - /// accessed simultaneously. - in_flight: Arc>, -} - -struct EnrichmentRequest { - file_id: FileId, - artist: String, - album: String, -} - -pub struct EnrichmentWorker { - rx: mpsc::Receiver, - provider: Arc, - db: Arc, - event_bus: Arc, - in_flight: Arc>, - config: MetadataProviderConfig, -} -``` - -##### Enqueue-time dedup - -When `EnrichmentQueue::try_push()` is called, it checks the -`in_flight` `DashSet` before pushing. If `(artist, album)` is -already in the set, the request is dropped (the worker will enrich -all files with the same album in one pass). This prevents 12 -simultaneous track opens from making 12 identical API calls. - -If `try_push` fails because the queue is full, log at WARN level -and increment `enrichment_queue_drops_total` metric. - -##### Worker loop (single-threaded, processes one at a time): - -1. Dequeue `EnrichmentRequest`. -2. Check `enrichment_attempts` — skip if `>= retry_max`. -3. **Atomic conflict check**: write uses conditional SQL: - ```sql - UPDATE file_metadata SET - genres_json = ?, label = ?, album_type = ?, cover_url = ?, - genre = ?, -- first genre for backward compat - enrichment_source = 'provider', - enriched_at = strftime('%s', 'now'), - enrichment_attempts = 0, - last_enrichment_error = NULL - WHERE file_id = ? - AND (enrichment_source IS NULL OR enrichment_source = 'embedded') - ``` - This prevents the TOCTOU race — if the orchestrator wrote between - dequeue and now, the `WHERE` clause prevents overwrite. The UPDATE - returns rows_affected=0, which the worker treats as "skip, already - enriched by higher-priority source". -4. Deduplicate by (artist, album) — if another file in the same album - was already enriched, reuse the cached `EnrichedMetadata` result - for all files with the same (artist, album) pair. -5. Call `provider.lookup(artist, album)`. -6. On success: execute atomic update (step 3) for all files with this - (artist, album). Publish `EventBus::FileModified` for each updated - file. Remove `(artist, album)` from `in_flight` set. -7. On failure: increment `enrichment_attempts`, set - `last_enrichment_error`. If `attempts < retry_max`, re-enqueue - with exponential backoff (`retry_backoff_ms * 2^attempts`). - If `attempts >= retry_max`, log at WARN and stop retrying. - Remove from `in_flight` set. - -##### Shutdown behavior - -Queue contents are lost on shutdown. This is acceptable — files will -be re-queued on next access since `enriched_at` is still NULL. -Enrichment is idempotent. - -#### 4.3.6. FUSE Integration Point - -In the FUSE `readdir` / `getattr` / `open` path -(`musicfs-fuse/src/ops.rs`), after loading `AudioMeta` from DB: - -```rust -if metadata_provider.is_enabled() - && file_meta.enriched_at.is_none() - && file_meta.enrichment_attempts < config.retry_max - && file_meta.audio.artist.is_some() - && file_meta.audio.album.is_some() -{ - if let Err(_) = enrichment_queue.try_push(EnrichmentRequest { - file_id: file_meta.id, - artist: file_meta.audio.artist.unwrap(), - album: file_meta.audio.album.unwrap(), - }) { - // Queue full — file will be retried on next access - tracing::warn!( - file_id = ?file_meta.id, - "enrichment queue full, dropping request" - ); - metrics::ENRICHMENT_QUEUE_DROPS.inc(); - } - // Non-blocking: returns immediately with embedded tags -} -``` - -The `enrichment_attempts < retry_max` check prevents files that have -permanently failed enrichment (e.g., metadata-agregator has no match) -from being re-queued on every access. - -#### 4.3.7. Conflict Resolution - -| Source | Priority | Writes When | -|--------|----------|-------------| -| `orchestrator` | Highest | Always overwrites (full-system mode push) | -| `provider` | Medium | Only if current source is NULL or `'embedded'` | -| `embedded` | Lowest | Implicit default from file tag parsing | - -Conflict resolution is enforced **atomically at write time** using -conditional SQL (`WHERE enrichment_source IS NULL OR -enrichment_source = 'embedded'`), not at dequeue time. This prevents -the TOCTOU race where the orchestrator writes between the worker's -check and the worker's write. - -#### 4.3.8. Proto Changes Required - -The existing `UpdateMetadataRequest` in `musicfs.proto` must be -extended to carry the new enrichment fields: - -```protobuf -// Add to UpdateMetadataRequest: -optional string label = 40; -optional string album_type = 41; -optional string cover_url = 42; -``` - -> **Note on genres:** metadata-agregator returns `repeated Genre` -> (objects with `id` + `name`). The provider extracts genre names -> and stores them as a JSON array in `genres_json`. The singular -> `genre` field in `UpdateMetadataRequest` (already exists at -> field 9) is set to the first/primary genre for backward compat. - -#### 4.3.9. `cover_url` Usage - -`cover_url` is stored in the metadata overlay but is **not used by -musicfs for artwork embedding or display** in this plan. It is -stored for consumption by external tools (e.g., media players that -query musicfs's gRPC `GetMetadata` and fetch artwork themselves). -Artwork download and caching is deferred to future work. - -## 5. Cross-Cutting Concerns - -### 5.1. Security & Privacy - -- gRPC connection to metadata-agregator is plaintext (internal network). - TLS can be added via config if needed. -- No PII involved — only music metadata. -- No API keys stored in musicfs — metadata-agregator handles provider - auth. - -### 5.2. Observability - -New tracing spans and metrics: - -| Metric | Type | Description | -|--------|------|-------------| -| `enrichment_queue_depth` | Gauge | Current queue size | -| `enrichment_queue_drops_total` | Counter | Requests dropped (queue full) | -| `enrichment_inflight_albums` | Gauge | In-flight (artist, album) dedup set size | -| `enrichment_lookups_total` | Counter | Total provider lookups | -| `enrichment_hits_total` | Counter | Successful matches | -| `enrichment_misses_total` | Counter | No match found | -| `enrichment_errors_total` | Counter | Provider errors | -| `enrichment_skipped_total` | Counter | Skipped (higher-priority source already wrote) | -| `enrichment_latency_ms` | Histogram | Lookup latency | - -### 5.3. Scalability & Performance - -- Queue is bounded (default 256) — backpressure via `try_push`. -- Album-level deduplication: 12 tracks in same album = 1 lookup. -- No impact on file read latency — enrichment is fully async. -- metadata-agregator caches in PostgreSQL, so repeated lookups are - cheap. - -### 5.4. Testing Plan - -| Test | Type | Validates | -|------|------|-----------| -| `test_provider_connect` | Unit | gRPC connection setup | -| `test_lookup_match` | Unit (mock) | SearchAlbums → EnrichedMetadata mapping | -| `test_lookup_no_match` | Unit (mock) | Graceful handling of empty results, increments attempts | -| `test_enrichment_queue_push` | Unit | Queue push + in_flight dedup | -| `test_enrichment_queue_full_drops` | Unit | try_push fails gracefully, logs, increments metric | -| `test_enrichment_worker_writes_db` | Integration | DB write after lookup | -| `test_enrichment_atomic_conflict` | Integration | Orchestrator writes between dequeue and worker write → worker does NOT overwrite | -| `test_enrichment_retry_backoff` | Unit | Failed attempts increment counter, exponential backoff | -| `test_enrichment_max_attempts_stop` | Unit | After retry_max failures, file not re-queued | -| `test_config_disabled` | Unit | No queue/worker when disabled | -| `test_album_dedup_simultaneous` | Integration | 12 tracks opened at once → 1 API call | -| `test_genre_backward_compat` | Unit | genres_json stored as array, genre set to first entry | - -## 6. Alternatives Considered - -### 6.1. Native .so Plugin - -Rejected. Requires bundling a separate async runtime + tonic gRPC -stack inside a dynamically loaded library. ABI instability, duplicate -runtimes, and deployment complexity outweigh the "purity" of using the -plugin system. - -### 6.2. Direct MusicBrainz/Discogs/Last.fm HTTP Clients (week-12 plan) - -Rejected. metadata-agregator already handles these providers with rate -limiting, caching, and deduplication. Embedding HTTP clients in musicfs -would duplicate this work and couple musicfs to specific provider APIs. - -### 6.3. WASM Plugin - -Rejected. WASI networking is immature. gRPC over WASM adds unnecessary -latency and complexity. - -### 6.4. On-Demand Blocking Lookup - -Rejected. Blocking file access while waiting for a gRPC response would -cause latency spikes and kill media player UX. Background async is the -only acceptable approach. - -## 7. Implementation Plan - -### Phase 1: Foundation (Day 1) - -- [ ] Add `MetadataProviderConfig` to config.rs -- [ ] Add DB schema columns: `enrichment_source`, `enriched_at`, - `enrichment_attempts`, `last_enrichment_error`, `genres_json`, - `label`, `album_type`, `cover_url` -- [ ] Add `label`, `album_type`, `cover_url` fields to - `UpdateMetadataRequest` in `musicfs.proto` -- [ ] Extend `ExternalMetadata` struct -- [ ] Update `config.example.toml` - -### Phase 2: Provider + Worker (Day 1–2) - -- [ ] Implement `MetadataAgregatorProvider` (gRPC client wrapper) -- [ ] Implement `EnrichmentQueue` with `DashSet` in-flight dedup -- [ ] Implement `EnrichmentWorker` with: - - Atomic conditional write (`WHERE enrichment_source IS NULL OR ...`) - - Retry tracking (`enrichment_attempts`, exponential backoff) - - Album-level result caching -- [ ] Add queue drop logging + metrics -- [ ] Wire into startup (musicfs-cli) — conditional on config - -### Phase 3: Integration + Tests (Day 2) - -- [ ] Wire enrichment trigger in FUSE getattr/readdir path - (with `enrichment_attempts < retry_max` guard) -- [ ] Write unit tests: atomic conflict, queue drops, retry backoff, - max attempts, genre backward compat -- [ ] Write integration test: 12-track simultaneous dedup -- [ ] Write integration test with in-memory DB + mock gRPC server -- [ ] Update architecture.md with metadata provider component - -## 8. Glossary / References - -| Term | Definition | -|------|------------| -| metadata-agregator | Go gRPC service that searches MusicBrainz and caches results in PostgreSQL | -| Enrichment | Adding genres, label, artwork URL to file metadata beyond what's in file tags | -| Overlay | musicfs mechanism for serving modified metadata without changing origin files | -| `AudioMeta` | Core metadata struct extracted from file tags by symphonia | -| `ExternalMetadata` | Metadata returned by external providers (plugin trait) | -| `enrichment_source` | Tracks who last wrote metadata: `embedded`, `provider`, or `orchestrator` | - -- [metadata-agregator proto](../../../../metadata-agregator/proto/metadata/v1/metadata.proto) -- [musicfs-plugins traits](../../crates/musicfs-plugins/src/traits.rs) -- [musicfs-cache overlay](../../crates/musicfs-cache/src/overlay.rs) -- [architecture.md](../architecture.md) diff --git a/docs/v2/features/metadata-overlay.md b/docs/v2/features/metadata-overlay.md deleted file mode 100644 index 60fd3e7..0000000 --- a/docs/v2/features/metadata-overlay.md +++ /dev/null @@ -1,1010 +0,0 @@ -# Metadata Overlay: Design Doc - -**Authors:** AI Assistant -**Status:** Draft -**Last Updated:** 2026-05-17 -**Reviewers:** [TBD] -**Approvers:** [TBD] -**Prerequisites:** [architecture.md](../architecture.md), [requirements.md](../requirements.md) - ---- - -[TOC] - ---- - -## 1. Abstract - -Metadata Overlay enables MusicFS to serve **modified audio metadata** to -consumers (Jellyfin, Plex, mpv, VLC) while preserving original files on origin -storage. When a media server reads a file through the FUSE mount, it receives -metadata headers **generated on-the-fly** from the database, seamlessly spliced -with original audio data from the origin. - -**Key constraints:** -- Never modify origin files (read-only architecture) -- Never duplicate entire files (storage-efficient) -- Support all audio formats via pluggable format handlers -- Transparent to consumers (standard file I/O) - -**Solution approach:** Store metadata as individual database columns. On -`read()`, generate format-specific headers on-the-fly (~10-50 μs) and splice -them with original audio bytes using offset translation. No pre-generated -headers are stored. - ---- - -## 2. Background - -### 2.1 Current State - -MusicFS serves files with their **original embedded metadata**. The metadata -extraction flow is: - -``` -Origin File → symphonia parser → AudioMeta struct → SQLite DB → Virtual paths - ↓ -FUSE read() ← CAS chunks ← Origin (unchanged bytes) -``` - -The database stores metadata for virtual path generation and search, but file -content is served verbatim from origin/CAS. Only 12 metadata fields are -stored: title, artist, album, album_artist, genre, year, track, disc, -duration_ms, bitrate, sample_rate, format. - -### 2.2 Pain Points - -| Problem | Impact | -|---------|--------| -| Cannot fix incorrect tags | Jellyfin shows wrong artist/album | -| Cannot add missing metadata | Files with no tags appear as "Unknown" | -| Origin is authoritative | User edits require modifying source files | -| Breaks torrent seeding | Modifying origin invalidates checksums | -| Missing fields | Only 12 of ~30 standard fields stored | - -### 2.3 User Stories - -1. **Tag Correction:** "Origin files have 'The Beatles' tagged as 'Beatles, - The'. I want Jellyfin to see the correct name without modifying my NAS." - -2. **Missing Metadata:** "My FLAC rips have no album art or year. I want to - add them in MusicFS so Plex displays proper covers." - -3. **Torrent Preservation:** "My music is seeding. I can't modify files but - want correct tags in my media server." - -4. **Multi-Library Views:** "I want one physical file to appear in both - 'Classical' and 'Relaxation' collections with different metadata." - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -| ID | Goal | Success Metric | -|----|------|----------------| -| G1 | Serve modified metadata transparently | Players read edited tags without special handling | -| G2 | Zero origin modification | Origin files byte-identical before/after | -| G3 | Zero storage overhead for headers | No pre-generated header blobs stored | -| G4 | MP3 and FLAC out of the box | Other formats added on demand via plugins | -| G5 | Pluggable format handlers | Add new format support without core changes | -| G6 | Unified metadata model | Single API regardless of underlying format | -| G7 | Sub-second edit latency | Metadata changes reflected on next read | - -### 3.2 Non-Goals - -| ID | Non-Goal | Rationale | -|----|----------|-----------| -| NG1 | Audio transcoding | Out of scope; separate feature | -| NG2 | Lossless round-trip | Synthesized headers may differ structurally from original | -| NG3 | Writing back to origin | Violates read-only principle | -| NG4 | Video file support | Focus on audio; defer to future | -| NG5 | Metadata sync to external DBs | Jellyfin/Plex have their own; not our concern | - ---- - -## 4. Proposed Design - -### 4.1 High-Level Architecture - -```plantuml -@startuml -!theme plain -skinparam componentStyle rectangle - -package "FUSE Layer" { - [getattr()] as GA - [read()] as RD -} - -package "Overlay Engine" { - [OverlayReader] as OR - [FormatHandlerRegistry] as FHR -} - -package "Storage" { - database "SQLite\n(metadata columns\n+ format_layout)" as DB - [CAS\n(origin audio chunks)] as CAS -} - -package "Format Handlers (Pluggable)" { - [Id3v2Handler] as H1 - [FlacHandler] as H2 - [WavHandler\n(on demand)] as H3 - [OggHandler\n(on demand)] as H4 - [Mp4Handler\n(on demand)] as H5 -} - -GA --> OR : virtual_size? -RD --> OR : read(ino, offset, size) -OR --> DB : get metadata + layout -OR --> FHR : synthesize(metadata, layout) -FHR --> H1 -FHR --> H2 -FHR --> H3 -FHR --> H4 -FHR --> H5 -OR --> CAS : read audio bytes - -note right of OR - On-the-fly generation: - 1. Read metadata from DB columns - 2. Generate header (~10-50 μs) - 3. Splice header + CAS audio - 4. Return to FUSE -end note - -@enduml -``` - -### 4.2 Core Flows - -#### 4.2.1 Flow: Initial Ingest (Origin Scan) - -Triggered on mount or rescan. Extracts metadata from origin files and -populates all database columns. - -```plantuml -@startuml -!theme plain - -participant "Origin\nFederation" as OF -participant "CAS" as CAS -participant "Format\nHandler" as FH -participant "Metadata\nParser" as MP -database "SQLite" as DB -participant "Tantivy" as TI -participant "Virtual\nTree" as VT - -OF -> OF : Scan origin directory -loop for each audio file - OF -> CAS : Fetch file header (first 256KB) - CAS -> CAS : Chunk and store full file - CAS --> OF : ChunkManifest - - OF -> FH : analyze(header_bytes, file_size) - note right of FH - Detects format, returns - FormatLayout with audio_start, - audio_end, format_data - (e.g. STREAMINFO for FLAC) - end note - FH --> OF : FormatLayout - - OF -> MP : extract(header_bytes) - note right of MP - Uses symphonia to parse - all embedded tags - end note - MP --> OF : metadata fields - - OF -> DB : INSERT INTO files\n(all metadata columns,\nformat_layout, chunk_manifest) - OF -> TI : Index metadata - OF -> VT : Add virtual tree node -end - -@enduml -``` - -#### 4.2.2 Flow: FUSE read() with Overlay - -The core read path. Headers are generated on-the-fly from DB columns — -nothing pre-computed is stored. - -```plantuml -@startuml -!theme plain - -participant "FUSE\nKernel" as FK -participant "Overlay\nReader" as OR -database "SQLite" as DB -participant "Format\nHandler" as FH -participant "CAS" as CAS - -FK -> OR : read(ino, offset, size) -OR -> OR : Lookup file by inode - -OR -> DB : SELECT metadata columns,\nformat_layout WHERE id = ? -note right of DB : ~1 μs via page cache -DB --> OR : FileMetadataRow - -OR -> FH : synthesize(metadata, layout) -note right of FH - On-the-fly generation - ~10-50 μs, pure CPU -end note -FH --> OR : synthetic_header bytes - -OR -> OR : header_len = synthetic_header.len()\nvirtual_size = header_len + audio_len - -alt offset falls in header region - OR -> OR : Slice from synthetic_header -else offset falls in audio region - OR -> OR : origin_offset = audio_start\n+ (offset - header_len) - OR -> CAS : read(file_id, origin_offset, size) - CAS --> OR : audio bytes -else offset spans boundary - OR -> OR : Take header tail - OR -> CAS : read(file_id, audio_start, remaining) - CAS --> OR : audio bytes - OR -> OR : Concatenate header + audio -end - -OR --> FK : reply.data(spliced bytes) - -@enduml -``` - -#### 4.2.3 Flow: FUSE getattr() with Overlay - -Returns the **virtual** file size (synthetic header + audio) instead of -the origin file size. - -```plantuml -@startuml -!theme plain - -participant "FUSE\nKernel" as FK -participant "Virtual\nTree" as VT -database "SQLite" as DB -participant "Format\nHandler" as FH - -FK -> VT : getattr(ino) -VT -> VT : Lookup VirtualNode - -VT -> DB : Read format_layout for file_id -DB --> VT : FormatLayout -VT -> FH : estimate_header_size(metadata) -note right of FH - Fast estimate without - full header synthesis -end note -FH --> VT : estimated_header_len -VT -> VT : virtual_size = estimated_header_len\n+ (audio_end - audio_start) -VT --> FK : FileAttr with virtual_size - -@enduml -``` - -#### 4.2.4 Flow: Metadata Update (User Edits Tags) - -Triggered via gRPC API or CLI. Updates DB columns directly. Next read() -generates a new header automatically. - -```plantuml -@startuml -!theme plain - -actor "User" as U -participant "CLI /\ngRPC" as API -participant "Metadata\nService" as MS -database "SQLite" as DB -participant "Tantivy" as TI -participant "Virtual\nTree" as VT -participant "Event\nBus" as EB - -U -> API : musicfs metadata set\n--title "Fix" --artist "Fix" -API -> MS : UpdateMetadata(file_id, fields) - -alt merge mode (default) - MS -> DB : SELECT current metadata - DB --> MS : current row - MS -> MS : Overwrite only provided fields -end - -MS -> DB : UPDATE files SET title=?, artist=?\nWHERE id=? -DB --> MS : ok - -MS -> TI : Re-index updated file -MS -> VT : Recompute virtual path -note right of VT - If artist/album/title changed - the file moves in the tree -end note - -MS -> EB : Emit MetadataChanged -note right of EB - FUSE attr cache invalidation, - gRPC event subscribers -end note - -MS --> API : success -API --> U : done - -@enduml -``` - -#### 4.2.5 Flow: Metadata Clear (Revert to Original) - -Removes user overrides. File reverts to serving original embedded metadata. - -```plantuml -@startuml -!theme plain - -actor "User" as U -participant "CLI /\ngRPC" as API -participant "Metadata\nService" as MS -participant "CAS" as CAS -participant "Metadata\nParser" as MP -database "SQLite" as DB -participant "Tantivy" as TI -participant "Virtual\nTree" as VT -participant "Event\nBus" as EB - -U -> API : musicfs metadata clear -API -> MS : ClearOverlay(file_id) - -MS -> CAS : Read origin file header -CAS --> MS : header bytes -MS -> MP : extract(header_bytes) -MP --> MS : original metadata - -MS -> DB : UPDATE files SET all columns\nto original values -DB --> MS : ok - -MS -> TI : Re-index with original metadata -MS -> VT : Recompute virtual path -MS -> EB : Emit MetadataCleared - -MS --> API : success -API --> U : done - -@enduml -``` - -#### 4.2.6 Flow: Batch Import - -Import metadata from external source (CSV, JSON, MusicBrainz dump). - -```plantuml -@startuml -!theme plain - -actor "User" as U -participant "CLI /\ngRPC" as API -participant "Import\nEngine" as IE -database "SQLite" as DB -participant "Tantivy" as TI -participant "Event\nBus" as EB - -U -> API : musicfs metadata import\n--format csv metadata.csv -API -> IE : ImportMetadata(file, format) - -IE -> IE : Parse source file (CSV/JSON) -IE -> IE : Match rows to files by\npath, ISRC, or MusicBrainz ID - -IE -> DB : BEGIN TRANSACTION - -loop for each matched row - IE -> DB : UPDATE files SET matched columns - IE -> TI : Re-index file - IE --> API : stream progress -end - -IE -> DB : COMMIT - -IE -> EB : Emit BatchImportComplete - -IE --> API : final summary -API --> U : updated N, skipped M, errors K - -@enduml -``` - -### 4.3 Offset Translation - -``` -Virtual File (what consumer sees): -┌─────────────────────┬────────────────────────────────────────────┐ -│ Synthetic Header │ Original Audio │ -│ (N bytes) │ (M bytes) │ -│ generated on-fly │ from CAS │ -└─────────────────────┴────────────────────────────────────────────┘ -0 N N+M - ↑ ↑ - header_len virtual_size - -Origin File (on storage): -┌─────────────────────┬────────────────────────────────────────────┐ -│ Original Header │ Original Audio │ -│ (X bytes) │ (M bytes) │ -└─────────────────────┴────────────────────────────────────────────┘ -0 X X+M - ↑ ↑ - layout.audio_start layout.audio_end - -Offset Translation: - virtual_offset → origin_offset - - if virtual_offset < N: - return synthetic_header[virtual_offset] - else: - origin_offset = X + (virtual_offset - N) - return cas_read(file_id, origin_offset) -``` - -### 4.4 Format Handler Plugin System - -#### 4.4.1 Handler Trait - -```rust -/// Trait for format-specific metadata handling. -/// -/// Implementations handle: -/// 1. Analyzing original files to find audio boundaries -/// 2. Synthesizing new headers from database metadata -/// -/// Plugins implement this trait and register via FormatHandlerRegistry. -pub trait FormatHandler: Send + Sync + 'static { - fn id(&self) -> &'static str; - fn name(&self) -> &'static str; - fn extensions(&self) -> &[&'static str]; - fn mime_types(&self) -> &[&'static str]; - - /// Analyze file bytes to determine audio layout. - fn analyze(&self, data: &[u8], file_size: u64) -> Result; - - /// Synthesize header bytes from metadata. Called on every read(). - fn synthesize( - &self, - metadata: &FileMetadataRow, - layout: &FormatLayout, - ) -> Result, FormatError>; - - /// Extract metadata from header bytes (for initial ingest). - fn extract(&self, data: &[u8]) -> Result; - - /// Estimate header size without full synthesis (for getattr). - fn estimate_header_size(&self, metadata: &FileMetadataRow) -> usize { - 10 * 1024 // 10KB default - } -} -``` - -#### 4.4.2 Handler Registry - -```rust -pub struct FormatHandlerRegistry { - handlers: HashMap>, - extension_map: HashMap, -} - -impl FormatHandlerRegistry { - pub fn new() -> Self { - // Only MP3 and FLAC shipped by default. - // Other handlers registered via load_plugins() or register(). - let mut r = Self { .. }; - r.register(Arc::new(Id3v2Handler::new())); // .mp3 - r.register(Arc::new(FlacHandler::new())); // .flac - r - } - pub fn register(&mut self, handler: Arc) { /* ... */ } - pub fn get_by_extension(&self, ext: &str) -> Option> { /* ... */ } - pub fn load_plugins(&mut self, plugin_dir: &Path) -> Result { /* ... */ } -} -``` - -#### 4.4.3 Format Complexity Summary - -| Format | Handler | Complexity | Shipped | -|--------|---------|------------|---------| -| **MP3** | `Id3v2Handler` | Low | **Yes** — built-in | -| **FLAC** | `FlacHandler` | Low | **Yes** — built-in | -| **WAV** | `WavHandler` | Low | On demand | -| **OGG/Opus** | `OggHandler` | Medium | On demand | -| **M4A/MP4** | `Mp4Handler` | High | On demand | - -MP3 and FLAC cover the vast majority of music libraries. Other formats -use the same `FormatHandler` trait and can be added as plugins or built-in -handlers when needed — the architecture does not change. - -### 4.5 Database Schema - -All metadata fields are individual columns. SQLite NULL columns cost 0 bytes. -Only `format_layout` and `custom_tags` are blobs. - -```sql -PRAGMA journal_mode = WAL; -PRAGMA foreign_keys = ON; -PRAGMA synchronous = NORMAL; - -CREATE TABLE IF NOT EXISTS files ( - id INTEGER PRIMARY KEY, - origin_id TEXT NOT NULL, - real_path TEXT NOT NULL, - virtual_path TEXT NOT NULL, - - -- ═══ Core Identification ═══ - title TEXT, - artist TEXT, - album TEXT, - album_artist TEXT, - track_number INTEGER, - track_total INTEGER, - disc_number INTEGER, - disc_total INTEGER, - date TEXT, -- "2024" or "2024-05-17" - year INTEGER, -- extracted for convenience - genre TEXT, - - -- ═══ Credits ═══ - composer TEXT, - comment TEXT, - lyrics TEXT, - copyright TEXT, - compilation INTEGER, -- 0/1 - - -- ═══ Sorting ═══ - artist_sort TEXT, - album_artist_sort TEXT, - album_sort TEXT, - title_sort TEXT, - - -- ═══ MusicBrainz IDs ═══ - mb_recording_id TEXT, -- Recording MBID - mb_album_id TEXT, -- Release MBID - mb_artist_id TEXT, -- Artist MBID - mb_album_artist_id TEXT, -- Album Artist MBID - mb_release_group_id TEXT, -- Release Group MBID - - -- ═══ ReplayGain ═══ - replaygain_track_gain REAL, -- dB - replaygain_track_peak REAL, -- 0.0-1.0+ - replaygain_album_gain REAL, - replaygain_album_peak REAL, - - -- ═══ Technical (from audio stream, read-only) ═══ - duration_ms INTEGER, - bitrate INTEGER, -- kbps - sample_rate INTEGER, -- Hz - channels INTEGER, - bits_per_sample INTEGER, - format TEXT, -- "flac", "mp3", etc. - encoder TEXT, -- encoding software - - -- ═══ Custom Tags (overflow for non-standard fields) ═══ - custom_tags TEXT, -- JSON: {"ISRC":"US1234","LABEL":"Sony"} - - -- ═══ Format Layout (for byte-range splicing) ═══ - -- Stored as msgpack blob. Contains audio_start, audio_end, - -- format_data (STREAMINFO for FLAC, stco for MP4, etc.) - format_layout BLOB, - - -- ═══ Sync State ═══ - origin_mtime INTEGER NOT NULL, - origin_size INTEGER NOT NULL, - content_hash TEXT, - chunk_manifest BLOB, - last_sync INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), - - -- ═══ Trash (existing feature) ═══ - trashed INTEGER NOT NULL DEFAULT 0, - original_path TEXT, - trashed_at INTEGER, - - UNIQUE(origin_id, real_path) -); - --- ═══ Indexes ═══ -CREATE INDEX IF NOT EXISTS idx_files_virtual ON files(virtual_path); -CREATE INDEX IF NOT EXISTS idx_files_artist_album ON files(artist, album); -CREATE INDEX IF NOT EXISTS idx_files_content_hash ON files(content_hash); -CREATE INDEX IF NOT EXISTS idx_files_real ON files(origin_id, real_path); -CREATE INDEX IF NOT EXISTS idx_files_origin ON files(origin_id); -CREATE INDEX IF NOT EXISTS idx_files_last_sync ON files(last_sync); - -CREATE INDEX IF NOT EXISTS idx_files_trashed ON files(trashed) WHERE trashed = 1; -CREATE INDEX IF NOT EXISTS idx_files_mb_album ON files(mb_album_id); -CREATE INDEX IF NOT EXISTS idx_files_mb_artist ON files(mb_artist_id); -CREATE INDEX IF NOT EXISTS idx_files_genre ON files(genre); -CREATE INDEX IF NOT EXISTS idx_files_year ON files(year); -CREATE INDEX IF NOT EXISTS idx_files_composer ON files(composer); - --- ═══ Artwork (unchanged, separate table) ═══ -CREATE TABLE IF NOT EXISTS artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE, - art_type TEXT NOT NULL, - chunk_hash TEXT NOT NULL, - width INTEGER, - height INTEGER, - mime_type TEXT, - UNIQUE(file_id, art_type) -); - -CREATE INDEX IF NOT EXISTS idx_artwork_file ON artwork(file_id); - --- ═══ Collections (unchanged) ═══ -CREATE TABLE IF NOT EXISTS collections ( - id INTEGER PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - query_json TEXT NOT NULL, - created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), - updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')) -); - --- ═══ Directories (unchanged) ═══ -CREATE TABLE IF NOT EXISTS directories ( - id INTEGER PRIMARY KEY, - path TEXT NOT NULL UNIQUE, - created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')) -); - -CREATE INDEX IF NOT EXISTS idx_directories_path ON directories(path); -``` - -### 4.6 Read Algorithm - -```rust -impl OverlayReader { - pub async fn read( - &self, - file_id: FileId, - offset: u64, - size: u32, - ) -> Result { - let file = self.db.get_file(file_id)?; - let layout: FormatLayout = rmp_serde::from_slice(&file.format_layout)?; - let handler = self.registry.get_by_format(&file.format)?; - - // Generate header on-the-fly (~10-50 μs) - let header = handler.synthesize(&file, &layout)?; - let header_len = header.len() as u64; - let audio_len = layout.audio_end - layout.audio_start; - let virtual_size = header_len + audio_len; - let virtual_end = (offset + size as u64).min(virtual_size); - - if offset >= virtual_size { - return Ok(Bytes::new()); - } - - let mut result = BytesMut::with_capacity((virtual_end - offset) as usize); - - // Region 1: Synthetic header - if offset < header_len { - let end = virtual_end.min(header_len); - result.extend_from_slice(&header[offset as usize..end as usize]); - } - - // Region 2: Origin audio data - if virtual_end > header_len { - let audio_start = header_len.max(offset) - header_len; - let audio_size = (virtual_end - header_len.max(offset)) as u32; - let origin_offset = layout.audio_start + audio_start; - - let audio = self.cas_reader.read(file_id, origin_offset, audio_size).await?; - result.extend_from_slice(&audio); - } - - Ok(result.freeze()) - } -} -``` - -### 4.7 API Design - -#### 4.7.1 gRPC Extensions - -```protobuf -service MetadataService { - rpc GetMetadata(GetMetadataRequest) returns (MetadataResponse); - rpc UpdateMetadata(UpdateMetadataRequest) returns (UpdateMetadataResponse); - rpc ClearOverlay(ClearOverlayRequest) returns (ClearOverlayResponse); - rpc BatchUpdateMetadata(BatchUpdateRequest) returns (stream BatchUpdateProgress); - rpc ImportMetadata(ImportMetadataRequest) returns (stream ImportProgress); -} - -message UpdateMetadataRequest { - int64 file_id = 1; - // Only set fields you want to change. - // Unset fields are left as-is (merge behavior). - optional string title = 2; - optional string artist = 3; - optional string album = 4; - optional string album_artist = 5; - optional uint32 track_number = 6; - optional uint32 disc_number = 7; - optional string date = 8; - optional string genre = 9; - optional string composer = 10; - optional string comment = 11; - optional string lyrics = 12; - optional string copyright = 13; - optional bool compilation = 14; - optional string artist_sort = 15; - optional string album_artist_sort = 16; - optional string album_sort = 17; - optional string title_sort = 18; - optional string mb_recording_id = 20; - optional string mb_album_id = 21; - optional string mb_artist_id = 22; - optional float replaygain_track_gain = 30; - optional float replaygain_track_peak = 31; - optional float replaygain_album_gain = 32; - optional float replaygain_album_peak = 33; - map custom_tags = 50; -} -``` - -#### 4.7.2 CLI Interface - -Two ways to set metadata: **flags** for quick single-field edits, **JSON** -for bulk or complex updates. Both can be combined. - -```bash -# ── View ── - -# Print all metadata as JSON -musicfs metadata get "/Artist/Album/01 - Track.flac" - -# Print specific field -musicfs metadata get "/Artist/Album/01 - Track.flac" --field artist - -# ── Edit via flags (one field at a time or several) ── - -musicfs metadata set "/Artist/Album/01 - Track.flac" \ - --title "Corrected Title" - -musicfs metadata set "/Artist/Album/01 - Track.flac" \ - --artist "Corrected Artist" \ - --album-artist "Corrected Artist" \ - --year 2024 \ - --genre "Rock" - -# Every DB column has a corresponding flag: -# --title, --artist, --album, --album-artist, -# --track-number, --track-total, --disc-number, --disc-total, -# --date, --year, --genre, -# --composer, --comment, --lyrics, --copyright, --compilation, -# --artist-sort, --album-artist-sort, --album-sort, --title-sort, -# --mb-recording-id, --mb-album-id, --mb-artist-id, -# --mb-album-artist-id, --mb-release-group-id, -# --replaygain-track-gain, --replaygain-track-peak, -# --replaygain-album-gain, --replaygain-album-peak, -# --encoder - -# Set a custom tag (anything not in the standard set) -musicfs metadata set "/path/to/file" --custom ISRC=US1234567890 - -# ── Edit via JSON (any number of fields at once) ── - -# Inline JSON -musicfs metadata set "/Artist/Album/01 - Track.flac" --json '{ - "title": "Corrected Title", - "artist": "Corrected Artist", - "year": 2024, - "custom_tags": {"ISRC": "US1234567890", "LABEL": "Sony"} -}' - -# From file -musicfs metadata set "/Artist/Album/01 - Track.flac" --json @metadata.json - -# Flags and JSON can be combined (flags take precedence) -musicfs metadata set "/path/to/file" --json @base.json --year 2025 - -# ── Revert ── - -# Revert to original embedded metadata -musicfs metadata clear "/Artist/Album/01 - Track.flac" - -# ── Diff ── - -# Show what changed vs original -musicfs metadata diff "/Artist/Album/01 - Track.flac" - -# ── Batch ── - -# Import from CSV (columns map to field names) -musicfs metadata import --format csv metadata.csv - -# Import from JSON (array of objects with "path" or "file_id" key) -musicfs metadata import --format json metadata.json - -# Export -musicfs metadata export --output metadata.json -musicfs metadata export --query "artist:Beatles" --output beatles.json -``` - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -| Concern | Mitigation | -|---------|------------| -| Plugin isolation | Native plugins require explicit trust; future WASM sandboxing | -| No credential exposure | Overlays contain only metadata, never auth tokens | -| Backup/restore | All data in SQLite, included in standard backup | - -### 5.2 Observability - -**Metrics:** -``` -musicfs_overlay_files_modified # Files with user-edited metadata -musicfs_overlay_generation_us # Histogram: header generation time -musicfs_overlay_read_total # Reads served via overlay -``` - -**Logging:** -``` -INFO overlay.update file_id=123 fields=[title,artist] -DEBUG overlay.read file_id=123 offset=0 size=65536 generation_us=42 -WARN overlay.format file_id=456 error="No handler for format=opus" -``` - -### 5.3 Scalability & Performance - -| Metric | Target | Notes | -|--------|--------|-------| -| Header generation | <100 μs | ~10-50 μs typical, pure CPU | -| read() overhead vs passthrough | <5% | One DB read + one synthesize | -| getattr() overhead | <1 μs | estimate_header_size(), no full synthesis | -| Storage per file | 0 extra | Metadata already in columns | -| Memory (LRU cache) | Optional | Cache hot headers if profiling shows need | - -### 5.4 Testing Plan - -| Test Type | Coverage | -|-----------|----------| -| **Unit** | FormatHandler implementations, offset arithmetic | -| **Integration** | Full read path with overlays, DB round-trip | -| **Format Matrix** | Each format × {overlay on, overlay off} | -| **Fuzzing** | Malformed files, boundary offsets, huge metadata | -| **Player Compat** | mpv, VLC, Jellyfin, Plex, ffprobe | - ---- - -## 6. Alternatives Considered - -### 6.1 Alternative A: Pre-generate and Store Headers in DB - -**Description:** Synthesize headers on metadata update, store as BLOB. - -**Rejected Because:** -- 1-10 KB per file × 1M files = 1-10 GB unnecessary storage -- Cache invalidation complexity (must regenerate on any field change) -- Generation is <100 μs — faster than a SQLite BLOB read of that size -- More moving parts for no measurable benefit - -### 6.2 Alternative B: NFO Sidecar Files - -**Description:** Generate `.nfo` XML files alongside audio files. - -**Rejected Because:** -- Only works with players that support NFO (Jellyfin, Plex) -- mpv, VLC, foobar2000 read embedded tags only -- Not transparent to all consumers - -### 6.3 Alternative C: Full File Rewrite + CAS Cache - -**Description:** Rewrite entire file with new metadata, cache in CAS. - -**Rejected Because:** -- Doubles storage for modified files -- High CPU/memory on first access -- Defeats CAS deduplication - -### 6.4 Alternative D: Metadata Blobs Instead of Columns - -**Description:** Store metadata as a single msgpack/JSON blob per file. - -**Rejected Because:** -- Not directly queryable (no `WHERE artist = ?`) -- Not indexable -- SQLite NULL columns cost 0 bytes — no space savings from blobs -- Schema is self-documenting with columns -- Virtual path templates can reference any column directly - ---- - -## 7. Implementation Plan - -### 7.1 Phase 1: Schema Migration + Core Types (3 days) - -| Deliverable | Details | -|-------------|---------| -| Schema migration | Add new columns to files table | -| `FormatLayout` struct | Audio boundary description | -| `FormatHandler` trait | Plugin interface | -| `FormatHandlerRegistry` | Built-in handler registration | - -**Exit Criteria:** DB migrates cleanly, types compile. - -### 7.2 Phase 2: Ingest Pipeline Update (3 days) - -| Deliverable | Details | -|-------------|---------| -| Update symphonia parser | Extract all new fields | -| Format analysis on ingest | Run `analyze()` → store `format_layout` | -| Populate new DB columns | All fields written on scan | - -**Exit Criteria:** Full rescan populates all metadata columns. - -### 7.3 Phase 3: Read Path + MP3/FLAC (5 days) - -| Deliverable | Details | -|-------------|---------| -| `OverlayReader` | Splice logic in FUSE read() | -| `Id3v2Handler` | analyze + synthesize for MP3 | -| `FlacHandler` | analyze + synthesize for FLAC | -| FUSE getattr() | Return virtual_size | - -**Exit Criteria:** ffprobe/mpv reads modified MP3 and FLAC tags. - -### 7.4 Phase 4: API + CLI (3 days) - -| Deliverable | Details | -|-------------|---------| -| gRPC MetadataService | get, set, clear, batch, import | -| CLI commands | `musicfs metadata {get,set,clear,diff,import,export}` | - -**Exit Criteria:** Full API functional end-to-end. - -### 7.6 Rollout - -```toml -[experimental] -metadata_overlay = true # Enable overlay feature - -[metadata_overlay] -# Additional format handlers loaded from this directory -plugin_dir = "/etc/musicfs/format-plugins/" -``` - -**Files with no registered handler** for their format are served with -original bytes unchanged (passthrough). No error, no degradation. - ---- - -## 8. Glossary & References - -### 8.1 Glossary - -| Term | Definition | -|------|------------| -| **Overlay** | Mode where file serves user-edited metadata instead of original | -| **Synthetic Header** | Format-specific metadata bytes generated on-the-fly | -| **Format Layout** | Description of audio/metadata byte boundaries in origin file | -| **Offset Translation** | Converting virtual file offset to origin file offset | - -### 8.2 References - -| Document | Link | -|----------|------| -| ID3v2.4 Specification | https://id3.org/id3v2.4.0-structure | -| FLAC Format | https://xiph.org/flac/format.html | -| OGG Encapsulation | https://xiph.org/ogg/doc/rfc3533.txt | -| MP4 Specification | ISO/IEC 14496-12 | -| MusicBrainz Picard Tag Mapping | https://picard-docs.musicbrainz.org/en/appendices/tag_mapping.html | -| symphonia StandardTagKey | https://docs.rs/symphonia-core/0.5.4/symphonia_core/meta/enum.StandardTagKey.html | -| lofty-rs | https://github.com/Serial-ATA/lofty-rs | -| MusicFS Architecture | [architecture.md](../architecture.md) | - -### 8.3 New Dependencies - -| Crate | Version | Purpose | -|-------|---------|---------| -| lofty | 0.24+ | Metadata header generation (all formats) | diff --git a/docs/v2/features/mkdir.md b/docs/v2/features/mkdir.md deleted file mode 100644 index 39d6a39..0000000 --- a/docs/v2/features/mkdir.md +++ /dev/null @@ -1,105 +0,0 @@ -**Date**: 2026-05-17 -**Status**: Shipped - -# Feature: Create Directory (mkdir) - -## Overview - -MusicFS supports creating directories in the virtual filesystem. This enables organizing files into custom folder structures beyond the auto-generated metadata-based layout. - -## Behavior - -### Basic Usage - -```bash -mkdir "/mnt/music/New Artist" -mkdir "/mnt/music/New Artist/New Album" -``` - -- Creates empty directory at specified path -- Parent directory must exist -- Standard POSIX semantics - -### Nested Directories - -```bash -# This works (shell handles -p) -mkdir -p "/mnt/music/A/B/C" - -# Equivalent to: -mkdir "/mnt/music/A" -mkdir "/mnt/music/A/B" -mkdir "/mnt/music/A/B/C" -``` - -The `-p` flag is handled by the shell, which makes multiple `mkdir` syscalls. - -### Brace Expansion - -```bash -# Shell expands this to multiple mkdir calls -mkdir "/mnt/music/Artist/{Album1,Album2,Album3}" - -# Equivalent to: -mkdir "/mnt/music/Artist/Album1" -mkdir "/mnt/music/Artist/Album2" -mkdir "/mnt/music/Artist/Album3" -``` - -Brace expansion is shell functionality, not filesystem. - -## Error Codes - -| Condition | Error | -|-----------|-------| -| Parent doesn't exist | `ENOENT` | -| Path already exists | `EEXIST` | - -## Persistence - -**Empty directories persist across remounts.** - -- User-created directories are stored in the `directories` table -- On mount, directories are restored from database -- Directories survive even when empty - -## Use Cases - -### Organizing Downloads - -```bash -# Create structure -mkdir "/mnt/music/Unsorted" -mkdir "/mnt/music/Unsorted/2026" - -# Move untagged files -mv "/mnt/music/Unknown Artist/Unknown Album/"*.flac "/mnt/music/Unsorted/2026/" -``` - -### Custom Collections - -```bash -# Create playlist-like structure -mkdir "/mnt/music/_Playlists" -mkdir "/mnt/music/_Playlists/Road Trip" - -# Move tracks (they'll still be in original location too - wait, no they won't) -# Note: mv moves, doesn't copy -``` - -## Implementation - -| Component | File | -|-----------|------| -| Tree | `crates/musicfs-cache/src/tree.rs` | -| FUSE | `crates/musicfs-fuse/src/filesystem.rs` | - -### Key Functions - -- `VirtualTree::mkdir()` - Create directory node in tree -- `Filesystem::mkdir()` - FUSE operation handler - -## Limitations - -- **No permissions**: Mode/umask parameters are ignored (always 0755) -- **No ownership**: UID/GID set to mounting user diff --git a/docs/v2/features/mv.md b/docs/v2/features/mv.md deleted file mode 100644 index ddd3671..0000000 --- a/docs/v2/features/mv.md +++ /dev/null @@ -1,94 +0,0 @@ -**Date**: 2026-05-17 -**Status**: Shipped - -# Feature: Move/Rename (mv) - -## Overview - -MusicFS supports moving and renaming files and directories within the virtual filesystem. Moves are persisted to the SQLite database and survive remounts. - -## Behavior - -### File Rename - -```bash -mv "/mnt/music/Artist/Album/old.flac" "/mnt/music/Artist/Album/new.flac" -``` - -- Renames file within same directory -- Updates `virtual_path` in database -- Original file on origin is unchanged - -### File Move - -```bash -mv "/mnt/music/Artist/Album/track.flac" "/mnt/music/Other Artist/Other Album/track.flac" -``` - -- Moves file to different directory -- **Requires target directory to exist** (use `mkdir` first) -- Returns `ENOENT` if target parent doesn't exist - -### Directory Rename - -```bash -mv "/mnt/music/Old Artist" "/mnt/music/New Artist" -``` - -- Renames directory and all descendants -- All files under the directory have their `virtual_path` updated in DB -- Single atomic operation - -### Directory Move - -```bash -mv "/mnt/music/Artist/Album" "/mnt/music/Other Artist/Album" -``` - -- Moves directory subtree to new parent -- **Requires target parent to exist** -- Returns `ENOENT` if target parent doesn't exist - -## Error Codes - -| Condition | Error | -|-----------|-------| -| Source doesn't exist | `ENOENT` | -| Target already exists | `EEXIST` | -| Target parent doesn't exist | `ENOENT` | -| Source is file but treated as dir | `EISDIR` | -| Source is dir but treated as file | `ENOTDIR` | - -## Persistence - -- File moves: `virtual_path` column updated in `files` table -- Directory moves: All matching `virtual_path` entries updated with new prefix -- User directories: Tracked in separate `directories` table -- Changes persist across unmount/remount cycles - -On mount, the CLI: -1. Scans origin files -2. For each file, checks DB for stored `virtual_path` (by origin_id + real_path) -3. Uses stored path if found, otherwise generates from metadata -4. Restores user-created directories from `directories` table - -## Limitations - -- **Read-only content**: File contents cannot be modified, only paths -- **No cross-origin moves**: All files remain on their original origin -- **No overwrite**: Moving to existing path fails (no implicit delete) - -## Implementation - -| Component | File | -|-----------|------| -| Database | `crates/musicfs-cache/src/db.rs` | -| Tree | `crates/musicfs-cache/src/tree.rs` | -| FUSE | `crates/musicfs-fuse/src/filesystem.rs` | - -### Key Functions - -- `Database::update_virtual_path()` - Update single file path -- `Database::rename_directory()` - Bulk update paths with prefix -- `VirtualTree::rename_file()` - Move file node in tree -- `VirtualTree::rename_directory()` - Move directory subtree diff --git a/docs/v2/features/rm.md b/docs/v2/features/rm.md deleted file mode 100644 index 6357d14..0000000 --- a/docs/v2/features/rm.md +++ /dev/null @@ -1,166 +0,0 @@ -**Date**: 2026-05-17 -**Status**: Shipped - -# Feature: Remove (rm) - -## Overview - -MusicFS supports removing files and directories. Deleted files are moved to a virtual `/.trash/` directory and can be restored. The trash is browsable — users can manually move files out. - -## Behavior - -### Remove File - -```bash -rm "/mnt/music/Artist/Album/track.flac" -``` - -- File moves to `/.trash/Artist/Album/track.flac` -- Original directory structure preserved in trash -- File still accessible via `/.trash/` path -- Database marks file as `trashed=1` with original path stored - -### Remove Empty Directory - -```bash -rmdir "/mnt/music/Empty Folder" -``` - -- Removes empty directory from tree -- Removes from `directories` table if user-created -- Fails with `ENOTEMPTY` if directory has children - -### Remove Directory Recursively - -```bash -rm -rf "/mnt/music/Artist" -``` - -- Shell handles recursion (depth-first unlink + rmdir) -- All files moved to `/.trash/Artist/...` -- Empty directories removed after files are trashed - -## The `.trash/` Directory - -Deleted files live in `/.trash/` with their original path structure: - -``` -/.trash/ -├── Artist/ -│ └── Album/ -│ ├── track1.flac -│ └── track2.flac -└── Other Artist/ - └── song.flac -``` - -### Browse Trash - -```bash -ls "/.trash/" -ls "/.trash/Artist/Album/" -``` - -### Manual Restore - -```bash -# Move file back manually - trashed flag is automatically cleared -mv "/.trash/Artist/Album/track.flac" "/Artist/Album/" -``` - -When moving a file out of `/.trash/`, the database `trashed` flag is automatically cleared. - -## CLI Commands - -All trash commands require either `--config` or `--cache-dir`: - -```bash -musicfs trash -c config.toml -musicfs trash --cache-dir ./dev/cache/musicfs -``` - -### List Deleted Files - -```bash -musicfs trash -c config.toml list -musicfs trash -c config.toml list --origin local-storage -musicfs trash -c config.toml list --since 7d -musicfs trash -c config.toml list --path "/Artist" -``` - -Output shows index, deletion time, and original path. - -### Restore Files - -```bash -# Restore single file or folder -musicfs trash -c config.toml restore "/Artist/Album/track.flac" - -# Restore entire folder recursively -musicfs trash -c config.toml restore "/Artist" - -# Restore everything -musicfs trash -c config.toml restore --all -``` - -CLI restore writes paths to a pending restore file and sends SIGHUP to the daemon. -The daemon processes pending restores and moves files back from `/.trash/`. - -### Empty Trash - -```bash -# Permanently delete all trashed files -musicfs trash -c config.toml empty - -# Delete old items only -musicfs trash -c config.toml empty --older-than 30d - -# Delete by path pattern -musicfs trash -c config.toml empty --pattern "/Artist" -``` - -**Warning:** Empty permanently removes files from MusicFS database. Origin files are unaffected. - -## Error Codes - -| Condition | Error | -|-----------|-------| -| Path doesn't exist | `ENOENT` | -| `rm` on directory (without `-r`) | `EISDIR` | -| `rmdir` on file | `ENOTDIR` | -| `rmdir` on non-empty directory | `ENOTEMPTY` | -| `rmdir` on `/.trash/` | `EPERM` | - -## Database Schema - -Files table extended with trash columns: - -```sql -trashed INTEGER NOT NULL DEFAULT 0, -original_path TEXT, -trashed_at INTEGER -``` - -Partial index for efficient trash queries: -```sql -CREATE INDEX idx_files_trashed ON files(trashed) WHERE trashed = 1; -``` - -## How It Works - -1. **Delete (`rm`)**: FUSE `unlink` moves file to `/.trash/`, marks `trashed=1` in DB -2. **Manual restore (`mv`)**: Moving out of `/.trash/` automatically clears `trashed` flag -3. **CLI restore**: Writes pending paths, sends SIGHUP to daemon, daemon processes restores -4. **Empty**: Deletes matching records from database - -## Persistence - -- Trashed files persist across remounts (stored in `/.trash/` subtree) -- Files marked with `trashed=1`, `original_path`, `trashed_at` in database -- PID file at `{cache_dir}/musicfs.pid` for CLI→daemon communication - -## Limitations - -- **No hard delete of remote files**: Origin content is never modified -- **Trash uses virtual space**: Files still in tree under `/.trash/` until emptied -- **CLI restore requires running daemon**: Manual `mv` works without daemon diff --git a/docs/v2/mvp-performance-review.md b/docs/v2/mvp-performance-review.md deleted file mode 100644 index b88e6e4..0000000 --- a/docs/v2/mvp-performance-review.md +++ /dev/null @@ -1,239 +0,0 @@ -# MusicFS MVP Performance Review - -**Date**: 2026-05-12 -**Test Data**: Metallica - 72 Seasons (12 FLAC tracks, 625MB, 16-bit/44.1kHz) -**Origin**: Local filesystem (Docker volume) -**System**: Linux, NixOS - ---- - -## Executive Summary - -**Phase 1 MVP is functional** - the system mounts, browses, and reads files successfully. Audio playback works with valid FLAC headers served. However, there's a **critical gap** between the architecture specification and current implementation regarding content chunking. - ---- - -## Benchmark Results - -### Throughput Comparison - -| Metric | Direct FS | MusicFS Cold | MusicFS Warm | Target (Spec) | Status | -|--------|-----------|--------------|--------------|---------------|--------| -| Single file read (64MB) | 0.022s (3 GB/s) | 0.035s (1.8 GB/s) | 0.020s (3.2 GB/s) | >500 MB/s | ✅ | -| Full album read (625MB) | 0.149s (4.2 GB/s) | 0.274s (2.3 GB/s) | 0.211s (3.0 GB/s) | >500 MB/s | ✅ | - -### Metadata Operations - -| Operation | Result | Target (Spec) | Status | -|-----------|--------|---------------|--------| -| Root listing | 0.006s | <10ms | ✅ | -| Full tree traversal (12 files) | 0.007s | <50ms | ✅ | -| stat() per operation | 0.003s | <1ms | ⚠️ | -| 4KB small reads (per op) | 0.006s | <1ms | ⚠️ | -| Random seek 1MB | 0.008-0.015s | <50ms | ✅ | -| Mount time | ~8ms | <500ms | ✅ | - -### Cache Performance - -| Metric | Value | -|--------|-------| -| Cache speedup (single file) | 1.75x | -| Cache speedup (full album) | 1.30x | -| Cache size | 25MB | -| Chunk count | 12 | -| Expected cache size | 625MB | - -### FUSE Overhead - -| Scenario | Overhead vs Direct | -|----------|-------------------| -| Single file cold cache | 59% slower | -| Single file warm cache | 9% faster* | -| Full album cold cache | 84% slower | -| Full album warm cache | 42% slower | - -*Warm cache appears faster due to OS page cache effects on both paths. - ---- - -## What's Working Well ✅ - -### 1. Mount Performance -- Mount completes in ~8ms (spec: <500ms) — **62x better than target** -- O(1) mount time achieved — no file scanning blocks mount -- Lazy loading working as designed per architecture section 4.3.1 - -### 2. Virtual Tree Organization -- Correct Artist/Album/Track hierarchy derived from metadata -- Example path: `/Metallica/72 Seasons/01. 72 Seasons.flac` -- Special character sanitization working (`/`, `\`, `:`, etc.) - -### 3. File Reading -- Valid FLAC headers served (`fLaC` magic bytes verified) -- Sequential reads work correctly -- Random access (seek) functional -- Concurrent reads from multiple processes work - -### 4. FUSE Integration -- Read-only enforcement (EROFS returned on write attempts) -- Proper inode assignment and file attributes -- AllowOther mount option working -- Clean unmount via fusermount3 - -### 5. Throughput -- Exceeds 500 MB/s target significantly (2-3 GB/s achieved) -- Parallel reads scale appropriately (4 files in 0.060s) - ---- - -## Critical Issues 🔴 - -### Issue 1: Incomplete File Caching - -**Symptom**: Cache is 25MB instead of expected 625MB (12 files × ~2MB each instead of full files) - -**Root Cause**: In `fetcher.rs:74`: -```rust -let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await?; -``` - -And in `local.rs:96-98`: -```rust -let mut buffer = vec![0u8; size as usize]; -let bytes_read = file.read(&mut buffer).await?; -buffer.truncate(bytes_read); -``` - -`tokio::fs::File::read()` reads **up to** buffer size but returns when the kernel buffer is exhausted (~2MB typical). Only first ~2MB of each file is being cached. - -**Impact**: -- Subsequent reads beyond 2MB offset hit origin every time -- No cache benefit for majority of file content -- Cache eviction policy not being exercised - -**Required Fix**: Use `read_to_end()` or loop until all bytes read: -```rust -let mut buffer = Vec::with_capacity(size as usize); -file.read_to_end(&mut buffer).await?; -``` - -### Issue 2: No CDC Chunking Implemented - -**Architecture Spec** (Section 4.3.2): -> "All file content is stored as content-addressed chunks... Avg chunk: 64KB, Min: 16KB, Max: 256KB" - -**Current Implementation**: Each file stored as ONE chunk (no FastCDC integration) - -**Impact**: -- No content deduplication possible -- Delta sync impossible (FR-11.2 unmet) -- Cache efficiency severely reduced for similar files - ---- - -## Architecture Gaps 🟡 - -| Spec Requirement | Current State | Gap | -|------------------|---------------|-----| -| CDC chunking (64KB avg) | No chunking | Missing FastCDC integration | -| Delta sync (>90% bandwidth reduction) | Not implemented | Requires CDC first | -| Deduplication (FR-20) | Not implemented | Requires CDC first | -| Search engine (tantivy) | Not implemented | Phase 3 scope | -| gRPC Control API | Not implemented | Phase 4 scope | -| Multi-origin federation | Single origin only | Phase 2 scope | -| Metadata persistence (SQLite) | In-memory HashMap | Missing persistence | - ---- - -## Performance Analysis - -### Why Warm Cache Appears Faster Than Direct FS - -The warm cache shows 3.2 GB/s vs direct 3.0 GB/s because: -1. OS page cache is warm for both MusicFS chunks AND origin files -2. Both measurements are essentially hitting RAM, variance expected -3. MusicFS chunks may have slightly better cache locality - -### stat() Latency Above Target - -Current: 3ms per stat() vs target <1ms - -Possible causes: -1. `RwLock` contention overhead -2. HashMap lookup plus FUSE context switch -3. Measurement includes full round-trip through FUSE - -Mitigation options: -- Consider lock-free concurrent data structures -- Implement finer-grained locking -- Cache hot inodes in separate fast-path structure - ---- - -## Recommendations - -### Immediate Fixes (Before Phase 2) - -1. **Fix file reading** — Use `read_to_end()` or implement proper streaming read loop -2. **Add CDC chunking** — Integrate FastCDC per architecture spec section 4.3.2 -3. **Persist metadata** — Move from in-memory HashMap to SQLite as specified - -### Phase 2 Priorities - -1. Complete CDC chunking implementation (prerequisite for delta sync) -2. Add SQLite metadata persistence (FR-7.2) -3. Implement multi-origin support (FR-13) - -### Testing Gaps to Address - -1. No automated E2E tests for real FUSE operations -2. No stress testing with concurrent access patterns -3. No large library testing (target: 1M+ files per NFR-3.1) -4. No offline mode testing (origin unavailable scenarios) - ---- - -## Test Environment Details - -``` -Origin Path: /home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/ -Mount Point: /tmp/musicfs-benchmark/mount -Cache Dir: /tmp/musicfs-benchmark/cache -Binary: target/release/musicfs (via nix develop) - -Files: - 01. 72 Seasons.flac 64MB - 02. Shadows Follow.flac 50MB - 03. Screaming Suicide.flac 45MB - 04. Sleepwalk My Life Away.flac 54MB - 05. You Must Burn!.flac 57MB - 06. Lux Æterna.flac 27MB - 07. Crown Of Barbed Wire.flac 46MB - 08. Chasing Light.flac 55MB - 09. If Darkness Had A Son.flac 51MB - 10. Too Far Gone_.flac 37MB - 11. Room Of Mirrors.flac 45MB - 12. Inamorata.flac 89MB - Total: 625MB, 12 tracks -``` - ---- - -## Conclusion - -**The MVP demonstrates core functionality works** — mounting, browsing, and reading audio files through FUSE. Throughput performance exceeds targets significantly. - -However, **the cache implementation is incomplete**: -- Only ~4% of file content is being cached (25MB/625MB) -- No CDC chunking means no deduplication or delta sync capability -- Architecture requirements FR-8.2, FR-11.2, FR-20 are unmet - -**Recommendation**: Fix the file reading issue and add CDC chunking before proceeding to Phase 2. The architecture is sound; implementation needs to catch up to specification. - ---- - -## References - -- [Architecture Specification](architecture.md) — Section 4.3.2 (CAS), Section 4.3.5 (Read Flow) -- [Requirements Specification](requirements.md) — FR-8 (Content Cache), FR-11 (Delta Sync), FR-20 (CAS) -- [Week 4b Plan](plans/week-04b-origin-connector.md) — ContentFetcher implementation diff --git a/docs/v2/plans/logging-comprehensive.md b/docs/v2/plans/logging-comprehensive.md deleted file mode 100644 index 0d9855c..0000000 --- a/docs/v2/plans/logging-comprehensive.md +++ /dev/null @@ -1,982 +0,0 @@ -# Comprehensive Logging Plan - -**Goal**: Add production-grade logging with trace-level observability, file rotation, and systemd integration -**Effort**: ~10-12 hours -**Dependencies**: Existing libraries only (no custom code) - -> **Review Status**: Reviewed by Oracle - all gaps addressed - ---- - -## Libraries Used - -| Need | Library | Status | -|------|---------|--------| -| Instrumentation | `tracing` | Already in workspace | -| Subscriber/filtering | `tracing-subscriber` | Already in workspace | -| File rotation | `tracing-appender` | Add to workspace | -| systemd journal | `tracing-journald` | Add to workspace | -| Compression | `logrotate` (Linux tool) | Config file only | - ---- - -## Phase 1: Config & Dependencies (2 hours) - -### 1.1 Add dependencies to workspace - -```toml -# Cargo.toml [workspace.dependencies] -tracing-appender = "0.2" -tracing-journald = "0.3" -``` - -```toml -# crates/musicfs-cli/Cargo.toml -tracing-appender.workspace = true -tracing-journald.workspace = true -``` - -### 1.2 Add LoggingConfig to config.rs - -```rust -// crates/musicfs-core/src/config.rs - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct Config { - pub mount_point: PathBuf, - pub cache_dir: PathBuf, - pub origins: Vec, - #[serde(default)] - pub cache: CacheConfig, - #[serde(default)] - pub health: HealthConfig, - #[serde(default)] - pub logging: LoggingConfig, // NEW -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct LoggingConfig { - #[serde(default = "default_log_dir")] - pub log_dir: PathBuf, - - #[serde(default)] - pub json_output: bool, - - #[serde(default = "default_true")] - pub journald: bool, - - #[serde(default = "default_log_level")] - pub level: String, -} - -impl Default for LoggingConfig { - fn default() -> Self { - Self { - log_dir: default_log_dir(), - json_output: false, - journald: true, - level: default_log_level(), - } - } -} - -fn default_log_dir() -> PathBuf { - PathBuf::from("/var/log/musicfs") -} -fn default_log_level() -> String { - "musicfs=info,warn".to_string() -} -fn default_true() -> bool { - true -} -``` - -### 1.3 Expand init_logging() in main.rs - -```rust -// crates/musicfs-cli/src/main.rs - -use tracing_appender::non_blocking::WorkerGuard; -use tracing_subscriber::{fmt, prelude::*, EnvFilter}; - -fn init_logging(config: &LoggingConfig) -> Result { - std::fs::create_dir_all(&config.log_dir)?; - - // File layer with daily rotation - let file_appender = tracing_appender::rolling::daily(&config.log_dir, "musicfs.log"); - let (non_blocking, guard) = tracing_appender::non_blocking(file_appender); - - let file_layer = if config.json_output { - fmt::layer() - .json() - .with_writer(non_blocking) - .with_ansi(false) - .boxed() - } else { - fmt::layer() - .with_writer(non_blocking) - .with_ansi(false) - .boxed() - }; - - // Journald layer (Linux only) - #[cfg(target_os = "linux")] - let journald_layer = if config.journald { - tracing_journald::layer() - .ok() - .map(|l| l.with_syslog_identifier("musicfs".to_string())) - } else { - None - }; - - // Stderr layer for interactive use - let stderr_layer = fmt::layer() - .with_writer(std::io::stderr) - .compact(); - - // Filter from config or env - let filter = EnvFilter::try_from_default_env() - .unwrap_or_else(|_| EnvFilter::new(&config.level)); - - // Compose - let subscriber = tracing_subscriber::registry() - .with(filter) - .with(file_layer) - .with(stderr_layer); - - #[cfg(target_os = "linux")] - let subscriber = subscriber.with(journald_layer); - - subscriber.init(); - - tracing::info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting"); - Ok(guard) -} -``` - -### 1.4 Add logrotate config - -```bash -# dist/logrotate.d/musicfs -/var/log/musicfs/*.log { - daily - rotate 30 - compress - delaycompress - missingok - notifempty - create 0640 musicfs musicfs -} -``` - ---- - -## Phase 2: Add tracing to musicfs-core (1 hour) - -### 2.1 Add dependency - -```toml -# crates/musicfs-core/Cargo.toml -[dependencies] -tracing.workspace = true # ADD THIS -``` - -### 2.2 Instrument core modules - -| File | What to Add | -|------|-------------| -| `config.rs` | Log config file loading, parse errors | -| `credentials.rs` | Log credential loading (redacted values) | -| `events.rs` | Log event publishing with counts | - ---- - -## Phase 3: Instrument Hot Paths (4 hours) - -### Priority order by impact - -| Crate | Files | What to Add | -|-------|-------|-------------| -| musicfs-fuse | `filesystem.rs` | `#[instrument]` on all FUSE ops, trace at decision points | -| musicfs-origins | `failover.rs`, `health.rs`, `router.rs` | Retry loops, state transitions, selection logic | -| musicfs-cache | `tree.rs`, `metadata.rs` | Tree mutations, cache hit/miss | -| musicfs-cas | `reader.rs`, `store.rs` | Chunk operations, dedup decisions | -| musicfs-sync | `delta.rs`, `watcher.rs` | Change detection, file events | - -### Instrumentation patterns - -```rust -// Function level - add to all public async functions -#[tracing::instrument(level = "debug", skip(self), fields(path = %path))] -pub async fn read(&self, path: &str) -> Result { - // ... -} - -// Decision points - add trace! at match/if branches -match result { - Ok(data) => { - tracing::trace!(bytes = data.len(), "read success"); - data - } - Err(e) => { - tracing::trace!(error = %e, "read failed"); - return Err(e); - } -} - -// State changes - use info! for important transitions -tracing::info!(old = ?old_status, new = ?new_status, origin = %id, "health changed"); - -// Cache operations -tracing::trace!(hit = true, fresh = true, "cache hit"); -tracing::trace!(hit = false, "cache miss"); -``` - -### FUSE operations (filesystem.rs) - highest priority - -| Operation | Level | Fields | -|-----------|-------|--------| -| `lookup()` | debug | parent, name, result_ino | -| `getattr()` | debug | ino, file_type | -| `readdir()` | debug | ino, entry_count | -| `read()` | debug | ino, offset, size, bytes_read | -| `open()` | debug | ino, flags | -| `release()` | trace | ino | - -### Origin operations - critical for debugging - -| Function | Level | Fields | -|----------|-------|--------| -| `read_with_failover()` | debug | path, origins_tried, success | -| `read_with_retry()` | trace | origin, attempt, success | -| `check_health()` | debug | origin, old_status, new_status | -| `select_origin()` | trace | candidates, selected, reason | - ---- - -## Phase 4: Update Production Files (1 hour) - -### 4.1 Update systemd service - -```ini -# dist/musicfs.service (add these lines) -Environment="RUST_LOG=musicfs=info,warn" -StandardOutput=journal -StandardError=journal -SyslogIdentifier=musicfs -RateLimitIntervalSec=30s -RateLimitBurst=1000 -``` - -### 4.2 Example config.toml - -```toml -# dist/config.example.toml -mount_point = "/mnt/music" -cache_dir = "/var/cache/musicfs" - -[logging] -log_dir = "/var/log/musicfs" -json_output = true -journald = true -level = "musicfs=info,warn" - -[cache] -metadata_cache_mb = 100 -content_cache_gb = 10 - -[health] -check_interval_secs = 30 -timeout_ms = 5000 - -[[origins]] -id = "local" -origin_type = "local" -priority = 1 -path = "/srv/music" -``` - ---- - -## Detailed Log Locations by Level - -### ERROR Level (25+ locations) - Unrecoverable Failures - -| File | Line | Log Message | -|------|------|-------------| -| `musicfs-grpc/src/webhook.rs` | 43 | `error!("Failed to initialize webhook HTTP client: {error}")` | -| `musicfs-grpc/src/webhook.rs` | 133 | `error!("Invalid HMAC secret key for webhook signature: {error}")` | -| `musicfs-plugins/src/manager.rs` | 272 | `error!("Plugin manager initialization failed: {error}")` | -| `musicfs-plugins/src/wasm.rs` | 142,183 | `error!("WASM plugin host initialization failed: {error}")` | -| `musicfs-search/src/index.rs` | 211,217 | `error!("Search index corrupted: failed to deserialize at position {pos}")` | -| `musicfs-cas/src/store.rs` | 105 | `error!("CAS chunk not found: {hash} - possible data loss")` | -| `musicfs-cas/src/store.rs` | 124-131 | `error!("CAS integrity check failed: expected {expected}, got {actual}")` | -| `musicfs-fuse/src/filesystem.rs` | 103 | `error!("Failed to mount filesystem at {mountpoint}: {error}")` | -| `musicfs-origins/src/failover.rs` | 76 | `error!("No origins available for path {path}")` | -| `musicfs-origins/src/failover.rs` | 125,186 | `error!("Max retries ({max_attempts}) exceeded for origin {origin_id}")` | -| `musicfs-origins/src/nfs.rs` | 63 | `error!("NFS stale file handle after {max_retries} retries for {path}")` | -| `musicfs-cas/src/reader.rs` | 75 | `error!("File manifest not found for file_id {file_id}")` | -| `musicfs-cas/src/fetcher.rs` | 60,68 | `error!("File/Origin not found for file_id {file_id}")` | -| `musicfs-search/src/indexer.rs` | 44,56 | `error!("Search indexer/commit failed: {error}")` | -| `musicfs-sync/src/watcher.rs` | 36,59,63 | `error!("Watcher failed for origin {origin_id}: {error}")` | - -### WARN Level (50+ locations) - Recoverable Issues - -| Category | File | Line | Log Message | -|----------|------|------|-------------| -| **Retry Logic** | `failover.rs` | 90 | `warn!("Origin {origin_id} failed: {error}, trying next (attempt {n}/{total})")` | -| **Retry Logic** | `failover.rs` | 111-118 | `warn!("Retrying origin {origin_id} after {delay:?} (attempt {n}/{max})")` | -| **Retry Logic** | `nfs.rs` | 47-52 | `warn!("NFS stale handle for {path} (attempt {n}/{max}), retrying")` | -| **Retry Logic** | `smb.rs` | 45 | `warn!("SMB connection lost (ENOTCONN), retrying (attempt {n}/{max})")` | -| **Retry Logic** | `webhook.rs` | 94-108 | `warn!("Webhook delivery failed to {url} (attempt {n}/{max}): {error}")` | -| **Fallback** | `failover.rs` | 70-73 | `warn!("No healthy origins for {path}, using fallback {origin_id}")` | -| **Timeout** | `smb.rs` | 107-109 | `warn!("SMB health check timed out after 5s for {origin_id}")` | -| **Timeout** | `nfs.rs` | 104-106 | `warn!("NFS health check timed out after 5s for {origin_id}")` | -| **Timeout** | `prefetch.rs` | 91 | `warn!("Prefetch event receive timed out after 1s")` | -| **Health** | `health.rs` | 209 | `warn!("Origin {origin_id} is degraded (failures: {count})")` | -| **Health** | `health.rs` | 217-220 | `warn!("Origin {origin_id} is now unhealthy after {n} consecutive failures")` | -| **Remote FS** | `smb.rs` | 118 | `warn!("SMB watch using inotify on {share_path} - may be unreliable")` | -| **Remote FS** | `nfs.rs` | 115 | `warn!("NFS watch using inotify on {mount_point} - may be unreliable")` | -| **Plugin** | `manager.rs` | 152 | `warn!("Failed to load plugin from {path}: {error}")` | -| **Plugin** | `manager.rs` | 193-194 | `warn!("Failed to unload plugin {plugin_id}: {error}")` | -| **Prefetch** | `prefetch.rs` | 97 | `warn!("Failed to record access pattern for {file_id}: {error}")` | -| **Prefetch** | `prefetch.rs` | 159-161 | `warn!("Prefetch skipped: concurrency limit reached ({max})")` | -| **Search** | `indexer.rs` | 49 | `warn!("Search indexer event receive error: {error}")` | -| **Search** | `indexer.rs` | 82 | `warn!("No metadata found for file {path}, skipping indexing")` | -| **Collections** | `collections.rs` | 146,180 | `warn!("Failed to save/delete collection {name}: {error}")` | - -### INFO Level (35+ locations) - Lifecycle & Major Operations - -| Category | File | Line | Log Message | -|----------|------|------|-------------| -| **Lifecycle** | `main.rs` | 118 | `info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting")` | -| **Lifecycle** | `filesystem.rs` | 94 | `info!("Mounting MusicFS at {:?}", mountpoint)` | -| **Lifecycle** | `filesystem.rs` | 154 | `info!("MusicFS initialized")` | -| **Lifecycle** | `filesystem.rs` | 159 | `info!("MusicFS destroyed")` | -| **Origin** | `registry.rs` | 28 | `info!("Registering origin {} with priority {}", id, priority)` | -| **Origin** | `registry.rs` | 36 | `info!("Unregistering origin {}", id)` | -| **Origin** | `watcher.rs` | 65 | `info!("Watching origin {} at {:?}", origin_id, path)` | -| **Config** | `main.rs` | 127 | `info!("Cache directory: {:?}", cache_dir)` | -| **Config** | `main.rs` | 141 | `info!("CAS store initialized")` | -| **Config** | `store.rs` | 51 | `info!("CAS store opened: {} chunks, {} bytes", count, size)` (ADD) | -| **Sync** | `main.rs` | 150,152 | `info!("Scanning music files...")` / `info!("Found {} music files", count)` | -| **Sync** | `delta.rs` | 104 | `info!("Delta complete: {} added, {} removed, {} modified", a, r, m)` | -| **Sync** | `delta.rs` | 63 | `info!("Sync started for origin {}", origin_id)` (ADD) | -| **Index** | `main.rs` | 160 | `info!("Virtual tree built")` | -| **Index** | `indexer.rs` | 62 | `info!("Indexer stopping")` | -| **Index** | `indexer.rs` | 114 | `info!("Indexed {} files", count)` | -| **Index** | `index.rs` | 170 | `info!("Search index committed")` | -| **Health** | `health.rs` | 202 | `info!("Origin {} is now healthy", id)` | -| **Health** | `health.rs` | 150 | `info!("Health monitor started with interval {:?}", interval)` (ADD) | -| **Plugin** | `manager.rs` | 127 | `info!("Initializing plugin system")` | -| **Plugin** | `manager.rs` | 150 | `info!("Loaded plugin '{}' with id {:?}", name, id)` | -| **Plugin** | `manager.rs` | 256 | `info!("Shutting down plugin system")` | -| **Cache** | `prefetch.rs` | 123 | `info!("Prefetch engine stopped")` | -| **Cache** | `prefetch.rs` | 174 | `info!("Prefetched {:?}: {} chunks, {} bytes", file_id, chunks, bytes)` | -| **Cache** | `eviction.rs` | 51 | `info!("Evicted {} bytes from cache", bytes)` | -| **Cache** | `prefetch.rs` | 73 | `info!("Prefetch engine started (lookahead: {}, max_concurrent: {})")` (ADD) | - -### DEBUG Level (60+ locations) - Operation Details - -| Category | File | Line | Log Message | -|----------|------|------|-------------| -| **FUSE lookup** | `filesystem.rs` | 162,195,200 | Entry + result/miss | -| **FUSE getattr** | `filesystem.rs` | 203,230,233 | Entry + result/miss | -| **FUSE readdir** | `filesystem.rs` | 237,263,303 | Entry + result/miss | -| **FUSE read** | `filesystem.rs` | 325,338,362,364 | Entry + file_id + result/error | -| **Local origin** | `local.rs` | 51,68 | readdir entry + result | -| **Local origin** | `local.rs` | 88-91,112 | read entry + result | -| **SMB origin** | `smb.rs` | 86,93 | readdir/read entry + result | -| **NFS origin** | `nfs.rs` | 81,89 | readdir/read entry + result | -| **Failover** | `failover.rs` | 66,82,87 | Entry + trying origin + success | -| **Tree lookup** | `tree.rs` | 124,132 | Entry + result | -| **Metadata cache** | `metadata.rs` | 36,40 | lookup + is_fresh entry/result | -| **CAS store** | `store.rs` | 70,101 | put/get entry | -| **File reader** | `reader.rs` | 66,86 | manifest cache + read entry | -| **Search** | `ops/search.rs` | 107,141,182 | readdir_query + readlink + execute_query | -| **Search index** | `index.rs` | 98,174 | index_file + search entry | -| **Fetcher** | `fetcher.rs` | 54,61,121 | fetch_file entry + meta + ensure_cached | - -**Key DEBUG fields**: `ino`, `parent`, `name`, `offset`, `size`, `bytes_read`, `origin_id`, `path`, `file_id`, `query`, `results_count`, `latency_ms` - -### TRACE Level (100+ locations) - Fine-Grained Flow - -| Category | File | Lines | What to Log | -|----------|------|-------|-------------| -| **Manifest cache** | `reader.rs` | 67-74 | Cache hit/miss decision | -| **Chunk iteration** | `reader.rs` | 107-127 | Each chunk: skip/read boundaries | -| **CAS dedup** | `store.rs` | 74-77 | Dedup hit decision | -| **CAS integrity** | `store.rs` | 121-134 | Verification result | -| **Tree lookup** | `tree.rs` | 118-129 | Path→inode + child lookup | -| **Tree parent** | `tree.rs` | 148-153 | Parent resolution path | -| **Prefetch event** | `prefetch.rs` | 91-120 | Event type match arms | -| **Prefetch semaphore** | `prefetch.rs` | 150-164 | In-flight check + acquire | -| **Delta scan** | `delta.rs` | 79-102 | Each file: cached/modified/unchanged/removed | -| **Delta entries** | `delta.rs` | 128-146 | Each entry: dir/audio/skip | -| **CDC chunking** | `cdc.rs` | 84-93 | Each chunk: offset/length/hash | -| **Failover origin** | `failover.rs` | 68-93 | Each origin attempt result | -| **Failover retry** | `failover.rs` | 107-122 | Each retry: attempt/success/delay | -| **Router select** | `router.rs` | 79-108 | Each candidate + selection reason | -| **FUSE node→attr** | `filesystem.rs` | 109-145 | Directory vs file conversion | -| **FUSE lookup** | `filesystem.rs` | 192-200 | Found/not found | -| **FUSE readdir** | `filesystem.rs` | 274-291 | Each child entry | -| **FUSE read** | `filesystem.rs` | 340-367 | file_id resolution + result | -| **Metadata tag** | `parser.rs` | 86-100 | Each tag extraction | -| **Health transition** | `health.rs` | 199-237 | State transition details | -| **Latency recording** | `router.rs` | 23-42 | Stats update per sample | - -**Key TRACE patterns**: -- Every `match` arm: `trace!("match arm: {variant}")` -- Every `if/else`: `trace!("branch: {condition}={value}")` -- Every loop iteration: `trace!("iteration {i}/{total}: ...")` -- Every cache lookup: `trace!("cache lookup key={key}, hit={hit}")` - ---- - -## gRPC Handler Instrumentation (ADDED - Oracle Review) - -**Gap identified**: 8/10 gRPC handlers had no logging. - -### server.rs - All Handlers - -| Handler | Line | Level | Log Message | -|---------|------|-------|-------------| -| `get_status()` | 209 | DEBUG | `debug!("gRPC get_status called")` | -| `get_cache_stats()` | 241 | DEBUG | `debug!("gRPC get_cache_stats called")` | -| `clear_cache()` | 278 | INFO | `info!("gRPC clear_cache: clearing {tier}")` | -| `prefetch()` | 296 | DEBUG | `debug!(file_count = paths.len(), "gRPC prefetch started")` | -| `list_origins()` | 322 | DEBUG | `debug!("gRPC list_origins called")` | -| `get_origin_health()` | 329 | DEBUG | `debug!(origin_id = %id, "gRPC get_origin_health")` | -| `rescan_origin()` | 337 | INFO | `info!(origin_id = %id, "gRPC rescan_origin started")` | -| `subscribe_events()` | 376 | INFO | `info!("gRPC subscribe_events: client connected")` | -| `shutdown()` | 402 | INFO | `info!(graceful = graceful, "gRPC shutdown requested")` | - -### search_service.rs - -| Handler | Line | Level | Log Message | -|---------|------|-------|-------------| -| `search()` | entry | DEBUG | `debug!(query = %q, limit = limit, "gRPC search")` | -| `search()` | result | DEBUG | `debug!(results = results.len(), "gRPC search completed")` | - -### Pattern: Use `#[instrument]` on all handlers - -```rust -#[tracing::instrument(level = "debug", skip(self, request), fields(method = "get_status"))] -async fn get_status(&self, request: Request<()>) -> Result, Status> { - // ... -} -``` - ---- - -## Async Task Spawn Instrumentation (ADDED - Oracle Review) - -**Gap identified**: 14 `tokio::spawn` sites need correlation IDs and span propagation. - -### Spawn Sites Requiring Instrumentation - -| File | Line | Task | Instrumentation | -|------|------|------|-----------------| -| `server.rs` | 305 | prefetch stream | `spawn(async { ... }.instrument(info_span!("prefetch_stream")))` | -| `server.rs` | 354 | rescan stream | `spawn(async { ... }.instrument(info_span!("rescan_stream", origin_id = %id)))` | -| `server.rs` | 384 | subscribe events | `spawn(async { ... }.instrument(info_span!("event_subscriber")))` | -| `search_service.rs` | spawn | search task | `spawn(async { ... }.instrument(debug_span!("search_task", query = %q)))` | -| `indexer.rs` | spawn | indexer loop | `spawn(async { ... }.instrument(info_span!("indexer")))` | -| `prefetch.rs` | 87 | prefetch engine | `spawn(async { ... }.instrument(info_span!("prefetch_engine")))` | -| `prefetch.rs` | 169 | prefetch file | `spawn(async { ... }.instrument(debug_span!("prefetch_file", file_id = ?id)))` | -| `health.rs` | 154 | health monitor | `spawn(async { ... }.instrument(info_span!("health_monitor")))` | -| `watcher.rs` | 34 | file watcher | `spawn(async { ... }.instrument(info_span!("file_watcher", origin_id = %id)))` | -| `artwork.rs` | spawn | image decode | `spawn_blocking(|| { ... })` - add span before spawn | - -### Pattern: Span Propagation - -```rust -use tracing::Instrument; - -// BEFORE (loses context) -tokio::spawn(async move { - do_work().await; -}); - -// AFTER (preserves correlation) -let span = tracing::info_span!("task_name", task_id = %id); -tokio::spawn(async move { - do_work().await; -}.instrument(span)); -``` - -### Add to init_logging() for request IDs - -```rust -// Generate request ID for correlation -use tracing::Span; -use uuid::Uuid; - -fn with_request_id(f: F) -> R -where F: FnOnce() -> R { - let request_id = Uuid::new_v4(); - let span = tracing::info_span!("request", request_id = %request_id); - span.in_scope(f) -} -``` - ---- - -## Database Operation Logging (ADDED - Oracle Review) - -**Gap identified**: Zero logging for rusqlite operations in db.rs, collections.rs, patterns.rs, artwork.rs. - -### db.rs - Core Database - -| Function | Line | Level | Log Message | -|----------|------|-------|-------------| -| `open()` | entry | INFO | `info!(path = ?path, "Opening metadata database")` | -| `open()` | success | INFO | `info!(file_count = count, "Database opened")` | -| `upsert_file()` | entry | DEBUG | `debug!(file_id = ?id, path = %path, "Upserting file")` | -| `upsert_file()` | error | ERROR | `error!(file_id = ?id, error = %e, "Failed to upsert file")` | -| `get_file_by_id()` | miss | TRACE | `trace!(file_id = ?id, "File not found in db")` | -| `delete_file()` | entry | DEBUG | `debug!(file_id = ?id, "Deleting file from db")` | -| `list_files_by_origin()` | result | DEBUG | `debug!(origin_id = %id, count = files.len(), "Listed files")` | - -### collections.rs - -| Function | Line | Level | Log Message | -|----------|------|-------|-------------| -| `create()` | entry | INFO | `info!(name = %name, "Creating collection")` | -| `save()` | error | WARN | `warn!(name = %name, error = %e, "Failed to save collection")` | -| `delete()` | entry | INFO | `info!(name = %name, "Deleting collection")` | -| `list()` | result | DEBUG | `debug!(count = collections.len(), "Listed collections")` | - -### patterns.rs - Access Patterns - -| Function | Line | Level | Log Message | -|----------|------|-------|-------------| -| `record_access()` | entry | TRACE | `trace!(file_id = ?id, "Recording access pattern")` | -| `predict_next()` | result | DEBUG | `debug!(predictions = preds.len(), "Predicted next files")` | - -### artwork.rs - -| Function | Line | Level | Log Message | -|----------|------|-------|-------------| -| `store()` | entry | DEBUG | `debug!(file_id = ?id, size_bytes = data.len(), "Storing artwork")` | -| `get()` | hit/miss | TRACE | `trace!(file_id = ?id, found = found, "Artwork lookup")` | - -### Pattern: Database Error Wrapper - -```rust -// Add to musicfs-cache/src/db.rs -fn log_db_result(op: &str, result: Result) -> Result { - match result { - Ok(v) => { - tracing::trace!(op = op, "db operation succeeded"); - Ok(v) - } - Err(e) => { - tracing::error!(op = op, error = %e, "db operation failed"); - Err(Error::Database(e.to_string())) - } - } -} -``` - ---- - -## Channel Operation Logging (ADDED - Oracle Review) - -**Gap identified**: No logging for channel capacity, close, or broadcast lag. - -### Channel Locations - -| File | Type | Log Points | -|------|------|------------| -| `events.rs` | broadcast | Lag warning when receiver falls behind | -| `watcher.rs` | mpsc | Channel close on watcher shutdown | -| `server.rs` | mpsc | gRPC stream channel capacity | -| `indexer.rs` | mpsc | Event queue depth | -| `health.rs` | mpsc | Health check channel | - -### Patterns - -```rust -// Broadcast lag detection (events.rs) -match rx.recv().await { - Ok(event) => { /* handle */ } - Err(broadcast::error::RecvError::Lagged(n)) => { - tracing::warn!(skipped = n, "Event subscriber lagged, skipped events"); - } - Err(broadcast::error::RecvError::Closed) => { - tracing::debug!("Event channel closed"); - break; - } -} - -// Channel capacity warning (before send) -if tx.capacity() < 10 { - tracing::warn!(remaining = tx.capacity(), "Channel near capacity"); -} - -// Channel close -impl Drop for EventBus { - fn drop(&mut self) { - tracing::debug!("Event bus shutting down"); - } -} -``` - ---- - -## Drop Implementation Logging (ADDED - Oracle Review) - -**Gap identified**: No logging in Drop impls for cleanup verification. - -| File | Type | Log Message | -|------|------|-------------| -| `manager.rs:276` | `PluginManager` | `debug!("PluginManager dropping, unloading {} plugins", self.plugins.len())` | -| `watcher.rs:157` | `WatchHandle` | `trace!(origin_id = %self.origin_id, "WatchHandle dropped")` | -| `prefetch.rs` | `PrefetchEngine` | `debug!("PrefetchEngine dropping, {} in-flight", self.in_flight.len())` | -| `server.rs` | gRPC server | `info!("gRPC server shutting down")` | - -### Pattern - -```rust -impl Drop for PluginManager { - fn drop(&mut self) { - tracing::debug!( - plugin_count = self.plugins.len(), - "PluginManager dropping" - ); - // existing cleanup... - } -} -``` - ---- - -## Credential Loading (ADDED - Oracle Review) - -**Gap identified**: No logging in credentials.rs::load(). - -| Function | Level | Log Message | -|----------|-------|-------------| -| `load()` entry | DEBUG | `debug!(origin_id = %origin_id, "Loading credentials")` | -| `load()` cache hit | TRACE | `trace!(origin_id = %origin_id, "Credential cache hit")` | -| `load()` success | INFO | `info!(origin_id = %origin_id, cred_type = %cred.type_name(), "Credential loaded")` | -| `load()` not found | DEBUG | `debug!(origin_id = %origin_id, "No credential found")` | -| `load()` error | WARN | `warn!(origin_id = %origin_id, error = %e, "Credential load failed")` | - -**SECURITY**: Never log credential values. The existing Debug impl with redaction is correct. - ---- - -## Security Considerations (ADDED - Oracle Review) - -### Never Log These - -| Data | Location | Mitigation | -|------|----------|------------| -| `WebhookConfig.secret` | webhook.rs | Add `#[serde(skip_serializing)]`, use custom Debug | -| Credential values | credentials.rs | Already redacted in Debug impl ✓ | -| Full file paths with usernames | everywhere | Sanitize `/home/{user}/` → `~/` | -| API keys/tokens | config.rs | Mark sensitive fields | - -### Sanitization Helper - -```rust -// Add to musicfs-core/src/lib.rs -pub fn sanitize_path(path: &Path) -> String { - if let Ok(home) = std::env::var("HOME") { - path.to_string_lossy() - .replace(&home, "~") - .to_string() - } else { - path.to_string_lossy().to_string() - } -} - -// Usage -debug!(path = %sanitize_path(&path), "Reading file"); -``` - -### WebhookConfig Fix - -```rust -// webhook.rs - add custom Debug -#[derive(Clone, Serialize, Deserialize)] -pub struct WebhookConfig { - pub url: String, - #[serde(skip_serializing)] - pub secret: Option, // Never serialize - // ... -} - -impl std::fmt::Debug for WebhookConfig { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("WebhookConfig") - .field("url", &self.url) - .field("secret", &self.secret.as_ref().map(|_| "[REDACTED]")) - .finish() - } -} -``` - ---- - -## Performance Considerations (ADDED - Oracle Review) - -### Hot Path Warnings - -| Path | Risk | Mitigation | -|------|------|------------| -| `reader.rs` chunk loop | 100s of TRACE logs per seek | Log summary only: `trace!(chunks_read = n, "Read complete")` | -| `store.rs` put/get | 1000s during sync | Keep at DEBUG, not TRACE | -| `delta.rs` file scan | Log per file during full scan | Use TRACE, batch summaries at DEBUG | -| `parser.rs` tag extraction | Many TRACE per file | Sample: log every 100th file | - -### Trace Sampling Config - -```rust -// Add to LoggingConfig -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct LoggingConfig { - // ... existing fields ... - - /// Sample rate for TRACE logs in hot paths (0.0-1.0, default 1.0) - #[serde(default = "default_sample_rate")] - pub trace_sample_rate: f32, -} - -fn default_sample_rate() -> f32 { 1.0 } - -// Usage in hot paths -if rand::random::() < config.trace_sample_rate { - trace!(...); -} -``` - -### Rate-Limited Warnings - -```rust -// For repeating warnings during outages (failover.rs) -use std::sync::atomic::{AtomicU64, Ordering}; -use std::time::{Duration, Instant}; - -static LAST_FAILOVER_WARN: AtomicU64 = AtomicU64::new(0); - -fn warn_rate_limited(origin_id: &str, error: &str) { - let now = Instant::now().elapsed().as_secs(); - let last = LAST_FAILOVER_WARN.load(Ordering::Relaxed); - if now - last >= 60 { // Max once per minute - LAST_FAILOVER_WARN.store(now, Ordering::Relaxed); - warn!(origin_id = %origin_id, error = %error, "Origin failover"); - } -} -``` - ---- - -## Standardized Field Names (ADDED - Oracle Review) - -Use these consistently across all log statements: - -| Field | Type | Usage | -|-------|------|-------| -| `origin_id` | String | Origin identifier (not `origin`) | -| `file_id` | FileId | File identifier | -| `path` | String | Virtual or real path (sanitized) | -| `size_bytes` | u64 | Size in bytes (not `size`, `bytes`, `len`) | -| `offset` | u64 | Read offset | -| `duration_ms` | u64 | Operation duration in milliseconds | -| `count` | usize | Generic count | -| `attempt` | u32 | Retry attempt number | -| `max_attempts` | u32 | Maximum retry attempts | -| `error` | impl Display | Error message (not `err`, `e`) | -| `request_id` | Uuid | Correlation ID for requests | - ---- - -## Instrumentation Patterns (ADDED - Oracle Review) - -### Use `#[instrument(err)]` for Automatic Error Logging - -```rust -// BEFORE: Manual error logging -pub async fn read(&self, path: &Path) -> Result { - match self.inner_read(path).await { - Ok(data) => Ok(data), - Err(e) => { - error!(path = ?path, error = %e, "Read failed"); - Err(e) - } - } -} - -// AFTER: Automatic with #[instrument] -#[tracing::instrument(level = "debug", skip(self), err)] -pub async fn read(&self, path: &Path) -> Result { - self.inner_read(path).await -} -``` - -### Span Events vs Regular Logs - -```rust -// Regular log - standalone event -info!("Operation completed"); - -// Span event - attached to current span context -tracing::Span::current().record("result", "success"); - -// Prefer span events for operation outcomes -#[instrument(fields(result))] -async fn operation() -> Result<()> { - // ... work ... - Span::current().record("result", "success"); - Ok(()) -} -``` - ---- - -## Fixes: Incorrect Line References (ADDED - Oracle Review) - -| File | Issue | Fix | -|------|-------|-----| -| `webhook.rs:43` | Uses `expect()` (panics) | Replace with `?` + error log | -| `webhook.rs:133` | Uses `expect()` (panics) | Replace with `?` + error log | - -```rust -// webhook.rs - BEFORE -let client = reqwest::Client::builder() - .timeout(Duration::from_secs(30)) - .build() - .expect("Failed to create HTTP client"); - -// webhook.rs - AFTER -let client = reqwest::Client::builder() - .timeout(Duration::from_secs(30)) - .build() - .map_err(|e| { - error!(error = %e, "Failed to create webhook HTTP client"); - WebhookError::ClientInit(e.to_string()) - })?; -``` - ---- - -## Log Levels Guide - -| Level | Use Case | Example | -|-------|----------|---------| -| `ERROR` | Unrecoverable failures | Mount failed, DB corruption | -| `WARN` | Recoverable issues | Origin timeout, retry needed | -| `INFO` | Lifecycle events | Service start/stop, health change | -| `DEBUG` | Operation details | Function entry, request params | -| `TRACE` | Fine-grained flow | Match arms, cache hit/miss | - ---- - -## Testing Checklist - -### Basic Functionality -- [ ] Log files created in configured directory -- [ ] Daily rotation creates new files at midnight -- [ ] JSON output parseable by `jq` -- [ ] `journalctl -t musicfs` shows logs -- [ ] `RUST_LOG=musicfs=trace` enables trace output -- [ ] WorkerGuard kept alive (logs flush on shutdown) -- [ ] Logrotate compresses old files - -### Correlation & Context (NEW) -- [ ] Request IDs propagate through async tasks -- [ ] Spawned task logs include parent span context -- [ ] gRPC handler logs show method name in span - -### Security (NEW) -- [ ] WebhookConfig.secret never appears in logs -- [ ] Credential values never appear in logs -- [ ] File paths with `/home/{user}` show as `~/` - -### Performance (NEW) -- [ ] TRACE sampling respects `trace_sample_rate` config -- [ ] Hot path chunk loops log summary, not per-chunk -- [ ] Origin failover warnings are rate-limited (1/minute) -- [ ] Database operations log without blocking - -### Database & Channels (NEW) -- [ ] Database open logs file count -- [ ] Channel capacity warnings appear when queue fills -- [ ] Broadcast lag warnings appear when subscriber falls behind -- [ ] Drop implementations log cleanup - ---- - -## Summary - -| Phase | Effort | Deliverables | -|-------|--------|--------------| -| 1. Config & Dependencies | 2h | LoggingConfig, init_logging(), logrotate, trace sampling | -| 2. Core instrumentation | 1h | tracing in musicfs-core, credentials, sanitization | -| 3. Hot path instrumentation | 4h | #[instrument] + trace! across 5 crates | -| 4. gRPC & async tasks | 2h | Handler instrumentation, spawn correlation | -| 5. Database & channels | 2h | rusqlite logging, channel capacity/close | -| 6. Production files | 1h | Updated systemd, example config | -| **Total** | **12h** | Full observability | - ---- - -## Files to Modify - -### Phase 1: Config & Dependencies -| File | Changes | -|------|---------| -| `Cargo.toml` (workspace) | Add tracing-appender, tracing-journald | -| `crates/musicfs-cli/Cargo.toml` | Add dependencies | -| `crates/musicfs-core/Cargo.toml` | Add tracing | -| `crates/musicfs-core/src/config.rs` | Add LoggingConfig with trace_sample_rate | -| `crates/musicfs-cli/src/main.rs` | Expand init_logging(), request ID helper | -| `crates/musicfs-core/src/lib.rs` | Add sanitize_path() helper | - -### Phase 2: Core Instrumentation -| File | Changes | -|------|---------| -| `crates/musicfs-core/src/credentials.rs` | Add load() logging (redacted) | -| `crates/musicfs-core/src/events.rs` | Add broadcast lag detection | - -### Phase 3: Hot Path Instrumentation -| File | Changes | -|------|---------| -| `crates/musicfs-fuse/src/filesystem.rs` | Add #[instrument], trace! | -| `crates/musicfs-origins/src/failover.rs` | Add #[instrument], trace!, rate-limited warn | -| `crates/musicfs-origins/src/health.rs` | Add state transition logging | -| `crates/musicfs-origins/src/router.rs` | Add selection logging | -| `crates/musicfs-cache/src/tree.rs` | Add mutation logging | -| `crates/musicfs-cache/src/metadata.rs` | Add hit/miss logging | -| `crates/musicfs-cas/src/reader.rs` | Add chunk assembly logging (summary, not per-chunk) | -| `crates/musicfs-cas/src/store.rs` | Add dedup logging | -| `crates/musicfs-sync/src/delta.rs` | Add change detection logging | - -### Phase 4: gRPC & Async Tasks (NEW) -| File | Changes | -|------|---------| -| `crates/musicfs-grpc/src/server.rs` | Add #[instrument] to all 10 handlers, spawn correlation | -| `crates/musicfs-grpc/src/search_service.rs` | Add #[instrument], spawn instrumentation | -| `crates/musicfs-grpc/src/webhook.rs` | Fix expect() → error!, custom Debug for secret | -| `crates/musicfs-cache/src/prefetch.rs` | Add spawn instrumentation, Drop logging | -| `crates/musicfs-search/src/indexer.rs` | Add spawn instrumentation | -| `crates/musicfs-sync/src/watcher.rs` | Add spawn instrumentation, Drop logging | -| `crates/musicfs-plugins/src/manager.rs` | Add Drop logging | - -### Phase 5: Database & Channels (NEW) -| File | Changes | -|------|---------| -| `crates/musicfs-cache/src/db.rs` | Add log_db_result() helper, open/upsert/query logging | -| `crates/musicfs-search/src/collections.rs` | Add CRUD operation logging | -| `crates/musicfs-cache/src/patterns.rs` | Add access pattern logging | -| `crates/musicfs-cache/src/artwork.rs` | Add store/get logging | - -### Phase 6: Production Files -| File | Changes | -|------|---------| -| `dist/musicfs.service` | Add logging directives | -| `dist/logrotate.d/musicfs` | New file | -| `dist/config.example.toml` | Add logging section with trace_sample_rate | diff --git a/docs/v2/plans/persistent-state-impl.md b/docs/v2/plans/persistent-state-impl.md deleted file mode 100644 index 3fc61bb..0000000 --- a/docs/v2/plans/persistent-state-impl.md +++ /dev/null @@ -1,796 +0,0 @@ -# Persistent State: Implementation Plan - -**Authors:** AI-assisted -**Status:** Draft -**Last Updated:** 2026-05-13 -**Reviewers:** TBD -**Approvers:** TBD -**Prerequisites:** [persistent-state.md](persistent-state.md) (research), [phase-a-stop-dying.md](phase-a-stop-dying.md) (signal handling + shutdown) -**Estimated Effort:** ~8 days - ---- - -[TOC] - ---- - -## 1. Abstract - -Wire up the existing SQLite persistence layer into the mount path so that subsequent mounts load from database instead of rescanning origins. This transforms mount time from O(N × origin_latency) to O(N × SQLite_read) — roughly 1000x faster for remote origins. - -**Storage decision: SQLite (Option A).** Rationale: -- `Database` struct with full CRUD already exists in `musicfs-cache/src/db.rs` -- Schema with `chunk_manifest BLOB` column already exists in `schema.sql` -- `ChunkManifest::from_db()` and `chunks_to_bytes()` already exist but are never called -- Row-to-`FileMeta` mapping already exists in `get_file_by_virtual_path()` -- WAL mode crash safety already configured -- 2-4 second bulk load for 1M rows is acceptable (target is <5s, not <500ms — the <500ms target is for the mount syscall itself, which returns immediately with lazy tree loading) - -No new storage engine. No new dependencies. Wire existing code. - ---- - -## 2. Background - -### 2.1 Current State - -`run_mount()` in `main.rs`: -1. Opens CAS store ✅ -2. Creates origin connection ✅ -3. `scan_music_files()` — walks entire origin, parses every file with symphonia ❌ **BOTTLENECK** -4. Builds VirtualTree from scan results (in-memory only) ❌ **LOST ON RESTART** -5. Registers every file in ContentFetcher (in-memory only) ❌ **LOST ON RESTART** -6. Mounts FUSE ✅ - -### 2.2 What Exists But Is Not Wired - -| Component | Exists | Wired Into Mount? | -|-----------|--------|--------------------| -| `Database::open()` + schema + WAL | ✅ | ❌ | -| `Database::upsert_file()` | ✅ | ❌ | -| `Database::get_file_by_virtual_path()` (returns `FileMeta`) | ✅ | ❌ | -| `schema.sql` with `chunk_manifest BLOB` column | ✅ | ❌ | -| `ChunkManifest::chunks_to_bytes()` (serialize) | ✅ | ❌ | -| `ChunkManifest::from_db()` (deserialize) | ✅ | ❌ | -| `TreeBuilder::add_file(&FileMeta)` | ✅ | ✅ (from scan, not from DB) | -| `ContentFetcher::register_file(FileMeta)` | ✅ | ✅ (from scan, not from DB) | -| `PatternStore::new(db_path)` (loads from SQLite on open) | ✅ | ❌ | -| `CollectionStore::new(db_path)` | ✅ | ❌ | -| `SearchIndex::open(path)` (opens tantivy from disk) | ✅ | ❌ | - -### 2.3 What's Missing - -| Component | Needs Building | -|-----------|----------------| -| `Database::list_all_files()` → `Vec` | New method (SQL exists, just needs `SELECT *`) | -| `Database::update_manifest(FileId, &[u8])` | New method (column exists) | -| `Database::get_manifest(FileId)` → `Option>` | New method | -| `Database::list_all_manifests()` → `Vec<(FileId, ChunkManifest)>` | New method | -| Background delta sync task | New (compare DB state vs origin) | -| First-mount detection | New (check `file_count() > 0`) | - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -- Subsequent mount loads tree from SQLite, not origin scan -- Chunk manifests persist to SQLite, loaded on mount (no re-download) -- tantivy index, PatternStore, CollectionStore opened on mount -- Background delta sync reconciles DB vs origin after mount -- First mount (empty DB) falls back to current full-scan behavior -- Mount time for 10K files: <1 second (subsequent mount) -- All existing tests pass, no regressions - -### 3.2 Non-Goals - -- Achieving <500ms mount for 1M+ files (requires lazy tree loading — future work) -- LRU eviction persistence (separate task, low urgency) -- Changing the storage engine (SQLite is the decision) -- Config file parsing changes (origin config stays in TOML, not DB) -- Schema migrations for existing data (fresh DB on first mount) - ---- - -## 4. Proposed Design - -### 4.1 Implementation Order - -``` -4.2 Database: list_all_files() + manifest CRUD (foundation) - ↓ -4.3 Mount path: load tree + fetcher from DB (core change) - ↓ -4.4 Persist manifests after fetch (write path) - ↓ -4.5 Open tantivy + PatternStore + CollectionStore (quick wiring) - ↓ -4.6 Background delta sync (post-mount reconciliation) - ↓ -4.7 First-mount detection + fallback (edge case) - ↓ -4.8 Shutdown: WAL checkpoint + flush (cleanup) -``` - -### 4.2 Database: New Methods - -**File**: `musicfs-cache/src/db.rs` - -#### list_all_files() - -Bulk load all files from DB. Reuses the existing row-to-FileMeta mapping from `get_file_by_virtual_path()`. - -```rust -pub fn list_all_files(&self) -> Result> { - let conn = self.conn.lock().unwrap(); - - let mut stmt = conn.prepare( - r#"SELECT id, origin_id, real_path, virtual_path, - title, artist, album, album_artist, genre, - year, track, disc, - duration_ms, bitrate, sample_rate, format, - origin_mtime, origin_size, content_hash - FROM files - ORDER BY virtual_path"# - ).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?; - - let files = stmt.query_map([], |row| { - // Same mapping as get_file_by_virtual_path - Ok(Self::row_to_file_meta(row)) - }) - .map_err(|e| Error::Database(format!("query failed: {}", e)))? - .filter_map(|r| r.ok()) - .collect(); - - Ok(files) -} -``` - -Extract the row mapping into a shared `row_to_file_meta(row)` helper to avoid duplication with `get_file_by_virtual_path()`. - -#### Manifest CRUD - -```rust -pub fn update_manifest(&self, file_id: FileId, manifest_blob: &[u8]) -> Result<()> { - let conn = self.conn.lock().unwrap(); - conn.execute( - "UPDATE files SET chunk_manifest = ?1 WHERE id = ?2", - params![manifest_blob, file_id.0], - ).map_err(|e| Error::Database(format!("update manifest failed: {}", e)))?; - Ok(()) -} - -pub fn get_manifest(&self, file_id: FileId) -> Result>> { - let conn = self.conn.lock().unwrap(); - conn.query_row( - "SELECT chunk_manifest FROM files WHERE id = ?1", - params![file_id.0], - |row| row.get(0), - ) - .optional() - .map_err(|e| Error::Database(format!("get manifest failed: {}", e))) -} - -pub fn list_all_manifests(&self) -> Result)>> { - let conn = self.conn.lock().unwrap(); - let mut stmt = conn.prepare( - "SELECT id, origin_size, origin_mtime, chunk_manifest FROM files WHERE chunk_manifest IS NOT NULL" - ).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?; - - let manifests = stmt.query_map([], |row| { - Ok(( - FileId(row.get(0)?), - row.get::<_, i64>(1)? as u64, - row.get::<_, i64>(2)?, - row.get::<_, Vec>(3)?, - )) - }) - .map_err(|e| Error::Database(format!("query failed: {}", e)))? - .filter_map(|r| r.ok()) - .collect(); - - Ok(manifests) -} -``` - -#### WAL Checkpoint - -```rust -pub fn checkpoint(&self) -> Result<()> { - let conn = self.conn.lock().unwrap(); - conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE)") - .map_err(|e| Error::Database(format!("WAL checkpoint failed: {}", e)))?; - info!("SQLite WAL checkpoint completed"); - Ok(()) -} -``` - -#### Tests - -```rust -#[test] -fn test_list_all_files() { - let db = Database::open_memory().unwrap(); - // Insert 3 files - // list_all_files() returns 3 - // Verify FileMeta fields match what was inserted -} - -#[test] -fn test_manifest_roundtrip() { - let db = Database::open_memory().unwrap(); - // Insert file, update_manifest with blob, get_manifest returns same blob -} - -#[test] -fn test_list_all_manifests_skips_null() { - let db = Database::open_memory().unwrap(); - // Insert 3 files, only 1 with manifest - // list_all_manifests() returns 1 -} -``` - ---- - -### 4.3 Mount Path: Load From DB - -**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()` - -The key change: replace `scan_music_files()` with DB load when data exists. - -```rust -fn run_mount(mountpoint: PathBuf, origin_path: Option, cache_dir: Option) -> Result<()> { - let origin_path = origin_path.context("--origin is required")?; - let runtime = tokio::runtime::Runtime::new()?; - let handle = runtime.handle().clone(); - - let (tree, reader, db) = runtime.block_on(async { - let cache_dir = resolve_cache_dir(cache_dir); - std::fs::create_dir_all(&cache_dir)?; - std::fs::create_dir_all(&mountpoint)?; - - // Open CAS store - let store = Arc::new(CasStore::open(CasConfig { - chunks_dir: cache_dir.join("chunks"), - ..Default::default() - }).await?); - - // Open database - let db_path = cache_dir.join("metadata.db"); - let db = Arc::new(Database::open_with_integrity_check(&db_path) - .or_else(|_| Database::open(&db_path))?); // Fallback to normal open if integrity check fails - - let fetcher = Arc::new(ContentFetcher::new(store.clone())); - let origin_id = OriginId::from("local"); - let origin = Arc::new(LocalOrigin::new(origin_id.clone(), origin_path.clone())); - fetcher.register_origin(origin); - - // Decide: load from DB or full scan - let file_count = db.file_count().unwrap_or(0); - - let files = if file_count > 0 { - // SUBSEQUENT MOUNT — load from DB - info!(file_count, "Loading metadata from database"); - let start = Instant::now(); - let files = db.list_all_files()?; - info!(elapsed_ms = start.elapsed().as_millis() as u64, "Database load complete"); - files - } else { - // FIRST MOUNT — full origin scan - info!("First mount: scanning origin"); - let files = scan_music_files(&origin_path, &origin_id).await?; - info!(file_count = files.len(), "Scan complete, persisting to database"); - - // Persist to DB for next mount - for file in &files { - if let Some(ref audio) = file.audio { - db.upsert_file( - &file.real_path.origin_id, - &file.real_path.path, - &file.virtual_path, - audio, - file.mtime, - file.size, - )?; - } - } - info!("Metadata persisted to database"); - files - }; - - // Build tree + register files (same as before, but from DB or scan) - let mut builder = TreeBuilder::new(); - for file in &files { - builder.add_file(file); - fetcher.register_file(file.clone()); - } - let tree = Arc::new(RwLock::new(builder.build())); - - // Load manifests from DB - let reader = Arc::new(FileReader::with_fetcher(store, fetcher)); - let manifest_count = load_manifests_from_db(&db, &reader)?; - if manifest_count > 0 { - info!(manifest_count, "Loaded chunk manifests from database"); - } - - Ok::<_, anyhow::Error>((tree, reader, db)) - })?; - - // Open search index - let search_dir = cache_dir.join("search.idx"); - let _search_index = SearchIndex::open_with_recovery(&search_dir) - .context("Failed to open search index")?; - - // Open pattern store - let patterns_path = cache_dir.join("patterns.db"); - let _pattern_store = PatternStore::new(&patterns_path, 30) - .context("Failed to open pattern store")?; - - // ... mount, signal handler, shutdown (same as current) ... - - // On shutdown: checkpoint WAL - db.checkpoint().unwrap_or_else(|e| warn!("WAL checkpoint failed: {}", e)); -} -``` - -Helper function: - -```rust -fn load_manifests_from_db(db: &Database, reader: &FileReader) -> Result { - let manifests = db.list_all_manifests()?; - let mut count = 0; - for (file_id, total_size, mtime, blob) in manifests { - if let Some(manifest) = ChunkManifest::from_db(file_id, total_size, mtime, &blob) { - reader.register_manifest(manifest); - count += 1; - } - } - Ok(count) -} -``` - ---- - -### 4.4 Persist Manifests After Fetch - -**File**: `musicfs-cas/src/fetcher.rs` - -After `fetch_file()` downloads and chunks a file, persist the manifest to SQLite. - -The fetcher currently doesn't have access to the Database. Two options: -1. Pass `Arc` to ContentFetcher (adds dependency musicfs-cas → musicfs-cache) -2. Emit an event with the manifest, have the caller persist it - -**Approach**: Option 2 — use the existing EventBus. Add a new event variant: - -**File**: `musicfs-core/src/events.rs` - -```rust -pub enum Event { - // ... existing variants - ManifestCached { - file_id: FileId, - manifest_blob: Vec, - }, -} -``` - -**File**: `musicfs-cas/src/fetcher.rs` — emit event after fetch: - -```rust -pub async fn fetch_file(&self, file_id: FileId) -> Result { - // ... existing fetch + chunk logic ... - - // Emit manifest for persistence - if let Some(bus) = &self.event_bus { - bus.publish(Event::ManifestCached { - file_id, - manifest_blob: manifest.chunks_to_bytes(), - }); - } - - Ok(manifest) -} -``` - -**File**: `musicfs-cli/src/main.rs` — subscribe to ManifestCached events: - -```rust -// Spawn manifest persistence listener -let db_for_manifests = db.clone(); -let mut manifest_rx = event_bus.subscribe(); -tokio::spawn(async move { - while let Ok(event) = manifest_rx.recv().await { - if let Event::ManifestCached { file_id, manifest_blob } = event { - if let Err(e) = db_for_manifests.update_manifest(file_id, &manifest_blob) { - warn!(file_id = ?file_id, error = %e, "Failed to persist manifest"); - } - } - } -}); -``` - ---- - -### 4.5 Open tantivy + PatternStore + CollectionStore - -These already have `open()` methods that load from disk. Just call them in the mount path. - -**File**: `musicfs-cli/src/main.rs` - -```rust -// After tree is built, before FUSE mount - -// Search index -let search_dir = cache_dir.join("search.idx"); -let search_index = Arc::new( - SearchIndex::open_with_recovery(&search_dir) - .unwrap_or_else(|e| { - warn!("Search index failed, creating fresh: {}", e); - SearchIndex::open(&search_dir).expect("Failed to create search index") - }) -); - -// Pattern store (already persists to SQLite, loads sequence_counts on open) -let patterns_path = cache_dir.join("patterns.db"); -let pattern_store = Arc::new( - PatternStore::new(&patterns_path, 30) - .unwrap_or_else(|e| { - warn!("Pattern store failed: {}", e); - PatternStore::new(&patterns_path, 30).expect("Failed to create pattern store") - }) -); - -// Collection store -let collections_path = cache_dir.join("collections.db"); -let collection_store = Arc::new( - CollectionStore::new(&collections_path) - .unwrap_or_else(|e| { - warn!("Collection store failed: {}", e); - CollectionStore::new(&collections_path).expect("Failed to create collection store") - }) -); -``` - -For tantivy: if this is a first mount, index all files after scan: - -```rust -if file_count == 0 { - // First mount — index all files - info!("First mount: building search index"); - let indexer = Indexer::new(search_index.clone(), event_bus.clone(), /* metadata_lookup */); - indexer.index_batch(&files)?; -} -``` - ---- - -### 4.6 Background Delta Sync - -After mount completes, spawn a background task that compares DB state against origin and reconciles differences. - -**File**: `musicfs-sync/src/delta.rs` or new `musicfs-cli/src/sync.rs` - -```rust -pub async fn background_delta_sync( - origin: Arc, - origin_id: OriginId, - db: Arc, - tree: Arc>, - fetcher: Arc, - event_bus: Arc, -) -> Result { - info!("Starting background delta sync"); - let start = Instant::now(); - - let mut added = 0u64; - let mut modified = 0u64; - let mut removed = 0u64; - let mut unchanged = 0u64; - - // Get all files currently in DB - let db_files: HashMap = db.list_all_files()? - .into_iter() - .map(|f| (f.real_path.path.clone(), f)) - .collect(); - - // Walk origin - let origin_files = scan_origin_recursive(&origin, Path::new("/")).await?; - - // Compare - for (path, origin_stat) in &origin_files { - match db_files.get(path) { - Some(db_file) if db_file.mtime == origin_stat.mtime && db_file.size == origin_stat.size => { - unchanged += 1; - } - Some(db_file) => { - // Modified — re-parse metadata, update DB, update tree - modified += 1; - // ... update logic ... - } - None => { - // New file — parse metadata, add to DB + tree - added += 1; - // ... add logic ... - } - } - } - - // Find removed files (in DB but not on origin) - let origin_paths: HashSet<_> = origin_files.keys().collect(); - for (path, db_file) in &db_files { - if !origin_paths.contains(path) { - removed += 1; - db.delete_file(db_file.id)?; - tree.write().remove_file(&db_file.virtual_path); - } - } - - let elapsed = start.elapsed(); - info!( - added, modified, removed, unchanged, - elapsed_ms = elapsed.as_millis() as u64, - "Delta sync complete" - ); - - Ok(SyncSummary { added, modified, removed, unchanged }) -} -``` - -Spawn in `run_mount()` after FUSE mount: - -```rust -// Background delta sync (non-blocking) -let sync_db = db.clone(); -let sync_tree = tree.clone(); -let sync_fetcher = fetcher.clone(); -let sync_origin = origin.clone(); -let sync_origin_id = origin_id.clone(); -let sync_bus = event_bus.clone(); -tokio::spawn(async move { - if let Err(e) = background_delta_sync( - sync_origin, sync_origin_id, sync_db, sync_tree, sync_fetcher, sync_bus, - ).await { - warn!("Delta sync failed: {}", e); - } -}); -``` - ---- - -### 4.7 First-Mount Detection - -Simple: check `db.file_count()`: - -```rust -let file_count = db.file_count().unwrap_or(0); - -if file_count > 0 { - // Load from DB -} else { - // Full scan + persist -} -``` - -This is already shown in Section 4.3. No separate implementation step. - ---- - -### 4.8 Shutdown: WAL Checkpoint + Flush - -**File**: `musicfs-cli/src/main.rs` — in the shutdown sequence (after signal, before dropping session): - -```rust -info!("Beginning ordered shutdown"); -shutdown_token.cancel(); -tokio::time::sleep(Duration::from_millis(500)).await; - -// Flush persistence -if let Err(e) = db.checkpoint() { - warn!("SQLite WAL checkpoint failed: {}", e); -} -info!("Background tasks stopped, state flushed"); -``` - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -- No new attack surface — SQLite file has same permissions as cache directory -- Metadata in DB is the same as what's already in the FUSE virtual tree (not new data) -- `chunk_manifest` BLOB is binary chunk hashes — not sensitive - -### 5.2 Observability - -- Mount time logged: "Loading metadata from database" with elapsed_ms -- First-mount detected and logged: "First mount: scanning origin" -- Delta sync summary logged: added/modified/removed/unchanged counts + elapsed -- WAL checkpoint logged on shutdown -- Manifest persistence failures logged at WARN (non-fatal) - -### 5.3 Scalability - -| Library Size | First Mount (scan) | Subsequent Mount (DB load) | -|---|---|---| -| 1K files | ~1-2s | <100ms | -| 10K files | ~10-20s | ~200ms | -| 100K files | ~2-5 min | ~1-2s | -| 1M files | ~20-60 min | ~2-4s | - -Delta sync runs in background — mount returns immediately, user sees stale-but-functional data while sync catches up. - -### 5.4 Testing - -```rust -// Test: subsequent mount loads from DB -#[tokio::test] -async fn test_mount_loads_from_db() { - let dir = TempDir::new().unwrap(); - let db = Database::open(dir.path().join("test.db")).unwrap(); - - // Insert files - for i in 0..100 { - db.upsert_file(/* ... */).unwrap(); - } - - // Load all - let files = db.list_all_files().unwrap(); - assert_eq!(files.len(), 100); - - // Build tree from DB files (same as mount path) - let mut builder = TreeBuilder::new(); - for f in &files { builder.add_file(f); } - let tree = builder.build(); - assert_eq!(tree.file_count(), 100); -} - -// Test: manifest roundtrip through DB -#[tokio::test] -async fn test_manifest_persists_and_loads() { - let dir = TempDir::new().unwrap(); - let db = Database::open(dir.path().join("test.db")).unwrap(); - - let id = db.upsert_file(/* ... */).unwrap(); - - let manifest = ChunkManifest { /* ... */ }; - let blob = manifest.chunks_to_bytes(); - db.update_manifest(id, &blob).unwrap(); - - let loaded = db.get_manifest(id).unwrap().unwrap(); - let restored = ChunkManifest::from_db(id, 1000, 0, &loaded).unwrap(); - assert_eq!(restored.chunks.len(), manifest.chunks.len()); -} - -// Test: first mount detects empty DB -#[tokio::test] -async fn test_first_mount_detection() { - let dir = TempDir::new().unwrap(); - let db = Database::open(dir.path().join("test.db")).unwrap(); - assert_eq!(db.file_count().unwrap(), 0); // First mount -} - -// Test: delta sync detects changes -#[tokio::test] -async fn test_delta_sync_detects_added_file() { - // DB has files A, B - // Origin has files A, B, C - // Delta sync should detect C as added -} - -// Test: delta sync detects removed file -#[tokio::test] -async fn test_delta_sync_detects_removed_file() { - // DB has files A, B, C - // Origin has files A, B - // Delta sync should detect C as removed -} - -// Test: shutdown checkpoints WAL -#[tokio::test] -async fn test_shutdown_checkpoints_wal() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("test.db"); - let db = Database::open(&db_path).unwrap(); - db.upsert_file(/* ... */).unwrap(); - - // WAL file should exist - let wal_path = db_path.with_extension("db-wal"); - // After checkpoint, WAL should be truncated - db.checkpoint().unwrap(); -} -``` - ---- - -## 6. Alternatives Considered - -### 6.1 sled for Tree Storage (Option B) - -sled is faster for bulk key-value reads (~1-2s for 1M entries vs SQLite's ~2-4s). Rejected because: -- SQLite code already exists (schema, CRUD, row mapping) -- sled would require new serialization layer (bincode/msgpack for FileMeta) -- Two persistence engines is more complex -- SQLite's 2-4s is acceptable for the target - -### 6.2 Flat File Snapshot (Option C) - -Fastest possible bulk load (<1s via mmap). Rejected because: -- No incremental updates — every change rewrites the entire file -- At 1M files (~500MB), delta sync triggers a 500MB write for each changed file -- No concurrent access safety -- No crash recovery for partial writes - -### 6.3 Lazy Tree Loading - -Instead of loading all files into memory on mount, load only the root directories and fetch deeper levels on demand from SQLite. This would achieve true O(1) mount. Deferred because: -- Requires significant refactoring of VirtualTree (currently all-in-memory) -- SQLite 2-4s load is good enough for production -- Can be added later as optimization without changing the persistence layer - -### 6.4 Separate Manifest Store - -Instead of storing manifests in the `files.chunk_manifest` column, use a separate sled tree or SQLite table. Rejected because the column already exists and the schema already supports it. - ---- - -## 7. Implementation Plan - -### 7.1 Task Sequence - -| Day | Task | Deliverable | -|-----|------|-------------| -| 1 | Database methods: `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`. Extract `row_to_file_meta()` helper. | New DB methods + tests | -| 2 | Rewrite `run_mount()`: DB load path vs scan path. First-mount detection. | Core mount change | -| 3 | Persist manifests: `ManifestCached` event + listener in main.rs. Load manifests on mount via `load_manifests_from_db()`. | Manifest persistence | -| 4 | Wire tantivy + PatternStore + CollectionStore into mount path. First-mount indexing. | Search/patterns on mount | -| 5 | Background delta sync: compare DB vs origin, update differences. | Delta sync task | -| 6 | Shutdown: WAL checkpoint. Upsert files to DB during first-mount scan. | Clean shutdown | -| 7 | Integration testing: full mount→read→restart→mount cycle. Verify tree + manifests survive restart. | E2E validation | -| 8 | Buffer for issues found during integration. | — | - -### 7.2 Verification Checklist - -- [ ] `cargo check` — zero errors -- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass -- [ ] Manual test: first mount (empty cache dir) — scans origin, creates DB -- [ ] Manual test: second mount (DB exists) — loads from DB, no origin scan -- [ ] Manual test: add file to origin, restart — delta sync discovers it -- [ ] Manual test: `kill -9` daemon, restart — DB loads, manifests intact -- [ ] Mount time for 10K test files: <1 second on subsequent mount -- [ ] `ls -la ~/.cache/musicfs/metadata.db` exists after first mount - ---- - -## 8. Files Changed - -| File | Change | -|------|--------| -| `musicfs-cache/src/db.rs` | `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`, `row_to_file_meta()` refactor | -| `musicfs-core/src/events.rs` | Add `ManifestCached` event variant | -| `musicfs-cli/src/main.rs` | Rewrite `run_mount()`: DB load vs scan, open tantivy/patterns/collections, manifest listener, delta sync spawn, shutdown checkpoint | -| `musicfs-cli/Cargo.toml` | Add `musicfs-search`, `musicfs-cache` dependencies (for PatternStore, CollectionStore, SearchIndex) | -| `musicfs-cas/src/fetcher.rs` | Emit `ManifestCached` event after `fetch_file()` | -| `musicfs-sync/src/delta.rs` | New `background_delta_sync()` function (or new file) | -| `musicfs-test-utils/tests/resilience.rs` | New tests: mount-from-DB, manifest roundtrip, delta sync, first-mount detection | - ---- - -## 9. Glossary / References - -| Term | Definition | -|------|------------| -| **First mount** | Initial mount with empty database — triggers full origin scan | -| **Subsequent mount** | Mount with existing database — loads from SQLite | -| **Delta sync** | Background task that compares DB state against origin after mount | -| **Stale data window** | Time between mount and delta sync completion when data may be outdated | -| **WAL checkpoint** | SQLite operation that flushes write-ahead log to main database file | - -| Document | Path | -|----------|------| -| Persistent state research | [persistent-state.md](persistent-state.md) | -| Phase A (signals, shutdown) | [phase-a-stop-dying.md](phase-a-stop-dying.md) | -| Phase B (crash recovery) | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) | -| Architecture | [architecture.md](../architecture.md) | diff --git a/docs/v2/plans/persistent-state.md b/docs/v2/plans/persistent-state.md deleted file mode 100644 index 6f47380..0000000 --- a/docs/v2/plans/persistent-state.md +++ /dev/null @@ -1,353 +0,0 @@ -# MusicFS Persistent State Plan - -**Date**: 2026-05-13 -**Status**: Research Complete — Design Decision Needed -**Prerequisites**: [architecture.md](../architecture.md), [resilience-fault-tolerance.md](resilience-fault-tolerance.md) -**Related Requirements**: G1 (O(1) mount time), NFR-1.7 (<500ms mount), FR-7.1 (cache persists across restarts) - ---- - -## 1. Problem Statement - -Every mount is a full cold start. The `run_mount()` function in `main.rs` does not use any persistent storage — it walks the entire origin filesystem, parses metadata from every audio file, and builds all runtime state from scratch. - -The architecture designed persistence infrastructure (SQLite schema, `chunk_manifest` column, `ChunkManifest::from_db()`, `chunks_to_bytes()`) but **none of it is wired into the mount path**. The mount flow doesn't even open the database. - -### Mount Time by Library Size (Current) - -| Library Size | Estimated Mount Time | Target (NFR-1.7) | -|---|---|---| -| 1K files | ~1-2s | <500ms | -| 10K files | ~10-20s | <500ms | -| 100K files | ~2-5 minutes | <500ms | -| 1M files | ~20-60 minutes | <500ms | -| 10M files (stretch) | hours | <500ms | - ---- - -## 2. In-Memory State Inventory - -### 2.1 State That Must Survive Restart - -These are the large, expensive-to-rebuild data structures. Losing them forces a full origin rescan. - -#### VirtualTree (~300-400MB at 1M files) - -**Location**: `musicfs-cache/src/tree.rs` - -**Contents**: -- `nodes: HashMap` — every directory and file node -- `path_to_inode: HashMap` — reverse path lookup -- `next_inode: AtomicU64` — inode counter - -**Currently rebuilt from**: Full recursive origin scan + metadata parse of every audio file. This is the single most expensive operation on mount — it touches every file on origin, runs symphonia metadata extraction, and builds the entire tree structure. - -**What's needed**: Load from persistent storage on mount. Rebuild only on first-ever mount or if storage is corrupt. - ---- - -#### ContentFetcher.file_meta (~200MB at 1M files) - -**Location**: `musicfs-cas/src/fetcher.rs` - -**Contents**: -- `file_meta: RwLock>` — full metadata for every file -- Each `FileMeta` contains: id, virtual_path, real_path (origin_id + path), size, mtime, content_hash, audio metadata - -**Currently rebuilt from**: Same origin scan that builds the tree. Every file is registered via `fetcher.register_file(meta)`. - -**What's needed**: This is essentially a duplicate of the tree data in a different shape. If the tree is loaded from storage, this map should be populated from the same source. - ---- - -#### FileReader.manifests (~100MB at 1M files) - -**Location**: `musicfs-cas/src/reader.rs` - -**Contents**: -- `manifests: RwLock>` — maps FileId to list of chunk hashes + offsets -- Each `ChunkManifest` contains: file_id, total_size, mtime, chunks (Vec with hash + offset + size) - -**Currently rebuilt from**: Re-fetched from origin on first `read()` after restart. The fetcher downloads the entire file, chunks it via CDC, stores chunks in CAS (dedup catches existing ones), and builds the manifest. This means every file is re-downloaded once after restart even though the chunks are already on disk. - -**What's needed**: Persist manifests to storage after fetch. Load on mount. This is the difference between "restart = re-download everything" and "restart = instant reads from cache." - -**Existing dead code**: SQLite `files` table has `chunk_manifest BLOB` column. `ChunkManifest::chunks_to_bytes()` and `ChunkManifest::from_db()` exist but are never called. - ---- - -#### LruEviction access times (~50MB at 100K chunks) - -**Location**: `musicfs-cache/src/eviction.rs` - -**Contents**: -- `access_times: RwLock>` — ordered by access time -- `hash_to_time: RwLock>` — reverse lookup - -**Currently rebuilt from**: Nothing. After restart, all chunks have equal eviction priority. The album you're currently listening to is just as likely to be evicted as something you played 6 months ago. - -**What's needed**: Persist last-access timestamps. On mount, load and reconstruct the LRU order so hot data stays cached. - ---- - -### 2.2 State That Survives But Is Ignored on Mount - -These persist on disk but `run_mount()` never opens them. - -| Component | Persisted To | Loaded on Mount? | Effect | -|---|---|---|---| -| SQLite metadata (files table) | `metadata.db` | ❌ | All metadata re-scanned from origin | -| tantivy search index | `search.idx/` | ❌ | Index rebuilt from scratch (or not at all) | -| PatternStore (access patterns) | SQLite (separate DB) | ❌ | Predictions reset to zero | -| CollectionStore (smart collections) | SQLite (same as patterns) | ❌ | Collections unavailable until opened | - -### 2.3 State That Correctly Does Not Need Persistence - -| Component | Why Transient Is Fine | -|---|---| -| OriginRegistry (origin connections) | Reconstructed from config on startup | -| Router (priorities, latency stats) | Priorities from config; latency stats warm up within seconds | -| HealthMonitor (health state) | All origins start as Unknown, converge within one check cycle (~30s) | -| EventBus (in-flight events) | Transient by nature | -| PrefetchEngine.in_flight | Transient work queue | -| PluginManager | Re-loaded from config + plugin directories | -| MusicFs.query_inodes | Transient search session state | -| CasStore.current_size | Recalculated on open (though currently broken — see resilience doc 3.10) | - ---- - -## 3. Storage Decision - -### 3.1 Requirements for Persistent State - -1. **Bulk sequential read on mount** — load ~1M records into in-memory structures as fast as possible -2. **Incremental updates at runtime** — delta sync adds/removes/modifies individual files -3. **Crash safety** — no corruption on unclean shutdown (SIGKILL, power loss) -4. **Manifest storage** — binary blobs (msgpack-encoded chunk lists), variable size (100 bytes to 10KB per file) -5. **LRU timestamps** — simple key-value (ChunkHash → last_access_timestamp) -6. **Already in project** — minimize new dependencies - -### 3.2 Options - -#### Option A: SQLite (Current Architecture Choice) - -**Already in project**: `rusqlite` dependency, `schema.sql` with `files` table, `Database` struct with full CRUD, `chunk_manifest BLOB` column ready. - -| Metric | Performance | -|---|---| -| Bulk load 1M rows | ~2-4 seconds (WAL mode, indexed) | -| Single row upsert | ~50μs | -| Crash safety | WAL mode — excellent | -| Manifest blobs | Native BLOB support, no size limit | - -**Pros**: Already built (schema, code, tests exist). Well-understood crash semantics. Single file backup. SQL queries for debugging. The `chunk_manifest` column and `from_db()`/`to_bytes()` methods are already written. - -**Cons**: Not the fastest for pure key-value workloads. WAL checkpoint can cause brief write pauses. Single-writer limitation (Mutex around connection). - -**Effort to wire up**: ~5-7 days (mostly connecting existing code, not writing new code) - ---- - -#### Option B: sled (Already in Project for CAS Index) - -**Already in project**: Used for CAS chunk hash → location mapping. - -| Metric | Performance | -|---|---| -| Bulk load 1M entries | ~1-2 seconds (LSM, sequential reads) | -| Single entry upsert | ~10-20μs | -| Crash safety | Built-in WAL — good | -| Manifest blobs | Native byte value support | - -**Pros**: Faster than SQLite for pure key-value. Already a dependency. Good for LRU timestamps (simple k/v). - -**Cons**: No SQL — querying for debugging is harder. No schema migration story. Limited tooling. Has known issues with large datasets (memory usage during compaction). Two persistence engines = two things to maintain. - -**Effort**: ~7-9 days (new serialization layer, no existing code to reuse) - ---- - -#### Option C: Flat File (bincode/msgpack dump) - -| Metric | Performance | -|---|---| -| Bulk load 1M entries | <1 second (mmap, zero-parse with bincode) | -| Single entry upsert | N/A — full rewrite required | -| Crash safety | Must write atomically (tmp + rename) | -| Manifest blobs | Part of serialized struct | - -**Pros**: Fastest possible bulk load. Simplest implementation. - -**Cons**: No incremental updates — every change requires serializing and rewriting the entire file. At 1M files (~500MB serialized), a single file modification triggers a 500MB write. No concurrent access. No recovery from partial corruption. - -**Effort**: ~3-4 days but creates ongoing maintenance burden for delta updates - ---- - -#### Option D: Hybrid (SQLite for metadata + sled for hot-path data) - -Use SQLite for structured metadata (files, collections, patterns — already built) and sled for hot-path key-value data (manifests, LRU timestamps — performance-critical). - -**Pros**: Each store optimized for its access pattern. SQLite for queryable metadata, sled for fast blob lookup. - -**Cons**: Two persistence engines to coordinate. Consistency between them on crash. More complex startup/shutdown. - ---- - -### 3.3 Recommendation - -**Pending your decision.** The tradeoffs are: -- **Simplest path**: Option A (SQLite) — most code already exists, just needs wiring -- **Fastest hot-path**: Option D (Hybrid) — but more complexity -- **Fastest bulk load**: Option C (Flat file) — but no incremental updates - -The choice depends on what you value most. SQLite at 1M files loads in ~2-4 seconds — is that acceptable vs the <500ms target? If not, a flat file or sled for the tree data with SQLite for everything else might be needed. - ---- - -## 4. What Needs to Change - -Regardless of storage choice, these are the code changes needed: - -### 4.1 Mount Path (musicfs-cli/src/main.rs) - -Current `run_mount()` flow: -``` -1. Open CAS store → O(1) -2. Create origin connection → O(1) -3. scan_music_files() — FULL ORIGIN WALK → O(N × origin_latency) ← BOTTLENECK -4. Build tree from scan results → O(N) -5. Register files in fetcher → O(N) -6. Mount FUSE → O(1) -``` - -Required flow: -``` -1. Open CAS store → O(1) -2. Open persistent state store → O(1) -3. IF store has data: - Load tree from store → O(N × local_read) ← ~1000x faster - Load manifests from store → O(N × local_read) - Load LRU access times from store → O(chunks) - ELSE (first mount): - Full origin scan (current behavior) → O(N × origin_latency) - Persist results to store → O(N × local_write) -4. Open tantivy search index → O(1) -5. Open PatternStore → O(1) -6. Create origin connections → O(1) -7. Mount FUSE → O(1) -8. Background: delta sync (origin vs store) → incremental, non-blocking -``` - -### 4.2 Runtime Persistence (Write Path) - -These operations must persist state changes as they happen, not just on shutdown: - -| Event | What to Persist | When | -|---|---|---| -| File discovered during sync | FileMeta → store | Immediately (in batch if scanning) | -| File removed during sync | Delete from store | Immediately | -| File metadata changed | Update FileMeta in store | Immediately | -| File content fetched (cache miss) | ChunkManifest → store | After fetch completes | -| Chunk accessed | Update LRU timestamp | Batched (every 10s or 100 accesses) | -| Search index updated | tantivy handles its own persistence | On commit (every 5s) | -| Access pattern recorded | PatternStore handles its own persistence | Already persisted per-access | - -### 4.3 Files That Need Changes - -| File | Change | -|---|---| -| `musicfs-cli/src/main.rs` | Rewrite `run_mount()` to load from store; add background delta sync | -| `musicfs-cache/src/db.rs` | Add `list_all_files()` bulk load; add manifest read/write methods (if SQLite) | -| `musicfs-cache/src/tree.rs` | Add `TreeBuilder::from_file_metas(iter)` — build tree from stored records | -| `musicfs-cas/src/reader.rs` | Load manifests from store on startup; persist after fetch | -| `musicfs-cas/src/fetcher.rs` | After `fetch_file()`, persist manifest to store | -| `musicfs-cache/src/eviction.rs` | Persist access times; load on startup | -| `musicfs-search/src/indexer.rs` | On mount, check what's already indexed vs what's in store — skip known files | -| `musicfs-sync/src/delta.rs` | Background delta sync: compare store state vs origin, sync differences | - -### 4.4 Shutdown Persistence - -On graceful shutdown (after signal handling from resilience plan Phase A is implemented): - -| Step | What | -|---|---| -| 1 | Flush any batched LRU timestamp updates | -| 2 | Commit tantivy index writer | -| 3 | WAL checkpoint SQLite (if SQLite): `PRAGMA wal_checkpoint(TRUNCATE)` | -| 4 | Flush sled (if sled): `sled::Db::flush()` | -| 5 | Close all database connections | - -On crash (no graceful shutdown): -- SQLite WAL mode: automatic recovery on next open (no data loss for committed transactions) -- sled: automatic recovery via internal WAL -- tantivy: up to 5 seconds of uncommitted documents lost, but recoverable from store -- LRU timestamps: batched updates may lose last batch (10s window) — acceptable - ---- - -## 5. Background Delta Sync - -After mounting from persistent state, the data may be stale (origin changed while daemon was stopped). A background sync reconciles: - -``` -1. Walk origin (or use watcher for inotify-capable origins) -2. For each file on origin: - a. Compare mtime + size against stored record - b. If unchanged → skip - c. If modified → re-parse metadata, update store, update tree, invalidate manifest - d. If new → parse metadata, add to store + tree -3. For each file in store not found on origin: - a. Remove from store + tree -4. Update search index for changed files -5. Log summary: "Delta sync complete: N added, M modified, K removed, T unchanged" -``` - -This runs in the background AFTER mount completes. Users see the filesystem immediately (from stored state), and it converges to current reality within minutes. - -### 5.1 Stale Data Window - -Between mount and delta sync completion, users may see: -- Files that were deleted on origin (will get ENOENT or EIO on read — origin returns not found) -- Files with old metadata (wrong track name, etc.) -- Missing files that were added to origin (won't appear until sync discovers them) - -This is acceptable — it's the same behavior as any cached filesystem (NFS, CIFS). The key insight: **stale data for 30 seconds is infinitely better than no data for 5 minutes.** - ---- - -## 6. First Mount vs Subsequent Mount - -| | First Mount (empty store) | Subsequent Mount (store has data) | -|---|---|---| -| **Tree source** | Origin scan + metadata parse | Load from store | -| **Manifests** | None (populated on first read) | Loaded from store | -| **Search index** | Built during/after scan | Opened from disk | -| **LRU data** | Empty (cold cache) | Loaded from store | -| **Mount time** | O(N × origin_latency) — same as today | O(N × local_read) — target <5s for 1M files | -| **Accuracy** | 100% current | Stale until delta sync completes | -| **Detection** | Store file doesn't exist or is empty | Store file exists with data | - ---- - -## 7. Estimated Effort - -| Task | Effort | Depends On | -|---|---|---| -| Rewrite `run_mount()` with store loading + fallback | 2 days | Storage decision | -| Persist chunk manifests after fetch | 1 day | Storage decision | -| Load manifests on mount + register in FileReader | 0.5 day | Above | -| Open tantivy on mount, skip known files | 1 day | — | -| Open PatternStore + CollectionStore on mount | 0.5 day | — | -| Background delta sync | 1.5 days | — | -| Persist LRU access times + load on mount | 1 day | Storage decision | -| First-mount detection + fallback to full scan | 0.5 day | — | -| **Total** | **~8 days** | | - ---- - -## 8. Open Decision - -**Which storage engine for the persistent state?** - -The answer drives the implementation of every task above. See Section 3 for tradeoffs. diff --git a/docs/v2/plans/phase-a-stop-dying.md b/docs/v2/plans/phase-a-stop-dying.md deleted file mode 100644 index eab0e5f..0000000 --- a/docs/v2/plans/phase-a-stop-dying.md +++ /dev/null @@ -1,569 +0,0 @@ -# Phase A: Stop Dying — Implementation Plan - -**Authors:** AI-assisted -**Status:** Draft -**Last Updated:** 2026-05-13 -**Reviewers:** TBD -**Approvers:** TBD -**Prerequisites:** [resilience-fault-tolerance.md](resilience-fault-tolerance.md), [resilience-testing.md](resilience-testing.md) -**Estimated Effort:** ~5 days - ---- - -[TOC] - ---- - -## 1. Abstract - -Implement the 6 most critical resilience fixes (issues 2.1, 2.2, 2.7, 2.9, 2.10, 3.7 from the [resilience audit](resilience-fault-tolerance.md)) that prevent MusicFS from dying on common operational events: signals, panics, lock poisoning, and systemd lifecycle. - -Issues 2.3 (shutdown orchestration), 2.4 (cache integrity), 2.5 (sync recovery), 2.6 (task supervisor), 2.8 (disk space) are deferred to Phase B — they depend on Phase A infrastructure or on the [persistent state](persistent-state.md) work. - -**Development flow** (TDD, per-issue): -1. Create stubs so the codebase compiles -2. Write RED tests that express the expected behavior -3. Implement the fix -4. Verify tests turn GREEN -5. Run full test suite — no regressions - ---- - -## 2. Background - -MusicFS currently dies on: -- Any signal (SIGTERM, SIGINT) — instant death, no cleanup -- Any panic in a writer thread — RwLock poisons, all FUSE ops crash -- systemd lifecycle — `Type=notify` but no `sd_notify`, ExecStop is a stub -- Crash leaves stale FUSE mount — users must manually `fusermount -u` - -The [resilience test crate](../../musicfs/crates/musicfs-test-utils/) and RED tests are already in place. This plan implements the fixes to turn them GREEN. - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -- Signal handler catches SIGTERM/SIGINT and initiates clean exit -- Panics are logged with full context before process terminates -- RwLock poisoning cannot cascade to kill FUSE operations -- systemd integration works (`sd_notify READY=1`, `ExecStopPost`) -- Stale FUSE mounts are detected and cleaned on startup -- All existing 162 tests continue to pass -- All Phase A RED tests turn GREEN - -### 3.2 Non-Goals - -- Graceful shutdown orchestration with ordered teardown (Phase B — needs CancellationToken plumbing through all components) -- Task supervisor for background task restart (Phase B) -- Cache integrity checks on startup (Phase B — needs persistent state) -- Disk space monitoring (Phase B) -- Interrupted sync recovery (Phase B — needs persistent state) - ---- - -## 4. Proposed Design - -### 4.1 Implementation Order - -Dependencies determine the order. Each issue is independent except where noted. - -``` -4.2 RwLock poison fix (no deps, instant win, unblocks safety) - ↓ -4.3 Panic hook (no deps, complements RwLock fix) - ↓ -4.4 systemd ExecStopPost (no deps, config-only change) - ↓ -4.5 sd_notify integration (no deps, new crate dependency) - ↓ -4.6 Signal handling (depends on: FUSE mount change to spawn_mount2) - ↓ -4.7 Stale mount detection (depends on: signal handling for clean test) -``` - -### 4.2 Issue 2.9: RwLock Poison Fix - -**Approach**: Replace `std::sync::RwLock` with `parking_lot::RwLock` in all production paths. `parking_lot` never poisons — a panic in a writer releases the lock and subsequent readers see the pre-panic state. - -**Why parking_lot over poison recovery**: The codebase already uses `parking_lot` in `prefetch.rs` and `index.rs`. Using it everywhere is consistent. The alternative (`.unwrap_or_else(|p| p.into_inner())`) is verbose and error-prone — one missed call re-introduces the bug. - -#### Step 1: Stubs (compile) - -None needed — `parking_lot::RwLock` is a drop-in replacement (same API, no `PoisonError`). - -#### Step 2: RED tests - -Already exist in `tests/resilience.rs`: -- `test_poisoned_tree_lock_returns_eio_not_panic` — currently passes (demonstrates the problem) -- `test_parking_lot_rwlock_survives_panic` — currently passes (proves the fix works) - -Additional test to add: verify FUSE filesystem survives a writer panic on the tree lock. - -#### Step 3: Implementation - -**Files to change:** - -| File | Change | -|------|--------| -| `musicfs-fuse/src/filesystem.rs` | `use std::sync::RwLock` → `use parking_lot::RwLock`; remove all `.unwrap()` on lock calls (parking_lot returns guard directly, not `Result`) | -| `musicfs-cas/src/reader.rs` | Same change for `manifests: RwLock>` | -| `musicfs-cas/src/fetcher.rs` | Same change for `origins` and `file_meta` locks | -| `musicfs-origins/src/registry.rs` | Same change for `origins` and `watch_handles` locks | -| `musicfs-cache/src/eviction.rs` | Same change for `access_times` and `hash_to_time` locks | -| `musicfs-core/src/metrics.rs` | Same change for histogram locks | -| `musicfs-cache/src/tree.rs` | Same change for `last_refresh` lock | - -**Pattern**: In each file: -```rust -// BEFORE -use std::sync::RwLock; -let guard = self.tree.read().unwrap(); - -// AFTER -use parking_lot::RwLock; -let guard = self.tree.read(); // No unwrap needed -``` - -For the `MusicFs` struct in `filesystem.rs`, the `tree` field is `Arc>` — this is passed in from `main.rs`. Change `main.rs` to use `parking_lot::RwLock` there too. - -#### Step 4: Verify - -```bash -cargo test # All 162+ tests pass -cargo test -p musicfs-test-utils # Resilience tests pass -cargo check # No warnings -``` - ---- - -### 4.3 Issue 2.2: Panic Hook - -**Approach**: Install a custom panic hook at daemon startup that logs the panic with `tracing::error!` before the default behavior (abort/unwind). This ensures panics are captured in log files and journald. - -#### Step 1: Stubs - -Add to `musicfs-core/src/lib.rs`: -```rust -pub fn install_panic_hook() { - // stub — will be implemented -} -``` - -#### Step 2: RED tests - -Write in `tests/resilience.rs`: -```rust -#[test] -fn test_panic_hook_logs_to_tracing() { - // Install hook with a test tracing subscriber - // Trigger panic via catch_unwind - // Verify error! log contains panic message + thread name -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-core/src/lib.rs` (or new `musicfs-core/src/panic.rs`) - -```rust -pub fn install_panic_hook() { - let default_hook = std::panic::take_hook(); - std::panic::set_hook(Box::new(move |info| { - let thread = std::thread::current(); - let thread_name = thread.name().unwrap_or(""); - - let message = if let Some(s) = info.payload().downcast_ref::<&str>() { - s.to_string() - } else if let Some(s) = info.payload().downcast_ref::() { - s.clone() - } else { - "unknown panic".to_string() - }; - - let location = info.location().map(|l| format!("{}:{}:{}", l.file(), l.line(), l.column())) - .unwrap_or_else(|| "unknown location".to_string()); - - tracing::error!( - thread = thread_name, - location = %location, - "PANIC: {}", - message - ); - - default_hook(info); - })); -} -``` - -**Call site**: `musicfs-cli/src/main.rs`, at the very top of `main()`: -```rust -fn main() -> Result<()> { - musicfs_core::install_panic_hook(); - let cli = Cli::parse(); - // ... -} -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-core # Panic hook unit tests -cargo test -p musicfs-test-utils # Resilience tests -``` - ---- - -### 4.4 Issue 3.7 + 2.7: systemd Service Fix + FUSE Cleanup - -**Approach**: Fix the systemd service file and add stale mount detection on startup. - -#### Step 1: No stubs needed (config change) - -#### Step 2: RED tests - -Already exists: `test_systemd_service_has_execstoppost` — currently fails because service file lacks `ExecStopPost`. - -Add test for stale mount detection: -```rust -#[test] -fn test_stale_mount_check_function_exists() { - // Verify the function signature exists - // (actual mount test needs privileged environment) -} -``` - -#### Step 3: Implementation - -**File**: `dist/musicfs.service` - -```diff - ExecStop=/usr/bin/musicfs shutdown -+ExecStopPost=/usr/bin/fusermount -uz %h/music || true - Restart=on-failure -``` - -Note: `fusermount -uz` is "lazy unmount" — always succeeds even if mount is busy. The `|| true` prevents systemd from treating cleanup failure as a service failure. - -**File**: `musicfs-cli/src/main.rs` — add stale mount check before mounting: - -```rust -fn check_stale_mount(mountpoint: &Path) -> Result<()> { - // Check /proc/mounts for existing mount at this path - if let Ok(mounts) = std::fs::read_to_string("/proc/mounts") { - for line in mounts.lines() { - if line.contains(&mountpoint.to_string_lossy().as_ref()) && line.contains("fuse") { - warn!("Stale FUSE mount detected at {:?}, attempting cleanup", mountpoint); - let status = std::process::Command::new("fusermount") - .args(["-uz", &mountpoint.to_string_lossy()]) - .status(); - match status { - Ok(s) if s.success() => info!("Stale mount cleaned up"), - Ok(s) => warn!("fusermount exited with: {}", s), - Err(e) => warn!("Failed to run fusermount: {}", e), - } - } - } - } - Ok(()) -} -``` - -Also fix the `test_systemd_service_has_execstoppost` test path — currently points to wrong location. - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils -- test_systemd # Service file test -``` - ---- - -### 4.5 Issue 2.10: sd_notify Integration - -**Approach**: Add `sd-notify` crate, call `READY=1` after mount, `STOPPING` on shutdown. - -#### Step 1: Stubs - -Add dependency to `musicfs-cli/Cargo.toml`: -```toml -sd-notify = "0.4" -``` - -#### Step 2: RED tests - -Write test that mocks the notify socket: -```rust -#[test] -fn test_sd_notify_ready_sent() { - // Create Unix datagram socket at $NOTIFY_SOCKET - // Call sd_notify::notify(READY=1) - // Verify message received on socket -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cli/src/main.rs` - -After `fs.mount()` succeeds (or more precisely, after `spawn_mount2` — see 4.6): -```rust -// Notify systemd we're ready -if let Err(e) = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]) { - debug!("sd_notify not available (not running under systemd): {}", e); -} -``` - -On shutdown path: -```rust -let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]); -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils -- test_sd_notify -cargo build -p musicfs-cli # Verify it compiles with new dep -``` - ---- - -### 4.6 Issue 2.1: Signal Handling - -**Approach**: Switch from `fuser::mount2` (blocking) to `fuser::spawn_mount2` (background), then listen for signals on the main thread. - -This is the most complex change in Phase A. It restructures the daemon's main loop. - -#### Step 1: Stubs - -Change `MusicFs::mount()` signature to return a session handle: - -```rust -// BEFORE -pub fn mount(self, mountpoint: &Path) -> Result<()> { - fuser::mount2(self, mountpoint, &options)?; - Ok(()) -} - -// AFTER (stub — returns BackgroundSession) -pub fn spawn_mount(self, mountpoint: &Path) -> Result { - let session = fuser::spawn_mount2(self, mountpoint, &options)?; - Ok(session) -} -``` - -Keep old `mount()` temporarily for compatibility. - -#### Step 2: RED tests - -Write in `tests/resilience.rs`: -```rust -#[tokio::test] -async fn test_sigterm_triggers_shutdown() { - // Spawn daemon as child process - // Wait for mount - // Send SIGTERM - // Verify clean exit within 10s - // Verify mountpoint is unmounted -} -``` - -This test requires the signal handler to exist. It will be RED until implementation. - -#### Step 3: Implementation - -**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()`: - -```rust -fn run_mount(mountpoint: PathBuf, origin_path: Option, cache_dir: Option) -> Result<()> { - let origin_path = origin_path.context("--origin is required")?; - let runtime = tokio::runtime::Runtime::new()?; - let handle = runtime.handle().clone(); - - let (tree, reader) = runtime.block_on(async { - // ... existing setup code (unchanged) ... - Ok::<_, anyhow::Error>((tree, reader)) - })?; - - // Check for stale mount before mounting - check_stale_mount(&mountpoint)?; - - let fs = MusicFs::with_reader(tree, reader, handle.clone()); - info!("Mounting filesystem at {:?}", mountpoint); - - // spawn_mount2 returns immediately — FUSE runs in background - let session = fs.spawn_mount(&mountpoint) - .context("Failed to mount filesystem")?; - - // Notify systemd - let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]); - info!("MusicFS ready, PID {}", std::process::id()); - - // Block on signal - runtime.block_on(async { - let mut sigterm = tokio::signal::unix::signal( - tokio::signal::unix::SignalKind::terminate() - )?; - let mut sigint = tokio::signal::unix::signal( - tokio::signal::unix::SignalKind::interrupt() - )?; - - tokio::select! { - _ = sigterm.recv() => { - info!("Received SIGTERM, shutting down"); - } - _ = sigint.recv() => { - info!("Received SIGINT, shutting down"); - } - } - - Ok::<_, anyhow::Error>(()) - })?; - - // Shutdown sequence - let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]); - info!("Unmounting filesystem"); - drop(session); // BackgroundSession::drop() calls unmount - info!("Shutdown complete"); - - Ok(()) -} -``` - -**File**: `musicfs-fuse/src/filesystem.rs` — add `spawn_mount()`: - -```rust -pub fn spawn_mount(self, mountpoint: &Path) -> Result { - info!("Mounting MusicFS at {:?}", mountpoint); - let options = vec![ - fuser::MountOption::RO, - fuser::MountOption::FSName("musicfs".to_string()), - fuser::MountOption::AutoUnmount, - fuser::MountOption::AllowOther, - ]; - let session = fuser::spawn_mount2(self, mountpoint, &options) - .map_err(musicfs_core::Error::Io)?; - Ok(session) -} -``` - -#### Step 4: Verify - -```bash -cargo build -p musicfs-cli -cargo test -p musicfs-test-utils -- test_sigterm # Process-level test -cargo test # No regressions -``` - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -- No new attack surface — changes are internal lifecycle management -- Panic hook does NOT log sensitive data (only panic message, thread name, location) -- `sd_notify` uses existing systemd socket — no new IPC - -### 5.2 Observability - -- Panic hook ensures all panics are captured in logs/journald -- Signal handling logs which signal triggered shutdown -- sd_notify gives systemd accurate service state -- Stale mount detection logs cleanup attempts - -### 5.3 Testing - -All changes follow the TDD flow: -1. Stubs compile -2. RED tests document expected behavior -3. Implementation turns tests GREEN -4. Full suite passes (no regressions) - ---- - -## 6. Alternatives Considered - -### 6.1 Poison Recovery Instead of parking_lot - -**Alternative**: Keep `std::sync::RwLock`, add `.unwrap_or_else(|p| p.into_inner())` to every lock call. - -**Rejected**: 30+ call sites to change, easy to miss one, and the pattern is verbose. `parking_lot` is already a dependency and is strictly better for this use case (faster, no poison, correct API). - -### 6.2 Keep mount2 (blocking) with Signal Thread - -**Alternative**: Keep `fuser::mount2`, spawn a separate thread for signal handling, use a channel to communicate shutdown. - -**Rejected**: `mount2` consumes `self` and blocks — there's no clean way to interrupt it from another thread. `spawn_mount2` is the canonical solution from the `fuser` crate. - -### 6.3 Defer sd_notify Until Full Shutdown Orchestration - -**Alternative**: Implement sd_notify only after CancellationToken + graceful shutdown are in place. - -**Rejected**: sd_notify `READY=1` is critical now — without it, `Type=notify` in the service file means systemd will timeout and kill the daemon on every start. The shutdown `STOPPING` notification is a bonus but not required for Phase A. - ---- - -## 7. Implementation Plan - -### 7.1 Task Sequence - -| Day | Task | Issue | Effort | Test Approach | -|-----|------|-------|--------|---------------| -| 1 (morning) | RwLock → parking_lot migration | 2.9 | 2h | Existing GREEN test validates; verify no `.unwrap()` on locks | -| 1 (afternoon) | Panic hook | 2.2 | 2h | New test: panic → verify tracing output | -| 2 (morning) | systemd ExecStopPost + stale mount check | 3.7 + 2.7 | 2h | Existing RED test → GREEN; new stale mount test | -| 2 (afternoon) | sd_notify integration | 2.10 | 2h | New test: mock socket → verify READY=1 | -| 3 | Signal handling (spawn_mount2 + signal loop) | 2.1 | 4h | Fork daemon → send SIGTERM → verify exit | -| 4 | Integration + regression testing | — | 4h | Full `cargo test`, manual FUSE mount test | -| 5 | Buffer for issues found during integration | — | 4h | — | - -### 7.2 Verification Checklist - -After all tasks complete: - -- [ ] `cargo check` — zero errors, zero warnings -- [ ] `cargo test` — all 162+ existing tests pass -- [ ] `cargo test -p musicfs-test-utils` — all resilience tests pass -- [ ] `cargo clippy` — no new warnings -- [ ] `grep -r '\.read()\.unwrap()\|\.write()\.unwrap()' crates/` — zero hits in production code (test code is OK) -- [ ] `dist/musicfs.service` contains `ExecStopPost` -- [ ] Manual test: `musicfs mount`, then `kill -TERM `, verify clean exit + mount gone -- [ ] Manual test: `kill -9 `, then `musicfs mount` again — no "already mounted" error - ---- - -## 8. Files Changed - -| File | Change | Issue | -|------|--------|-------| -| `musicfs-fuse/src/filesystem.rs` | `std::sync::RwLock` → `parking_lot::RwLock`; add `spawn_mount()` | 2.9, 2.1 | -| `musicfs-cas/src/reader.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-cas/src/fetcher.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-origins/src/registry.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-cache/src/eviction.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-cache/src/tree.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-core/src/metrics.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 | -| `musicfs-core/src/lib.rs` | Add `install_panic_hook()` | 2.2 | -| `musicfs-cli/src/main.rs` | Panic hook, signal handler, spawn_mount2, sd_notify, stale mount check | 2.1, 2.2, 2.7, 2.10 | -| `musicfs-cli/Cargo.toml` | Add `sd-notify`, `tokio-util` deps | 2.10, 2.1 | -| `dist/musicfs.service` | Add `ExecStopPost`, fix `ExecStop` | 3.7 | -| `tests/resilience.rs` | Update/add tests for signal, panic hook, sd_notify | all | - ---- - -## 9. Glossary / References - -| Term | Definition | -|------|------------| -| **parking_lot** | Fast, poison-free lock implementation. Already a project dependency. | -| **spawn_mount2** | `fuser` API that mounts FUSE in a background thread, returning a `BackgroundSession` handle | -| **sd_notify** | systemd notification protocol. `READY=1` signals service started, `STOPPING` signals shutdown. | -| **BackgroundSession** | Handle returned by `spawn_mount2`. Dropping it unmounts the filesystem. | - -| Document | Path | -|----------|------| -| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) | -| Resilience testing | [resilience-testing.md](resilience-testing.md) | -| Architecture | [architecture.md](../architecture.md) | diff --git a/docs/v2/plans/phase-b-crash-recovery.md b/docs/v2/plans/phase-b-crash-recovery.md deleted file mode 100644 index bba7cf1..0000000 --- a/docs/v2/plans/phase-b-crash-recovery.md +++ /dev/null @@ -1,830 +0,0 @@ -# Phase B: Crash Recovery — Implementation Plan - -**Authors:** AI-assisted -**Status:** Draft -**Last Updated:** 2026-05-13 -**Reviewers:** TBD -**Approvers:** TBD -**Prerequisites:** [phase-a-stop-dying.md](phase-a-stop-dying.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md) -**Estimated Effort:** ~5 days - ---- - -[TOC] - ---- - -## 1. Abstract - -Phase A made the daemon survive signals and panics. Phase B makes it **recover from crashes** — startup integrity checks for all storage layers (SQLite, tantivy, sled), graceful shutdown with ordered teardown of background tasks, disk space pre-checks, and a task supervisor that restarts dead background tasks. - -This covers issues 2.3, 2.4, 2.6, and 2.8 from the [resilience audit](resilience-fault-tolerance.md), deferred from Phase A. - -Issue 2.5 (interrupted sync recovery) is deferred to after [persistent state](persistent-state.md) is wired up — checkpoint/resume requires the DB to be in the mount path. - -**RED tests to turn GREEN** (from current `resilience.rs`): -- `test_sqlite_integrity_check_detects_corruption` — currently `todo!()` -- `test_tantivy_corruption_triggers_rebuild` — currently `todo!()` -- `test_sled_corruption_triggers_repair` — currently `todo!()` -- `test_cas_put_handles_enospc` — currently fails (no size pre-check) -- `test_tantivy_survives_uncommitted_crash` — currently `todo!()` - -**New tests to write:** -- Shutdown orchestration: CancellationToken propagation, ordered teardown, tantivy flush -- Task supervisor: panic detection, restart with backoff, status reporting - ---- - -## 2. Background - -### 2.1 What Phase A Delivered - -- Signal handling via `spawn_mount2` + tokio signal loop ✅ -- Panic hook logging via `tracing::error!` ✅ -- RwLock → `parking_lot` (no more poison cascade) ✅ -- sd_notify READY/STOPPING ✅ -- ExecStopPost + stale mount detection ✅ - -### 2.2 What's Still Broken After Phase A - -The daemon now **stops cleanly** on signals but: - -1. **Shutdown is unordered** — `drop(session)` unmounts FUSE, but background tasks (health monitor, indexer, watcher, prefetcher) are killed mid-operation by runtime drop. No tantivy flush, no SQLite checkpoint. - -2. **No startup integrity checks** — if the daemon was `kill -9`'d (or OOM-killed, power loss), SQLite/tantivy/sled may have partial writes. Currently these propagate as runtime errors or silent corruption. - -3. **Background tasks are fire-and-forget** — health monitor, watcher, indexer, prefetcher use `tokio::spawn` with no `JoinHandle` stored. If a task panics, it's silently dead. - -4. **CAS accepts oversized writes** — `put()` doesn't check `max_size` before writing. Cache grows unbounded. - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -- Graceful shutdown flushes tantivy, checkpoints SQLite WAL, stops background tasks in order -- Corrupted SQLite detected on open via `PRAGMA integrity_check` -- Corrupted tantivy index detected and rebuilt from scratch -- Corrupted sled index detected and repaired -- CAS rejects writes that would exceed `max_size` -- Background tasks are supervised — panics detected, critical tasks restarted -- All 5 RED tests turn GREEN -- All new tests for shutdown + supervisor are GREEN - -### 3.2 Non-Goals - -- Interrupted sync recovery (2.5) — depends on persistent state work -- Disk space monitoring daemon (periodic `statvfs`) — Phase C -- Connection pooling, config reload, watchdog — Phase C/D -- Passthrough mode when cache dies — Phase F - ---- - -## 4. Proposed Design - -### 4.1 Implementation Order - -``` -4.2 CAS size pre-check (no deps, simplest fix) - ↓ -4.3 SQLite integrity check (no deps) - ↓ -4.4 tantivy corruption recovery (no deps) - ↓ -4.5 sled corruption recovery (no deps) - ↓ -4.6 Graceful shutdown orchestration (depends on: Phase A signal handler) - ↓ -4.7 Task supervisor (depends on: 4.6 CancellationToken) -``` - -### 4.2 Issue 2.8: CAS Size Pre-Check - -**Problem**: `CasStore::put()` writes data without checking if it would exceed `max_size`. The existing test `test_cas_put_handles_enospc` creates a store with `max_size: 100` and writes 1000 bytes — currently succeeds when it should fail. - -#### Step 1: Stubs — none needed - -#### Step 2: RED test — already exists - -```rust -// Currently FAILS — this is what we need to fix -#[tokio::test] -async fn test_cas_put_handles_enospc() { - let store = CasStore::open(CasConfig { max_size: 100, ... }).await.unwrap(); - let large_data = vec![0u8; 1000]; - let result = store.put(&large_data).await; - assert!(result.is_err()); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cas/src/store.rs` — add size check at top of `put()`: - -```rust -pub async fn put(&self, data: &[u8]) -> Result { - let hash = ChunkHash::from_bytes(data); - let path = self.chunk_path(&hash); - - if path.exists() { - trace!(hash = %hash, size_bytes = data.len(), "dedup hit"); - return Ok(hash); - } - - // NEW: Pre-check size limit - if self.config.max_size > 0 { - let new_size = self.current_size.load(Ordering::SeqCst) + data.len() as u64; - if new_size > self.config.max_size { - warn!( - current_size = self.current_size.load(Ordering::SeqCst), - chunk_size = data.len(), - max_size = self.config.max_size, - "CAS store full, rejecting write" - ); - return Err(CasError::StoreFull { - current: self.current_size.load(Ordering::SeqCst), - max: self.config.max_size, - }); - } - } - - // ... rest of put() unchanged -} -``` - -Also add new error variant: - -```rust -pub enum CasError { - // ... existing variants - #[error("Store full: {current} / {max} bytes")] - StoreFull { current: u64, max: u64 }, -} -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_cas_put_handles_enospc -``` - ---- - -### 4.3 Issue 2.4 (part 1): SQLite Integrity Check - -**Problem**: `Database::open()` runs schema but no integrity check. After crash, corrupt pages serve bad data silently. - -#### Step 1: Stubs - -Add to `musicfs-cache/src/db.rs`: - -```rust -pub fn open_with_integrity_check(path: &Path) -> Result { - todo!() -} -``` - -#### Step 2: RED test — already exists as `todo!()` - -Replace the `todo!()` with a real test: - -```rust -#[tokio::test] -async fn test_sqlite_integrity_check_detects_corruption() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("test.db"); - - // Create valid DB with data - { - let db = Database::open(&db_path).unwrap(); - db.upsert_file( - &OriginId::from("test"), - Path::new("/test.flac"), - &VirtualPath::new("/Test.flac"), - &AudioMeta::default(), - UNIX_EPOCH, - 1000, - ).unwrap(); - } - - // Corrupt the file - let mut data = std::fs::read(&db_path).unwrap(); - let mid = data.len() / 2; - data[mid..mid+100].fill(0xFF); - std::fs::write(&db_path, &data).unwrap(); - - // open_with_integrity_check should detect corruption - let result = Database::open_with_integrity_check(&db_path); - assert!(result.is_err()); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cache/src/db.rs` - -```rust -pub fn open_with_integrity_check(path: &Path) -> Result { - debug!(?path, "Opening database with integrity check"); - - let conn = Connection::open(path) - .map_err(|e| Error::Database(format!("open failed: {}", e)))?; - - // Quick integrity check — verifies page-level consistency - let integrity: String = conn - .query_row("PRAGMA integrity_check(1)", [], |row| row.get(0)) - .map_err(|e| Error::Database(format!("integrity check failed: {}", e)))?; - - if integrity != "ok" { - warn!(path = ?path, result = %integrity, "Database integrity check failed"); - return Err(Error::DatabaseCorrupted(format!( - "integrity check failed: {}", integrity - ))); - } - - conn.execute_batch(SCHEMA) - .map_err(|e| Error::Database(format!("schema init failed: {}", e)))?; - - let db = Self { conn: Arc::new(Mutex::new(conn)) }; - let count = db.file_count().unwrap_or(0); - info!(path = ?path, file_count = count, "Database opened (integrity verified)"); - Ok(db) -} -``` - -Also add the error variant to `musicfs-core/src/error.rs`: - -```rust -pub enum Error { - // ... existing - #[error("Database corrupted: {0}")] - DatabaseCorrupted(String), -} -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_sqlite_integrity -``` - ---- - -### 4.4 Issue 2.4 (part 2): tantivy Corruption Recovery - -**Problem**: If tantivy `meta.json` or segment files are corrupted, `Index::open_in_dir()` panics or returns an error. No recovery path — daemon crashes. - -#### Step 1: Stubs - -Add to `musicfs-search/src/index.rs`: - -```rust -pub fn open_with_recovery(index_path: &Path) -> Result { - todo!() -} -``` - -#### Step 2: RED test — replace `todo!()` with real test - -```rust -#[tokio::test] -async fn test_tantivy_corruption_triggers_rebuild() { - let dir = TempDir::new().unwrap(); - let index_path = dir.path().join("search_idx"); - - // Create valid index with data - { - let index = SearchIndex::open(&index_path).unwrap(); - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - index.commit().unwrap(); - } - - // Corrupt meta.json - std::fs::write(index_path.join("meta.json"), b"corrupted").unwrap(); - - // open_with_recovery should detect corruption and rebuild empty - let index = SearchIndex::open_with_recovery(&index_path).unwrap(); - let results = index.search("a", 10).unwrap(); - assert_eq!(results.len(), 0); // Rebuilt empty but functional -} -``` - -Also replace the tantivy crash test `todo!()`: - -```rust -#[test] -fn test_tantivy_survives_uncommitted_crash() { - let dir = TempDir::new().unwrap(); - let index_path = dir.path().join("search_idx"); - - { - let index = SearchIndex::open(&index_path).unwrap(); - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - index.commit().unwrap(); - // Write without commit, then "crash" (drop without commit) - index.index_file(&make_file_meta(2, "/b.flac", 1000)).unwrap(); - // mem::forget would leak, just drop naturally - } - - let index = SearchIndex::open(&index_path).unwrap(); - let results = index.search("a", 10).unwrap(); - assert_eq!(results.len(), 1); // Committed doc survives -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-search/src/index.rs` - -```rust -pub fn open_with_recovery(index_path: &Path) -> Result { - match Self::open(index_path) { - Ok(index) => { - // Verify index is functional with a simple search - match index.reader.searcher().num_docs() { - docs => { - info!(docs, "Search index opened successfully"); - Ok(index) - } - } - } - Err(e) => { - warn!( - error = %e, - path = ?index_path, - "Search index corrupted, rebuilding from scratch" - ); - // Delete corrupted index - if index_path.exists() { - std::fs::remove_dir_all(index_path) - .map_err(|e| SearchError::Io(e))?; - } - // Create fresh index - Self::open(index_path) - } - } -} -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_tantivy -``` - ---- - -### 4.5 Issue 3.5: sled Corruption Recovery - -**Problem**: `sled::open()` on a corrupted DB returns `sled::Error::Corruption` which propagates as `CasError::Sled` and crashes the daemon on startup. - -#### Step 1: Stubs — none needed, modify existing `open()` - -#### Step 2: RED test — replace `todo!()` - -```rust -#[tokio::test] -async fn test_sled_corruption_triggers_repair() { - let dir = TempDir::new().unwrap(); - let chunks_dir = dir.path().join("chunks"); - let config = CasConfig { chunks_dir: chunks_dir.clone(), max_size: 10_000_000, shard_levels: 2 }; - - // Create valid store with data - { - let store = CasStore::open(config.clone()).await.unwrap(); - store.put(b"test data").await.unwrap(); - } - - // Corrupt sled index files - let sled_dir = chunks_dir.join("index.sled"); - if sled_dir.exists() { - for entry in std::fs::read_dir(&sled_dir).unwrap() { - let entry = entry.unwrap(); - if entry.metadata().unwrap().is_file() { - std::fs::write(entry.path(), b"corrupted").unwrap(); - } - } - } - - // Re-open should recover (repair or recreate) - let result = CasStore::open(config).await; - assert!(result.is_ok(), "sled should recover from corruption"); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cas/src/store.rs` — modify `open()`: - -```rust -pub async fn open(config: CasConfig) -> Result { - fs::create_dir_all(&config.chunks_dir).await?; - - let index_path = config.chunks_dir.join("index.sled"); - let index = match sled::open(&index_path) { - Ok(db) => db, - Err(e) => { - warn!(error = %e, path = ?index_path, "sled index corrupted, attempting recovery"); - - // Try repair - match sled::Config::new().path(&index_path).repair(true).open() { - Ok(db) => { - info!("sled index repaired successfully"); - db - } - Err(repair_err) => { - warn!(error = %repair_err, "sled repair failed, recreating index"); - // Delete and recreate - if index_path.exists() { - std::fs::remove_dir_all(&index_path) - .map_err(|e| CasError::Io(e))?; - } - sled::open(&index_path)? - } - } - } - }; - - let current_size = Self::calculate_size(&config.chunks_dir).await; - - Ok(Self { - config, - index, - current_size: AtomicU64::new(current_size), - }) -} -``` - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_sled_corruption -``` - ---- - -### 4.6 Issue 2.3: Graceful Shutdown Orchestration - -**Problem**: On signal, `drop(session)` unmounts FUSE, then `drop(runtime)` kills all tokio tasks abruptly. No tantivy flush, no SQLite WAL checkpoint, no ordered task shutdown. - -**Approach**: `CancellationToken` from `tokio_util` propagated to all background tasks. Signal triggers token cancellation, then ordered shutdown. - -#### Step 1: Add dependency - -```toml -# musicfs-cli/Cargo.toml -tokio-util = { version = "0.7", features = ["rt"] } -``` - -#### Step 2: Tests - -```rust -#[tokio::test] -async fn test_shutdown_cancels_background_tasks() { - let token = CancellationToken::new(); - let stopped = Arc::new(AtomicBool::new(false)); - let stopped_clone = stopped.clone(); - let token_clone = token.clone(); - - tokio::spawn(async move { - token_clone.cancelled().await; - stopped_clone.store(true, Ordering::SeqCst); - }); - - assert!(!stopped.load(Ordering::SeqCst)); - token.cancel(); - tokio::time::sleep(Duration::from_millis(50)).await; - assert!(stopped.load(Ordering::SeqCst)); -} - -#[tokio::test] -async fn test_shutdown_flushes_tantivy() { - let dir = TempDir::new().unwrap(); - let index = SearchIndex::open(dir.path().join("idx")).unwrap(); - - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - // Graceful shutdown should commit - index.commit().unwrap(); - - let index2 = SearchIndex::open(dir.path().join("idx")).unwrap(); - assert_eq!(index2.search("a", 10).unwrap().len(), 1); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cli/src/main.rs` — restructure the signal loop: - -The current code: -```rust -// Wait for signal -runtime.block_on(async { ... signal select ... })?; -// Drop session, exit -``` - -Change to: -```rust -let shutdown_token = CancellationToken::new(); - -// TODO: Pass token to health monitor, watcher, indexer, prefetcher -// (requires their start() methods to accept CancellationToken) -// For now, we just use it for the shutdown sequence - -runtime.block_on(async { - // ... signal select ... - - // Ordered shutdown - info!("Beginning ordered shutdown"); - shutdown_token.cancel(); - - // Wait briefly for tasks to notice cancellation - tokio::time::sleep(Duration::from_millis(500)).await; - - // Flush search index if available - // (requires SearchIndex to be accessible — currently not wired in main.rs) - - info!("Background tasks stopped"); -})?; -``` - -**Note**: Full CancellationToken propagation through health monitor, watcher, indexer, and prefetcher `start()` methods requires changing their signatures. The current `mpsc::channel<()>` stop mechanism in each task should be replaced with or supplemented by the token. This can be done incrementally — start by adding the token to `run_mount()`, then wire it into each task as they're touched. - -For this phase, the minimum viable change is: -1. Create the token in `run_mount()` -2. Cancel it on signal -3. Add a brief sleep for tasks to notice -4. The existing `drop(session)` and runtime drop handle cleanup - -Full per-task CancellationToken wiring is tracked as follow-up work. - ---- - -### 4.7 Issue 2.6: Task Supervisor - -**Problem**: 13 `tokio::spawn()` calls with no `JoinHandle` stored. Dead tasks go unnoticed. - -**Approach**: New `TaskSupervisor` struct in `musicfs-core` that stores handles, checks liveness, and restarts critical tasks. - -#### Step 1: Stubs - -**File**: `musicfs-core/src/supervisor.rs` (new file) - -```rust -pub struct TaskSupervisor { ... } - -pub enum TaskStatus { - Running, - Failed { error: String, at: Instant }, - Restarting { attempt: u32 }, - Stopped, -} - -impl TaskSupervisor { - pub fn new() -> Self; - pub fn spawn_supervised(&self, name: &str, future: impl Future) -> (); - pub fn spawn_critical(&self, name: &str, factory: impl Fn() -> impl Future) -> (); - pub fn task_status(&self, name: &str) -> TaskStatus; - pub fn check_all(&self) -> Vec<(String, TaskStatus)>; -} -``` - -#### Step 2: Tests - -```rust -#[tokio::test] -async fn test_supervisor_detects_task_completion() { - let supervisor = TaskSupervisor::new(); - supervisor.spawn_supervised("fast", async { /* returns immediately */ }); - tokio::time::sleep(Duration::from_millis(50)).await; - // Task completed normally — should be Stopped, not Failed -} - -#[tokio::test] -async fn test_supervisor_detects_panic() { - let supervisor = TaskSupervisor::new(); - supervisor.spawn_supervised("panicker", async { - panic!("boom"); - }); - tokio::time::sleep(Duration::from_millis(50)).await; - assert!(matches!(supervisor.task_status("panicker"), TaskStatus::Failed { .. })); -} - -#[tokio::test] -async fn test_supervisor_restarts_critical_task() { - let count = Arc::new(AtomicU32::new(0)); - let c = count.clone(); - - let supervisor = TaskSupervisor::new(); - supervisor.spawn_critical("restartable", move || { - let c = c.clone(); - async move { - let n = c.fetch_add(1, Ordering::SeqCst); - if n == 0 { panic!("first run fails"); } - // Second run: stay alive - loop { tokio::time::sleep(Duration::from_secs(60)).await; } - } - }); - - tokio::time::sleep(Duration::from_secs(2)).await; - assert_eq!(count.load(Ordering::SeqCst), 2); - assert!(matches!(supervisor.task_status("restartable"), TaskStatus::Running)); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-core/src/supervisor.rs` - -```rust -use parking_lot::RwLock; -use std::collections::HashMap; -use std::sync::Arc; -use std::time::{Duration, Instant}; -use tokio::task::JoinHandle; -use tracing::{error, info, warn}; - -pub struct TaskSupervisor { - tasks: Arc>>, -} - -struct TaskEntry { - handle: JoinHandle<()>, - status: TaskStatus, - restart_count: u32, - last_restart: Option, -} - -#[derive(Debug, Clone)] -pub enum TaskStatus { - Running, - Failed { error: String, at: Instant }, - Restarting { attempt: u32 }, - Stopped, -} - -impl TaskSupervisor { - pub fn new() -> Self { - Self { - tasks: Arc::new(RwLock::new(HashMap::new())), - } - } - - pub fn spawn_supervised(&self, name: &str, future: F) - where - F: std::future::Future + Send + 'static, - { - let tasks = self.tasks.clone(); - let name_owned = name.to_string(); - - let handle = tokio::spawn(async move { - future.await; - }); - - // Monitor the handle - let tasks_monitor = self.tasks.clone(); - let name_monitor = name.to_string(); - let monitor_handle = handle; - - self.tasks.write().insert( - name_owned, - TaskEntry { - handle: monitor_handle, - status: TaskStatus::Running, - restart_count: 0, - last_restart: None, - }, - ); - } - - pub fn task_status(&self, name: &str) -> TaskStatus { - let mut tasks = self.tasks.write(); - if let Some(entry) = tasks.get_mut(name) { - if entry.handle.is_finished() { - entry.status = TaskStatus::Failed { - error: "Task exited".into(), - at: Instant::now(), - }; - } - entry.status.clone() - } else { - TaskStatus::Stopped - } - } -} -``` - -**Note**: The full `spawn_critical` with automatic restart requires a task factory (`Fn() -> Future`) pattern. The supervisor spawns a monitor task that awaits the `JoinHandle`, and on failure, calls the factory again with exponential backoff (1s→5s→30s, max 5 restarts). This is the most complex piece — the detailed implementation is in the test code above. - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -- `PRAGMA integrity_check` is read-only — no risk to data -- sled repair may lose recently-written entries — acceptable for a cache -- tantivy rebuild deletes index entirely — no sensitive data exposure (metadata only) - -### 5.2 Observability - -- SQLite integrity check result logged at INFO (ok) or WARN (failed) -- sled repair attempts logged at WARN -- tantivy rebuild logged at WARN with file count before/after -- CAS `StoreFull` error logged at WARN with current/max sizes -- Task supervisor logs all state transitions (started, failed, restarting, stopped) - -### 5.3 Testing - -| Test | Status Before | Status After | Issue | -|------|---------------|--------------|-------| -| `test_cas_put_handles_enospc` | ❌ FAILED | ✅ GREEN | 2.8 | -| `test_sqlite_integrity_check_detects_corruption` | ❌ todo!() | ✅ GREEN | 2.4 | -| `test_tantivy_corruption_triggers_rebuild` | ❌ todo!() | ✅ GREEN | 2.4 | -| `test_tantivy_survives_uncommitted_crash` | ❌ todo!() | ✅ GREEN | 5.2 | -| `test_sled_corruption_triggers_repair` | ❌ todo!() | ✅ GREEN | 3.5 | -| `test_shutdown_cancels_background_tasks` | NEW | ✅ GREEN | 2.3 | -| `test_shutdown_flushes_tantivy` | NEW | ✅ GREEN | 2.3 | -| `test_supervisor_detects_panic` | NEW | ✅ GREEN | 2.6 | -| `test_supervisor_restarts_critical_task` | NEW | ✅ GREEN | 2.6 | - ---- - -## 6. Alternatives Considered - -### 6.1 Full `PRAGMA integrity_check` vs Quick Check - -`PRAGMA integrity_check` scans every page — slow for large DBs (seconds for 1M rows). `PRAGMA integrity_check(1)` stops after the first error — fast enough for startup. We use the quick variant. - -### 6.2 tantivy Repair vs Rebuild - -tantivy has no built-in repair. If `meta.json` is corrupt or segments are missing, the only option is delete + recreate. This is acceptable because the search index can be rebuilt from SQLite metadata (once persistent state is wired up). For now, rebuild produces an empty index. - -### 6.3 sled Repair vs Recreate - -sled has `Config::repair(true)` which attempts to recover. If repair fails, we delete and recreate. After recreation, the index is empty but chunk files still exist on disk — a future reconciliation pass can rebuild the index from chunk files (Phase F). - -### 6.4 Custom Supervisor vs `tokio-graceful` Crate - -`tokio-graceful` provides shutdown coordination but not task restart. Our needs are specific (restart with backoff, status reporting, critical vs non-critical distinction). A custom `TaskSupervisor` is simpler and avoids a dependency for ~100 lines of code. - ---- - -## 7. Implementation Plan - -### 7.1 Task Sequence - -| Day | Task | Issue | Effort | Test | -|-----|------|-------|--------|------| -| 1 (morning) | CAS size pre-check + `StoreFull` error variant | 2.8 | 1h | `test_cas_put_handles_enospc` → GREEN | -| 1 (afternoon) | SQLite `open_with_integrity_check` + `DatabaseCorrupted` error | 2.4 | 2h | `test_sqlite_integrity_check` → GREEN | -| 2 (morning) | tantivy `open_with_recovery` (detect + delete + recreate) | 2.4 | 2h | `test_tantivy_corruption` + `test_tantivy_survives_uncommitted_crash` → GREEN | -| 2 (afternoon) | sled recovery in `CasStore::open` (repair + fallback recreate) | 3.5 | 2h | `test_sled_corruption` → GREEN | -| 3 | Graceful shutdown with CancellationToken | 2.3 | 4h | `test_shutdown_cancels_background_tasks`, `test_shutdown_flushes_tantivy` → GREEN | -| 4 | Task supervisor implementation | 2.6 | 4h | `test_supervisor_detects_panic`, `test_supervisor_restarts` → GREEN | -| 5 | Integration + regression testing | — | 4h | Full `cargo test`, verify no regressions | - -### 7.2 Verification Checklist - -After all tasks: - -- [ ] `cargo check` — zero errors, zero warnings -- [ ] `cargo test --workspace --exclude musicfs-grpc` — all tests pass (exclude pre-existing grpc issue) -- [ ] `cargo test -p musicfs-test-utils --test resilience` — 5 previously-RED tests now GREEN -- [ ] `cargo clippy` — no new warnings -- [ ] Remaining RED tests are only for Phases C-F (health timeout, parallel checks, fd exhaustion, chunk auto-repair, passthrough mode) - ---- - -## 8. Files Changed - -| File | Change | Issue | -|------|--------|-------| -| `musicfs-cas/src/store.rs` | Size pre-check in `put()`, `StoreFull` error, sled recovery in `open()` | 2.8, 3.5 | -| `musicfs-cache/src/db.rs` | `open_with_integrity_check()` with `PRAGMA integrity_check(1)` | 2.4 | -| `musicfs-core/src/error.rs` | Add `DatabaseCorrupted(String)` variant | 2.4 | -| `musicfs-search/src/index.rs` | `open_with_recovery()` — detect, delete, recreate | 2.4 | -| `musicfs-core/src/supervisor.rs` | NEW — `TaskSupervisor`, `TaskStatus`, spawn/monitor/restart | 2.6 | -| `musicfs-core/src/lib.rs` | Re-export supervisor module | 2.6 | -| `musicfs-cli/src/main.rs` | CancellationToken creation, ordered shutdown sequence | 2.3 | -| `musicfs-cli/Cargo.toml` | Add `tokio-util` dependency | 2.3 | -| `musicfs-test-utils/tests/resilience.rs` | Replace `todo!()` stubs with real tests, add supervisor tests | all | - ---- - -## 9. Glossary / References - -| Term | Definition | -|------|------------| -| **CancellationToken** | `tokio_util::sync::CancellationToken` — cooperative cancellation signal for async tasks | -| **PRAGMA integrity_check** | SQLite command that verifies page-level data consistency | -| **sled repair** | sled's built-in recovery mode that attempts to reconstruct a corrupted database | -| **TaskSupervisor** | New struct that monitors `JoinHandle`s and restarts failed tasks with backoff | -| **StoreFull** | New `CasError` variant returned when a write would exceed `max_size` | - -| Document | Path | -|----------|------| -| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) | -| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) | -| Resilience testing | [resilience-testing.md](resilience-testing.md) | -| Persistent state | [persistent-state.md](persistent-state.md) | diff --git a/docs/v2/plans/phase-c-hardening.md b/docs/v2/plans/phase-c-hardening.md deleted file mode 100644 index c185ed3..0000000 --- a/docs/v2/plans/phase-c-hardening.md +++ /dev/null @@ -1,598 +0,0 @@ -# Phase C: Production Hardening — Implementation Plan - -**Authors:** AI-assisted -**Status:** Draft -**Last Updated:** 2026-05-13 -**Reviewers:** TBD -**Approvers:** TBD -**Prerequisites:** [phase-b-crash-recovery.md](phase-b-crash-recovery.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md) -**Estimated Effort:** ~4 days - ---- - -[TOC] - ---- - -## 1. Abstract - -Phase C merges the practical items from Phases C and D of the resilience audit into a single implementation pass. It fixes the remaining 6 RED tests and addresses production-critical issues: health check hangs that block all origin monitoring, unbounded FUSE reads that can freeze the filesystem, broken CAS size accounting that disables eviction, and concurrent mount protection. - -**Deferred items** (depend on unimplemented features or low urgency): interrupted sync recovery (needs persistent state), SIGHUP config reload, connection pooling (S3/SFTP are stubs), event bus backpressure, FUSE session recovery, offline mode state machine, DNS failure handling, stale-data awareness. - -**RED tests to turn GREEN:** -- `test_local_origin_health_check_has_timeout` (D1) -- `test_health_checks_run_in_parallel` (D2) -- `test_fd_exhaustion_handling` (E — 5.3) -- `test_corrupt_chunk_auto_refetched` (F — 6.4) -- `test_missing_chunk_triggers_origin_fetch` (F — 6.4) -- `test_passthrough_mode_when_cache_disk_dead` (F — 6.6) - ---- - -## 2. Background - -After Phase A+B, the daemon survives signals, recovers from storage corruption on startup, supervises background tasks, and rejects oversized CAS writes. But: - -1. **Health checks hang on dead origins** — `check_one()` calls `origin.health().await` with no timeout. A dead NAS (local origin pointing to network mount) blocks health monitoring for ALL origins because checks run sequentially. - -2. **FUSE reads have no timeout** — `reader.read()` in the FUSE `read()` callback has no timeout. A slow or hung origin blocks the FUSE thread indefinitely. - -3. **CAS size tracking is broken** — `calculate_size()` only scans top-level of `chunks_dir`, missing all chunks in shard subdirectories (`aa/bb/`). `current_size` is always ~0, eviction never triggers. - -4. **Corrupt chunks return EIO** — when `verify_integrity()` detects a bad chunk, it returns `CasError::IntegrityError`. The reader propagates this as EIO to FUSE. It should auto-re-fetch from origin instead. - -5. **No concurrent mount protection** — two `musicfs mount` commands can run simultaneously, corrupting SQLite and sled. - -6. **fd exhaustion is unhandled** — no graceful behavior when file descriptors run out. - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -- Health checks complete within 5 seconds regardless of origin responsiveness -- Health checks run in parallel (3 origins checked in ~5s, not ~15s) -- FUSE reads timeout after 30 seconds (returns EIO, doesn't hang) -- CAS size accounting is correct (recursive shard scan) -- Corrupt/missing chunks are auto-re-fetched from origin transparently -- PID file prevents concurrent mounts -- fd exhaustion produces clean errors, not panics -- All 6 remaining RED tests turn GREEN - -### 3.2 Non-Goals - -- Interrupted sync recovery (C1) — blocked on persistent state -- systemd watchdog (C3) — useful but not critical yet -- SIGHUP config reload (C4) — nice-to-have -- Connection pooling (C5) — S3/SFTP origins are stubs -- Event bus backpressure (C8) — low urgency -- FUSE session recovery (C10) — complex edge case -- Offline mode state machine (D3) — needs broader design -- DNS failure handling (D5) — depends on C5 -- Stale-data awareness (D6) — low severity for music FS - ---- - -## 4. Proposed Design - -### 4.1 Implementation Order - -``` -4.2 Health check timeout + parallel checks (2 RED tests, independent) - ↓ -4.3 Fix CAS calculate_size() (independent, unblocks eviction) - ↓ -4.4 FUSE read timeout (independent) - ↓ -4.5 CAS chunk auto-re-fetch on corruption (2 RED tests) - ↓ -4.6 PID file / flock (independent) - ↓ -4.7 fd exhaustion handling (1 RED test) -``` - -### 4.2 Issues D1+D2: Health Check Timeout + Parallel Checks - -**Problem**: `check_one()` awaits `origin.health()` with no timeout. `check_all()` iterates sequentially. One hung origin blocks everything. - -#### Step 1: No stubs needed - -#### Step 2: RED tests already exist - -`test_local_origin_health_check_has_timeout` — FaultyOrigin with `TimeoutMs(5000)`, asserts check completes in <2s. - -`test_health_checks_run_in_parallel` — 3 origins each with `TimeoutMs(200)`, asserts `check_all()` completes in <350ms (parallel), not ~600ms (sequential). - -#### Step 3: Implementation - -**File**: `musicfs-origins/src/health.rs` - -Wrap `origin.health()` in `check_one()` with timeout: - -```rust -async fn check_one(&self, id: &OriginId, origin: &Arc) { - let start = Instant::now(); - let health_timeout = Duration::from_secs(5); - - let status = match tokio::time::timeout(health_timeout, origin.health()).await { - Ok(status) => status, - Err(_) => { - warn!(origin_id = %id, timeout_ms = health_timeout.as_millis() as u64, - "Health check timed out"); - HealthStatus::Unhealthy - } - }; - - let latency_ms = start.elapsed().as_millis() as u64; - // ... rest unchanged -} -``` - -Change `check_all()` to use `futures::future::join_all`: - -```rust -pub async fn check_all(&self) { - let origins: Vec<_> = self.origins.iter() - .map(|e| (e.key().clone(), e.value().clone())) - .collect(); - - let checks: Vec<_> = origins.iter() - .map(|(id, origin)| self.check_one(id, origin)) - .collect(); - - futures::future::join_all(checks).await; -} -``` - -Add `futures` to `musicfs-origins/Cargo.toml` (or use `tokio::join!` macro if count is small/known). - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_local_origin_health_check -cargo test -p musicfs-test-utils --test resilience -- test_health_checks_run_in_parallel -``` - ---- - -### 4.3 Issue C6: Fix CAS calculate_size() - -**Problem**: `calculate_size()` only scans direct children of `chunks_dir`. Chunks live in shard subdirectories (`chunks/aa/bb/`). Size is always ~0, eviction never triggers. - -#### Step 1: No stubs needed - -#### Step 2: Test - -```rust -#[tokio::test] -async fn test_cas_size_tracking_is_correct() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { chunks_dir: dir.path().join("chunks"), max_size: 10_000_000, shard_levels: 2 }; - let store = CasStore::open(config).await.unwrap(); - - let data = vec![0u8; 1000]; - store.put(&data).await.unwrap(); - - // Size should reflect the chunk we just wrote (~1000 bytes) - assert!(store.current_size() >= 1000, "current_size should track chunk data, got {}", store.current_size()); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cas/src/store.rs` — make `calculate_size` recursive: - -```rust -async fn calculate_size(dir: &Path) -> u64 { - Self::calculate_size_recursive(dir).await -} - -#[async recursion::async_recursion] -async fn calculate_size_recursive(dir: &Path) -> u64 { - let mut size = 0u64; - if let Ok(mut entries) = fs::read_dir(dir).await { - while let Ok(Some(entry)) = entries.next_entry().await { - if let Ok(meta) = entry.metadata().await { - if meta.is_file() { - size += meta.len(); - } else if meta.is_dir() { - // Skip sled index directory - let name = entry.file_name(); - if name != "index.sled" { - size += Self::calculate_size_recursive(&entry.path()).await; - } - } - } - } - } - size -} -``` - -Alternative without `async_recursion` (use `Box::pin`): - -```rust -fn calculate_size_recursive(dir: &Path) -> Pin + Send + '_>> { - Box::pin(async move { - let mut size = 0u64; - if let Ok(mut entries) = fs::read_dir(dir).await { - while let Ok(Some(entry)) = entries.next_entry().await { - if let Ok(meta) = entry.metadata().await { - if meta.is_file() { - size += meta.len(); - } else if meta.is_dir() { - let name = entry.file_name(); - if name != "index.sled" { - size += Self::calculate_size_recursive(&entry.path()).await; - } - } - } - } - } - size - }) -} -``` - ---- - -### 4.4 Issue C7: FUSE Read Timeout - -**Problem**: FUSE `read()` calls `handle.block_on(reader.read(...))` with no timeout. A slow origin blocks the entire FUSE thread. - -#### Step 1: No stubs needed - -#### Step 2: Test - -```rust -#[tokio::test] -async fn test_fuse_read_timeout_returns_eio() { - // Uses FaultyOrigin with TimeoutMs(60_000) — simulates hung read - // FUSE read should timeout at 30s and return EIO, not hang forever - // (This test validates the timeout wrapper, not actual FUSE mount) -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-fuse/src/filesystem.rs` — wrap the read with timeout: - -```rust -fn read(&mut self, _req: &Request, ino: u64, _fh: u64, offset: i64, size: u32, _flags: i32, _lock_owner: Option, reply: ReplyData) { - // ... file_id lookup unchanged ... - - let reader = reader.clone(); - let handle = self.runtime_handle.clone(); - let result = std::thread::scope(|_| { - handle.block_on(async { - tokio::time::timeout( - Duration::from_secs(30), - reader.read(file_id, offset as u64, size), - ).await - }) - }); - - match result { - Ok(Ok(data)) => { - trace!(ino, bytes_read = data.len(), "read successful"); - reply.data(&data); - } - Ok(Err(e)) => { - warn!(ino, error = %e, "read failed"); - reply.error(libc::EIO); - } - Err(_timeout) => { - warn!(ino, offset, size, "read timed out after 30s"); - reply.error(libc::EIO); - } - } -} -``` - ---- - -### 4.5 Issues 6.4: CAS Chunk Auto-Re-Fetch on Corruption/Missing - -**Problem**: When `store.get()` finds a corrupt or missing chunk, it returns an error. The reader propagates this as EIO to FUSE. It should try to re-fetch the chunk from the origin instead. - -#### Step 1: No stubs needed — modify `FileReader::read()` - -#### Step 2: RED tests already exist - -`test_corrupt_chunk_auto_refetched` — corrupts chunk file on disk, expects read to succeed (re-fetched from origin). - -`test_missing_chunk_triggers_origin_fetch` — deletes chunk file, expects read to succeed. - -Both currently fail because the reader doesn't attempt re-fetch on chunk errors. - -#### Step 3: Implementation - -**File**: `musicfs-cas/src/reader.rs` — add retry-with-refetch in the chunk read loop: - -```rust -pub async fn read(&self, file_id: FileId, offset: u64, size: u32) -> Result { - let manifest = self.get_or_fetch_manifest(file_id).await?; - - // ... offset/end calculation unchanged ... - - for chunk_ref in &manifest.chunks { - // ... range check unchanged ... - - let chunk_data = match self.store.get(&chunk_ref.hash).await { - Ok(data) => data, - Err(CasError::IntegrityError { .. }) | Err(CasError::NotFound(_)) => { - // Chunk is corrupt or missing — try to re-fetch from origin - warn!(hash = %chunk_ref.hash, "Chunk corrupt/missing, attempting re-fetch"); - if let Some(fetcher) = &self.fetcher { - // Re-fetch the entire file (will re-chunk and store) - let new_manifest = fetcher.fetch_file(file_id).await?; - // Update cached manifest - self.manifests.write().insert(file_id, new_manifest); - // Retry the get - self.store.get(&chunk_ref.hash).await? - } else { - return Err(ReaderError::Cas(CasError::NotFound(chunk_ref.hash.as_hex()))); - } - } - Err(e) => return Err(ReaderError::Cas(e)), - }; - - // ... slice extraction unchanged ... - } - - Ok(result.freeze()) -} -``` - -**Important**: The re-fetch downloads the entire file from origin and re-chunks it. For a single corrupt chunk this is wasteful (fetches all chunks to fix one), but it's the simplest correct approach. Chunk-level re-fetch would require the origin to support byte-range reads mapped to chunk boundaries — possible but complex. The file-level approach reuses existing `fetch_file()` logic. - -#### Step 4: Verify - -```bash -cargo test -p musicfs-test-utils --test resilience -- test_corrupt_chunk -cargo test -p musicfs-test-utils --test resilience -- test_missing_chunk -``` - -**Note on test updates**: The existing RED tests reference `store.chunk_path()` which is private. The tests will need to either: -- Make `chunk_path()` pub(crate) or add a test helper -- Or construct the path manually using the sharding logic - -The tests also need a `ContentFetcher` with a real `LocalOrigin` to re-fetch from. The current tests create a CAS store but no fetcher — they need to be updated to include the full pipeline. - ---- - -### 4.6 Issue C9: PID File / flock - -**Problem**: Two `musicfs mount` commands can run simultaneously, both writing to the same SQLite/sled files. - -#### Step 1: No stubs needed - -#### Step 2: Test - -```rust -#[test] -fn test_pid_file_prevents_concurrent_mount() { - let dir = TempDir::new().unwrap(); - let lock_path = dir.path().join("musicfs.lock"); - - // First lock succeeds - let lock1 = try_acquire_lock(&lock_path); - assert!(lock1.is_ok()); - - // Second lock fails - let lock2 = try_acquire_lock(&lock_path); - assert!(lock2.is_err()); - - // Release first, second succeeds - drop(lock1); - let lock3 = try_acquire_lock(&lock_path); - assert!(lock3.is_ok()); -} -``` - -#### Step 3: Implementation - -**File**: `musicfs-cli/src/main.rs` - -```rust -use std::fs::File; -use std::os::unix::io::AsRawFd; - -struct LockFile { - _file: File, -} - -fn try_acquire_lock(path: &Path) -> Result { - let file = File::create(path).context("Failed to create lock file")?; - let fd = file.as_raw_fd(); - - let ret = unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) }; - if ret != 0 { - let err = std::io::Error::last_os_error(); - if err.kind() == std::io::ErrorKind::WouldBlock { - anyhow::bail!("MusicFS is already running (lock file: {:?})", path); - } - return Err(err).context("Failed to acquire lock"); - } - - // Write PID for debugging - use std::io::Write; - let mut f = &file; - writeln!(f, "{}", std::process::id())?; - - Ok(LockFile { _file: file }) -} -``` - -Call in `run_mount()` before mounting: - -```rust -let lock_path = cache_dir.join("musicfs.lock"); -let _lock = try_acquire_lock(&lock_path) - .context("Failed to acquire lock — is another instance running?")?; -``` - -Lock is released automatically when `_lock` is dropped (process exit or scope end). - ---- - -### 4.7 Issue 5.3: fd Exhaustion Handling - -**Problem**: When fd limit is hit, operations fail with EMFILE. Currently this propagates as panics or unhelpful errors. - -#### Step 1: Replace the `todo!()` test - -#### Step 2: Test - -```rust -#[test] -#[cfg(target_os = "linux")] -fn test_fd_exhaustion_handling() { - use rlimit::{Resource, setrlimit, getrlimit}; - - let (orig_soft, orig_hard) = getrlimit(Resource::NOFILE).unwrap(); - - // Set very low limit - setrlimit(Resource::NOFILE, 64, 64).unwrap(); - - let dir = TempDir::new().unwrap(); - let rt = tokio::runtime::Runtime::new().unwrap(); - - let result = rt.block_on(async { - CasStore::open(CasConfig { - chunks_dir: dir.path().join("chunks"), - max_size: 1_000_000, - shard_levels: 2, - }).await - }); - - // Should either succeed (sled uses fewer than 64 fds) or fail gracefully - // Must NOT panic - match result { - Ok(_store) => { /* lucky — enough fds */ } - Err(e) => { - // Error message should be meaningful - let msg = format!("{}", e); - assert!(!msg.contains("panic"), "Should not panic on fd exhaustion"); - } - } - - setrlimit(Resource::NOFILE, orig_soft, orig_hard).unwrap(); -} -``` - -#### Step 3: Implementation - -This is primarily a **test** — verifying that existing code handles fd exhaustion without panicking. The fix is ensuring all I/O paths return `Result` rather than `.unwrap()` on file operations. Phase A's RwLock migration already removed the biggest panic source. The remaining `.unwrap()` calls are in test code only. - -No production code change required if existing error paths handle I/O errors correctly. The test validates this. - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Observability - -- Health check timeout logged at WARN with origin_id and timeout duration -- FUSE read timeout logged at WARN with inode, offset, size -- CAS chunk re-fetch logged at WARN with chunk hash -- PID file path logged at INFO on lock acquisition - -### 5.2 Performance - -- Health checks now parallel: O(1) wall-clock time instead of O(N) per check cycle -- FUSE read timeout: 30s cap prevents indefinite hangs but doesn't improve happy-path latency -- `calculate_size()` recursive scan: runs once at startup, negligible cost - -### 5.3 Testing - -| Test | Status Before | Status After | Issue | -|------|---------------|--------------|-------| -| `test_local_origin_health_check_has_timeout` | ❌ FAILED | ✅ GREEN | D1 | -| `test_health_checks_run_in_parallel` | ❌ FAILED | ✅ GREEN | D2 | -| `test_fd_exhaustion_handling` | ❌ todo!() | ✅ GREEN | 5.3 | -| `test_corrupt_chunk_auto_refetched` | ❌ FAILED | ✅ GREEN | 6.4 | -| `test_missing_chunk_triggers_origin_fetch` | ❌ FAILED | ✅ GREEN | 6.4 | -| `test_passthrough_mode_when_cache_disk_dead` | ❌ todo!() | ✅ GREEN | 6.6 | -| `test_cas_size_tracking_is_correct` | NEW | ✅ GREEN | C6 | -| `test_pid_file_prevents_concurrent_mount` | NEW | ✅ GREEN | C9 | - -**Note on passthrough mode** (6.6): The test expects reads to succeed when the cache dir is read-only. With chunk auto-re-fetch (4.5), this partially works — if the origin is alive and the chunk isn't in cache, the fetcher reads from origin. But the fetcher tries to _write_ the chunk to CAS, which will fail on a read-only cache dir. The implementation needs a fallback path: if CAS write fails after origin fetch, return the data anyway without caching. This makes `test_passthrough_mode_when_cache_disk_dead` pass. - ---- - -## 6. Alternatives Considered - -### 6.1 Per-Origin Configurable Timeout vs Universal 5s - -Could allow `health_check_timeout_ms` per origin config. Rejected for Phase C — universal 5s is correct for all current origin types. Can be made configurable later. - -### 6.2 Chunk-Level Re-Fetch vs File-Level Re-Fetch - -When one chunk is corrupt, we could re-fetch just that chunk's byte range from origin. Requires the origin to support byte-range reads and the system to know which byte range maps to which chunk. Complex. File-level re-fetch reuses existing `fetch_file()` and is correct, just slightly wasteful. Good enough for Phase C. - -### 6.3 `advisory-lock` Crate vs Raw `flock` - -The `advisory-lock` crate wraps flock nicely but adds a dependency for 10 lines of code. Raw `libc::flock` is simple enough and avoids the dependency. - ---- - -## 7. Implementation Plan - -### 7.1 Task Sequence - -| Day | Task | Issue | Effort | Tests | -|-----|------|-------|--------|-------| -| 1 (morning) | Health check timeout in `check_one()` | D1 | 1h | `test_local_origin_health_check_has_timeout` → GREEN | -| 1 (morning) | Parallel `check_all()` with `join_all` | D2 | 1h | `test_health_checks_run_in_parallel` → GREEN | -| 1 (afternoon) | Fix `calculate_size()` recursion | C6 | 1h | `test_cas_size_tracking_is_correct` → GREEN | -| 1 (afternoon) | FUSE read timeout wrapper | C7 | 1h | New timeout test | -| 2 (morning) | CAS chunk auto-re-fetch on corruption/missing | 6.4 | 3h | `test_corrupt_chunk_auto_refetched` + `test_missing_chunk_triggers_origin_fetch` → GREEN | -| 2 (afternoon) | Passthrough fallback (CAS write fails → return data anyway) | 6.6 | 1h | `test_passthrough_mode_when_cache_disk_dead` → GREEN | -| 3 (morning) | PID file / flock | C9 | 1h | `test_pid_file_prevents_concurrent_mount` → GREEN | -| 3 (morning) | fd exhaustion test | 5.3 | 1h | `test_fd_exhaustion_handling` → GREEN | -| 3 (afternoon) | Integration + regression testing | — | 2h | Full `cargo test` | -| 4 | Buffer | — | 4h | — | - -### 7.2 Verification Checklist - -After all tasks: - -- [ ] `cargo check` — zero errors, zero warnings -- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass -- [ ] `cargo test -p musicfs-test-utils --test resilience` — **25 passed, 0 failed** (all RED tests GREEN) -- [ ] `cargo clippy` — no new warnings - ---- - -## 8. Files Changed - -| File | Change | Issue | -|------|--------|-------| -| `musicfs-origins/src/health.rs` | Timeout in `check_one()`, `join_all` in `check_all()` | D1, D2 | -| `musicfs-origins/Cargo.toml` | Add `futures` dependency (for `join_all`) | D2 | -| `musicfs-cas/src/store.rs` | Recursive `calculate_size()`, skip `index.sled` dir | C6 | -| `musicfs-fuse/src/filesystem.rs` | `tokio::time::timeout(30s)` around reader.read() | C7 | -| `musicfs-cas/src/reader.rs` | Auto-re-fetch on `IntegrityError` / `NotFound` | 6.4 | -| `musicfs-cas/src/fetcher.rs` | Possible: make `fetch_file` return data even if CAS write fails | 6.6 | -| `musicfs-cli/src/main.rs` | PID file with flock, fd exhaustion handling | C9, 5.3 | -| `musicfs-test-utils/tests/resilience.rs` | Replace remaining todo!()s, add new tests, update chunk tests with fetcher pipeline | all | - ---- - -## 9. Glossary / References - -| Term | Definition | -|------|------------| -| **join_all** | `futures::future::join_all` — runs multiple futures concurrently, waits for all | -| **flock** | Advisory file locking syscall — `LOCK_EX | LOCK_NB` for exclusive non-blocking | -| **EMFILE** | "Too many open files" errno — returned when process fd limit is reached | -| **Passthrough mode** | When CAS is unavailable, read directly from origin without caching | - -| Document | Path | -|----------|------| -| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) | -| Phase B plan | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) | -| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) | diff --git a/docs/v2/plans/resilience-fault-tolerance.md b/docs/v2/plans/resilience-fault-tolerance.md deleted file mode 100644 index 67a5d0d..0000000 --- a/docs/v2/plans/resilience-fault-tolerance.md +++ /dev/null @@ -1,1417 +0,0 @@ -# MusicFS Resilience & Fault Tolerance Plan - -**Date**: 2026-05-13 -**Status**: Research Complete — Ready for Implementation -**Prerequisites**: [architecture.md](../architecture.md), [requirements.md](../requirements.md) -**Related Requirements**: NFR-7 (Availability), NFR-8 (Data Integrity), FR-25 (Resilience) - ---- - -## 1. Audit Summary - -MusicFS is designed as a critical filesystem daemon. Like any Linux filesystem, it must never "just die" — it must survive crashes, network failures, disk pressure, and signal interrupts with a clear recovery path for every failure mode. - -### Current Resilience Posture - -**Working well:** -- Origin failover with retry (100ms→500ms→2s) via `FailoverExecutor` -- NFS stale handle retry (`retry_on_stale` in nfs.rs) -- SMB disconnect retry (`retry_on_disconnect` in smb.rs) -- Webhook delivery retries with configurable count -- Health monitoring with consecutive failure tracking and degraded state -- SQLite WAL mode (`PRAGMA journal_mode = WAL`) — crash-safe -- SQLite NORMAL sync (`PRAGMA synchronous = NORMAL`) — good perf/safety tradeoff -- FUSE operations return proper error codes (ENOENT/EROFS/EIO), never panic -- Broadcast lag handling (RecvError::Lagged) in server/webhook/indexer -- systemd restart on failure (`Restart=on-failure`, `RestartSec=5`) - -**Critical gaps identified:** 10 issues, detailed below. - ---- - -## 2. Critical Gaps - -### 2.1 No Signal Handling (SIGTERM/SIGINT/SIGHUP) - -**Location**: `musicfs-cli/src/main.rs` - -**Problem**: `main.rs` has no `tokio::signal::ctrl_c()` or `unix::signal(SIGTERM)`. The FUSE mount blocks the main thread — there's no way to trigger graceful cleanup. When systemd sends SIGTERM, the process just dies with no flush, no unmount, no DB sync. - -**Impact**: Corrupted tantivy index, orphaned FUSE mount (users see "Transport endpoint is not connected"), dirty cache state. - -**Required**: -- `tokio::signal::ctrl_c()` + `tokio::signal::unix::signal(SignalKind::terminate())` listener -- Signal triggers `CancellationToken` that propagates to all background tasks -- FUSE session unmount via `fuser::Session::unmount()` or `fusermount -u` -- Flush tantivy writer, close SQLite connections, stop health monitor/watcher/indexer -- Log clean shutdown with timing - -**Architecture ref**: FR-17.5 (graceful shutdown with drain), FR-1.4 (release all resources on unmount) - ---- - -### 2.2 No Panic Hook / catch_unwind - -**Location**: None (completely absent) - -**Problem**: No `std::panic::set_hook()` anywhere. A panic in any background task (health monitor, watcher, indexer) silently kills that task — the daemon continues in degraded state with zero notification. A panic in the FUSE thread kills the whole daemon instantly. - -**Impact**: Silent degradation or instant death with no diagnostic output. - -**Required**: -- Custom panic hook that logs the panic with `error!()` before default behavior -- Include thread name, backtrace, and panic payload in log -- For background tasks: `catch_unwind` wrapper that logs + triggers task restart -- For FUSE thread: panic hook should attempt emergency unmount before abort - ---- - -### 2.3 No Graceful Shutdown Orchestration - -**Location**: `musicfs-cli/src/main.rs`, all background task spawns - -**Problem**: -- `musicfs shutdown` CLI command prints "gRPC client integration pending" — it's a stub -- No `CancellationToken` or shutdown signal propagation to background tasks -- `WatchHandle::drop()` tries `try_send(())` but that's best-effort -- Health monitor, indexer, prefetcher loop forever with no shutdown path - -**Impact**: On shutdown, background tasks are killed mid-operation. Partial writes, corrupt indexes. - -**Required**: -- `tokio_util::sync::CancellationToken` shared across all components -- Each background task checks `token.cancelled()` in its loop -- Shutdown sequence with ordering: - 1. Stop accepting new FUSE operations (drain timeout from ShutdownRequest) - 2. Cancel background tasks (watcher → indexer → health monitor → prefetcher) - 3. Flush tantivy index writer - 4. Close SQLite connections (checkpoint WAL) - 5. Unmount FUSE - 6. Exit - ---- - -### 2.4 No Cache Integrity Validation on Startup - -**Location**: `musicfs-cache/src/db.rs` (`Database::open()`) - -**Problem**: Architecture requires "The system SHALL validate cache integrity on startup" (FR-25.5). Currently `Database::open()` just opens and runs schema — no integrity check. No `PRAGMA integrity_check`, no tantivy index validation, no CAS chunk verification. If the process was killed during a write, corrupt data silently persists. - -**Impact**: Corrupt metadata served to FUSE clients after crash recovery. - -**Required**: -- On startup: `PRAGMA integrity_check` on SQLite (quick mode for large DBs) -- Validate tantivy index can be opened and searched -- Spot-check random CAS chunks (verify hash matches content) -- If corruption detected: log warning, offer `--repair` mode -- Repair mode: rebuild tantivy index from SQLite, re-verify CAS chunks - ---- - -### 2.5 No Interrupted Sync Recovery - -**Location**: `musicfs-sync/src/delta.rs` - -**Problem**: Architecture requires "The system SHALL recover from interrupted synchronization" (NFR-8.3). `DeltaDetector::detect_changes()` has no checkpoint/resume mechanism. If killed during sync, next restart re-scans from scratch. No partial manifest tracking — partially cached files have no state. - -**Impact**: Wasted bandwidth, slow restart after crash during large sync. - -**Required**: -- Sync state table in SQLite: `sync_progress(origin_id, phase, last_path, files_processed, started_at)` -- Checkpoint after each batch of files processed -- On restart: check for incomplete sync, resume from last checkpoint -- Partial manifests: mark files as `sync_state = 'partial'` until all chunks cached -- On read of partial file: fetch remaining chunks on demand - ---- - -### 2.6 Spawned Tasks Are Fire-and-Forget - -**Location**: 13 `tokio::spawn()` calls across server.rs, search_service.rs, indexer.rs, health.rs, watcher.rs, prefetch.rs, artwork.rs - -**Problem**: None of the spawned tasks have their `JoinHandle` stored for monitoring. If health monitor panics → no failover, origins silently become "unknown". If watcher panics → no change detection, stale data forever. If indexer panics → search silently stops updating. - -**Impact**: Silent feature degradation, impossible to detect or recover. - -**Required**: -- `TaskSupervisor` struct that stores `JoinHandle<()>` for each critical task -- Periodic check (every 30s): is the task still running? -- If task died: log error, attempt restart with backoff -- Critical tasks (must restart): health monitor, file watcher, search indexer -- Non-critical tasks (log and continue): prefetcher, webhook sender -- Expose task health via gRPC `GetStatus()` response - ---- - -### 2.7 No FUSE Unmount on Crash - -**Location**: `dist/musicfs.service`, `musicfs-cli/src/main.rs` - -**Problem**: When the daemon dies, the FUSE mount becomes a dead mountpoint. `ls /mnt/music` hangs or returns "Transport endpoint is not connected". `ExecStop` calls `musicfs shutdown` which is a stub. No `fusermount -u` anywhere. - -**Impact**: Users must manually `fusermount -u /mnt/music` after every crash. - -**Required**: -- `ExecStopPost=/usr/bin/fusermount -u /mnt/music` in systemd service -- In signal handler: attempt `fuser::Session::unmount()` before exit -- On startup: check if mountpoint is already mounted (stale), auto-unmount if so -- Timeout on unmount attempt (5s), then force unmount - ---- - -### 2.8 No Disk Space Handling - -**Location**: `musicfs-cas/src/store.rs` - -**Problem**: CAS `put()` writes files with no check for ENOSPC. If cache disk fills up, chunk writes fail silently or crash. No emergency eviction, no watermark monitoring. - -**Impact**: Daemon crash or cache corruption when disk fills. - -**Required**: -- Check available disk space before CAS write -- High watermark (90% full): trigger aggressive LRU eviction -- Critical watermark (95% full): stop prefetching, evict aggressively -- Emergency (99% full): reject new cache writes, serve only cached data -- Periodic disk space monitoring (every 60s) with metric export -- `statvfs()` for disk checks — cheap syscall - ---- - -### 2.9 `.unwrap()` on RwLock in Production FUSE Paths - -**Location**: `musicfs-fuse/src/filesystem.rs` (every FUSE operation), `musicfs-cas/src/reader.rs`, `musicfs-origins/src/registry.rs` - -**Problem**: `self.tree.read().unwrap()` appears in every FUSE operation (lookup, getattr, readdir, open, read). `self.manifests.write().unwrap()` in ContentReader. `self.origins.read().unwrap()` in OriginRegistry. If any writer panics while holding a write lock, **every** subsequent FUSE operation panics → instant daemon death. - -**Impact**: Single poisoned RwLock = total daemon crash. This is the #1 single-point-of-failure. - -**Required**: -- Replace `.unwrap()` with `.read().unwrap_or_else(|poisoned| poisoned.into_inner())` for read locks (safe: readers don't mutate) -- For write locks: log error + return EIO to FUSE caller -- Alternative: use `parking_lot::RwLock` which doesn't poison on panic -- Audit all 30+ `.unwrap()` calls on locks in production paths - ---- - -### 2.10 No sd_notify Integration - -**Location**: `musicfs-cli/src/main.rs`, `dist/musicfs.service` - -**Problem**: systemd service has `Type=notify` but no code calls `sd_notify(READY=1)`. systemd will think the service never started and kill it after `TimeoutStartSec` (default 90s). - -**Impact**: Service fails to start under systemd. - -**Required**: -- Add `sd-notify` crate dependency -- Call `sd_notify::notify(false, &[NotifyState::Ready])` after FUSE mount succeeds -- Call `sd_notify::notify(false, &[NotifyState::Stopping])` on shutdown -- Call `sd_notify::notify(false, &[NotifyState::Status("Serving N files")])` periodically -- If `WatchdogSec` configured: periodic `sd_notify::notify(false, &[NotifyState::Watchdog])` - ---- - -## 3. Medium Gaps - -### 3.1 No systemd Watchdog Integration - -**Priority**: Medium -**Location**: `dist/musicfs.service`, `musicfs-cli/src/main.rs` - -**Problem**: The systemd service has no `WatchdogSec` directive and no code sends periodic keepalive pings. systemd has a built-in watchdog mechanism: if a service declares `WatchdogSec=30s`, systemd expects `sd_notify(WATCHDOG=1)` every 15 seconds (half the interval). If the daemon hangs (deadlock, infinite loop, blocked on I/O), systemd detects the silence and restarts it. - -Currently, if MusicFS deadlocks (e.g., a poisoned RwLock cascading, a stuck `block_on()` in the FUSE thread, or a sled compaction blocking the tokio runtime), the process stays alive but completely unresponsive. Users see hung `ls` commands, and systemd thinks everything is fine because the process PID still exists. - -**Current code**: No watchdog-related code exists anywhere. The systemd unit has `Restart=on-failure` but that only triggers on process death, not hangs. - -**Impact**: Daemon can hang indefinitely with no automatic recovery. Users must manually `kill -9` the process. - -**Required**: -- Add `WatchdogSec=30s` to `dist/musicfs.service` -- Spawn a dedicated watchdog task in `main.rs` that sends `sd_notify(WATCHDOG=1)` every 15s -- The watchdog task should also perform a lightweight health check before sending: - - Can we acquire a read lock on the virtual tree? (proves FUSE path isn't deadlocked) - - Is the tokio runtime responsive? (proves async tasks can run) - - Are critical background tasks still alive? (proves supervisor is working) -- If any check fails: log error, skip the watchdog ping → systemd kills and restarts -- Depends on: sd_notify integration (2.10), task supervisor (2.6) - -**Architecture ref**: NFR-10.3 (health check endpoint/signal) - -**Files**: `dist/musicfs.service`, `musicfs-cli/src/main.rs` - ---- - -### 3.2 No Connection Pooling for Remote Origins - -**Priority**: Medium -**Location**: `musicfs-origins/src/s3.rs`, `musicfs-origins/src/sftp.rs` - -**Problem**: S3 and SFTP origins are currently feature-gated stubs. The SFTP stub comments explicitly note "Use deadpool connection pool, not `Arc>`" as an Oracle fix. When these origins are implemented, each read operation will establish a new connection — SSH handshake (SFTP) or HTTPS/TLS negotiation (S3). For a music player seeking through a file, this means dozens of connections per second. - -**Current code**: -- `s3.rs`: 51-line stub with commented implementation showing raw per-request `get_object()` calls -- `sftp.rs`: 12-line stub noting `deadpool` connection pool requirement -- No connection pool crate in workspace `Cargo.toml` -- NFS and SMB origins delegate to local filesystem operations (no network connection to pool) - -**Impact**: -- SFTP: Each `read()` = SSH handshake (200-500ms). Seeking in a file = unusable latency. -- S3: AWS SDK has internal connection pooling via hyper, but without explicit pool management, connection limits aren't enforced and idle connections aren't cleaned up. -- Under load (10+ concurrent readers from remote origins), connection exhaustion is likely. - -**Required**: -- SFTP: Use `deadpool-russh` or custom pool with `deadpool` generic pool - - Pool size: configurable, default 4 per origin - - Connection health check: send keepalive before returning from pool - - Idle timeout: close connections idle >60s - - Connection recovery: if SSH session drops, create new session transparently -- S3: Configure hyper connection pool settings explicitly - - `pool_max_idle_per_host`: 4 (default is unlimited) - - `pool_idle_timeout`: 90s - - Request timeout: 30s (as noted in Oracle fixes in s3.rs comments) -- All remote origins: wrap operations with `tokio::time::timeout(30s)` to prevent hung connections from blocking indefinitely -- Add `deadpool` to workspace dependencies - -**Architecture ref**: NFR-6.2 (connection pooling for remote origins) - -**Files**: `musicfs-origins/src/sftp.rs`, `musicfs-origins/src/s3.rs`, `musicfs-origins/Cargo.toml` - ---- - -### 3.3 No Backpressure on Event Bus - -**Priority**: Low -**Location**: `musicfs-core/src/events.rs` - -**Problem**: The `EventBus` uses `tokio::sync::broadcast::channel(1024)`. When the channel is full (1024 events buffered), `broadcast` silently drops the oldest events for slow receivers. The current `publish()` method only logs when there are zero receivers — it has no detection for slow-receiver drops. - -On the receiver side, we recently added `RecvError::Lagged(n)` handling in server.rs, webhook.rs, and indexer.rs — those log a warning and continue. But the publish side has no awareness that events are being dropped. - -**Current code** (`events.rs`): -```rust -pub fn publish(&self, event: Event) { - trace!(event = ?event, "Publishing event"); - let receiver_count = self.sender.receiver_count(); - if self.sender.send(event).is_err() && receiver_count > 0 { - debug!(receiver_count = receiver_count, "Event dropped, no active receivers"); - } -} -``` - -The `send()` return value is `Result` — the `Err` case means zero receivers. It does NOT indicate channel-full drops. Those happen silently on the receiver side via `Lagged`. - -**Impact**: -- During heavy file change events (large origin rescan), the watcher may publish faster than the indexer can consume -- Search index falls behind, webhook notifications are missed, gRPC event streams have gaps -- No metric to detect this is happening — silent data loss - -**Required**: -- Add a metric counter: `musicfs_events_lagged_total` incremented each time a receiver sees `Lagged(n)`, with the lag count added -- Add channel capacity to `GetStatus()` response so operators can tune it -- Make channel capacity configurable (currently hardcoded 1024 in `Default::default()`) -- Consider: if lag count exceeds threshold (e.g., 1000 events in 60s), temporarily pause the publisher (watcher) to let consumers catch up -- Alternative: switch to `tokio::sync::mpsc` per-subscriber with bounded channels and explicit backpressure (more complex but no silent drops) - -**Architecture ref**: FR-18.1 (emit events for file access — events must not be silently lost) - -**Files**: `musicfs-core/src/events.rs`, `musicfs-core/src/config.rs` (add event_bus_capacity config) - ---- - -### 3.4 No FUSE Session Recovery - -**Priority**: Low -**Location**: `musicfs-fuse/src/filesystem.rs` - -**Problem**: The FUSE mount is established via `fuser::mount2(self, mountpoint, &options)` which blocks the calling thread until the filesystem is unmounted. If the FUSE kernel module encounters issues (e.g., kernel memory pressure, `/dev/fuse` fd becomes invalid, or the FUSE connection is interrupted by a kernel update), the mount becomes unusable with no recovery path. - -**Current code** (`filesystem.rs`): -```rust -pub fn mount(self, mountpoint: &Path) -> musicfs_core::Result<()> { - let options = vec![ - fuser::MountOption::RO, - fuser::MountOption::FSName("musicfs".to_string()), - fuser::MountOption::AutoUnmount, - fuser::MountOption::AllowOther, - ]; - fuser::mount2(self, mountpoint, &options).map_err(musicfs_core::Error::Io)?; - Ok(()) -} -``` - -Note: `MountOption::AutoUnmount` is set, which means the kernel will auto-unmount if the process dies. But this doesn't help with: -- FUSE connection interruption while process is alive -- Kernel-side FUSE abort (e.g., `echo 1 > /sys/fs/fuse/connections//abort`) -- `/dev/fuse` errors during memory pressure - -**Impact**: If the FUSE connection drops, the daemon is alive but the mount is dead. No recovery without full restart. - -**Required**: -- Detect FUSE connection loss (the `mount2` call returns with an error or the `destroy()` callback is invoked unexpectedly) -- On unexpected FUSE disconnect: log error, attempt remount after brief delay -- Maximum remount attempts: 3, with exponential backoff (1s, 5s, 15s) -- If remount fails: log critical error, trigger clean shutdown -- Consider using `fuser::spawn_mount2()` instead of `mount2()` — returns a `BackgroundSession` that can be monitored and re-established -- Note: remounting requires rebuilding the virtual tree (or keeping it alive separately from the FUSE session) - -**Architecture ref**: FR-1.1 (mount as FUSE filesystem), NFR-7.2 (graceful degradation) - -**Files**: `musicfs-fuse/src/filesystem.rs`, `musicfs-cli/src/main.rs` - ---- - -### 3.5 sled Crash Recovery Not Verified - -**Priority**: Medium -**Location**: `musicfs-cas/src/store.rs` - -**Problem**: sled (used as CAS chunk index mapping `ChunkHash → ChunkLocation`) has built-in crash recovery — it uses a log-structured merge tree with write-ahead logging. However, MusicFS never verifies that sled's recovery succeeded or that the index is consistent with the actual chunk files on disk. - -**Current code** (`store.rs`): -```rust -pub async fn open(config: CasConfig) -> Result { - fs::create_dir_all(&config.chunks_dir).await?; - let index_path = config.chunks_dir.join("index.sled"); - let index = sled::open(&index_path)?; // No recovery verification - let current_size = Self::calculate_size(&config.chunks_dir).await; - // ... -} -``` - -Failure scenarios: -1. **Chunk file written, sled index not updated** (crash between fs::write and index.insert): Orphaned chunk file on disk, invisible to the system. Wastes disk space. -2. **sled index updated, chunk file not written** (crash between index.insert and fs::write — unlikely due to ordering but possible with async I/O): Index points to nonexistent chunk. `get()` will fail with `CasError::NotFound`. -3. **sled corruption**: sled can fail to open with `sled::Error::Corruption`. Currently this propagates as `CasError::Sled` and crashes the daemon. -4. **Size accounting drift**: `current_size` is calculated by scanning files in `chunks_dir`, but only at the top level (`read_dir` without recursion). Since chunks are stored in sharded subdirectories (e.g., `aa/bb/aabb...`), `calculate_size()` misses all chunk files. The `current_size` is always ~0. - -**Specific bug found**: `calculate_size()` only counts files directly in `chunks_dir`, but chunks are stored in `chunks_dir/aa/bb/` (2 levels deep per `shard_levels: 2`). This means `current_size` is always wrong, and cache size enforcement/eviction never works correctly. - -**Impact**: -- Cache size tracking is broken (always reports ~0 bytes) -- Eviction never triggers (cache grows unbounded until disk fills) -- After crash: orphaned chunks waste disk, missing chunks cause read errors -- sled corruption = daemon won't start - -**Required**: -- Fix `calculate_size()`: recursively scan shard directories, or compute size from sled index entries -- On startup: verify sled opens cleanly; if `sled::Error::Corruption`, attempt `sled::Config::repair()` -- Consistency check (optional, `--verify-cas` flag): - - For each entry in sled index: verify chunk file exists on disk - - For each chunk file on disk: verify entry exists in sled index - - Report orphaned files and missing chunks - - Option to auto-repair: delete orphaned files, remove dangling index entries -- Consider atomic write pattern: write chunk to `.tmp` file, `rename()` to final path, then update sled index — `rename()` is atomic on Linux - -**Architecture ref**: NFR-8.1 (verify chunk integrity via checksums), NFR-8.4 (detect and report cache corruption) - -**Files**: `musicfs-cas/src/store.rs` - ---- - -### 3.6 No Config Reload (SIGHUP) - -**Priority**: Low -**Location**: `musicfs-core/src/config.rs`, `musicfs-cli/src/main.rs` - -**Problem**: The architecture requires "The system SHALL support runtime configuration changes" (FR-17.4). Currently, configuration is loaded once at startup from TOML file and CLI args. There is no way to change configuration without restarting the daemon. - -**Current code**: `Config` struct in `config.rs` has fields for origins, cache, logging, template, search, and prefetch. All are set once. The prefetch engine has `update_config()` method but it's never called at runtime. - -Common operations that should be hot-reloadable: -- Adding/removing origins (e.g., plugging in an external drive) -- Changing cache size limits -- Adjusting log level -- Enabling/disabling prefetching -- Updating path template - -Operations that require restart: -- Mount point change -- FUSE options -- gRPC socket path - -**Impact**: Any config change requires daemon restart → FUSE unmount → all in-flight reads fail → media players stop playback. - -**Required**: -- Register SIGHUP handler via `tokio::signal::unix::signal(SignalKind::hangup())` -- On SIGHUP: re-read config file, diff against current config -- Hot-reloadable fields: log level, cache limits, prefetch config, origin list -- Cold fields (require restart): mount point, FUSE options, socket path -- Emit `ConfigReloaded` event on successful reload -- Log what changed: `info!(changed_fields = ?diff, "Configuration reloaded")` -- Validate new config before applying (don't break on invalid TOML) -- Expose via gRPC: `ReloadConfig()` RPC for programmatic reload - -**Architecture ref**: FR-17.4 (runtime configuration changes) - -**Files**: `musicfs-core/src/config.rs`, `musicfs-cli/src/main.rs`, `musicfs-grpc/src/server.rs` - ---- - -### 3.7 ExecStop Is a Stub - -**Priority**: High (but overlaps with 2.3 and 2.7) -**Location**: `dist/musicfs.service`, `musicfs-cli/src/main.rs` - -**Problem**: The systemd service has `ExecStop=/usr/bin/musicfs shutdown` but the `run_shutdown()` function in `main.rs` just prints a message and exits: - -```rust -fn run_shutdown(graceful: bool, timeout: u32) -> Result<()> { - println!("Shutdown requested (graceful: {}, timeout: {}s)", graceful, timeout); - println!("gRPC client integration pending"); - Ok(()) -} -``` - -When systemd stops the service, `ExecStop` runs first. Since it does nothing, systemd then sends SIGTERM. Since there's no signal handler (gap 2.1), the daemon is killed immediately. This is a triple failure: ExecStop is a no-op, SIGTERM has no handler, and there's no ExecStopPost to clean up. - -**Current flow on `systemctl stop musicfs`**: -1. systemd runs `ExecStop=/usr/bin/musicfs shutdown` → prints message, exits 0 -2. systemd sends SIGTERM to main daemon PID → daemon dies instantly -3. FUSE mount becomes stale (no `fusermount -u`) -4. No WAL checkpoint, no tantivy flush, no sled flush - -**Impact**: Every `systemctl stop musicfs` leaves a stale mount and risks data corruption. - -**Required** (this is solved by combining fixes from 2.1, 2.3, and 2.7): -- Short term: Change `ExecStop` to `ExecStop=/usr/bin/fusermount -u /mnt/music` (at least unmounts cleanly) -- Add `ExecStopPost=/usr/bin/fusermount -uz /mnt/music` as safety net (lazy unmount, always succeeds) -- Medium term: Implement gRPC shutdown RPC, make `musicfs shutdown` actually connect and send `ShutdownRequest` -- Long term: Signal handler catches SIGTERM, initiates graceful shutdown sequence, `ExecStop` becomes optional - -**Architecture ref**: FR-17.5 (graceful shutdown with drain), FR-1.4 (release all resources on unmount) - -**Files**: `dist/musicfs.service`, `musicfs-cli/src/main.rs`, `musicfs-grpc/src/server.rs` - ---- - -### 3.8 No Timeout on Origin Operations in FUSE Path - -**Priority**: Medium -**Location**: `musicfs-fuse/src/filesystem.rs`, `musicfs-cas/src/fetcher.rs` - -**Problem**: When a FUSE `read()` triggers a cache miss, the request flows through `FileReader → ContentFetcher → Origin.read()`. For remote origins, this can block for an unbounded duration — the failover retry config has delays (100ms, 500ms, 2s) but no overall timeout. If an origin is responding but extremely slowly (trickle attack, network congestion), all 3 retry attempts could each take minutes. - -**Current code** (`filesystem.rs` read path): -```rust -let result = std::thread::scope(|_| { - handle.block_on(async { - reader.read(file_id, offset as u64, size).await - }) -}); -``` - -This `block_on` has no timeout. A hung origin blocks the FUSE thread. Since fuser processes FUSE requests sequentially (single-threaded filesystem impl), one hung read blocks ALL FUSE operations — `ls`, `stat`, everything. - -**Impact**: One slow origin request can freeze the entire filesystem for all users. Media players hang, file managers become unresponsive, and the daemon appears dead even though it's technically alive. - -**Required**: -- Wrap the FUSE read path with `tokio::time::timeout(Duration::from_secs(30), reader.read(...))` -- On timeout: return `EIO` to FUSE, log warning with origin and path -- Add per-origin timeout configuration (local: 5s, remote: 30s) -- The S3 origin stub already notes this requirement: "Wrap all remote calls with `tokio::time::timeout(30s)`" -- Consider: FUSE `read()` has a kernel-side timeout too (usually 30s), but relying on kernel timeout gives poor error messages - -**Architecture ref**: NFR-1.6 (read cache miss remote: max 1000ms — current code has no enforcement) - -**Files**: `musicfs-fuse/src/filesystem.rs`, `musicfs-cas/src/fetcher.rs`, `musicfs-cas/src/reader.rs` - ---- - -### 3.9 No Protection Against Concurrent Mount Attempts - -**Priority**: Low -**Location**: `musicfs-cli/src/main.rs` - -**Problem**: Nothing prevents two instances of `musicfs mount /mnt/music` from running simultaneously. The second instance would try to mount on the same mountpoint, potentially succeeding (FUSE allows it on some kernels) or failing with confusing errors. Two daemons writing to the same SQLite database and sled index would cause corruption. - -**Current code**: No PID file, no flock, no socket check. - -**Impact**: Accidental double-start corrupts cache databases. - -**Required**: -- Create a lock file at `{cache_dir}/musicfs.lock` using `flock(LOCK_EX | LOCK_NB)` -- If lock fails: print "MusicFS is already running (PID: N)" and exit 1 -- Write current PID to lock file for debugging -- Lock is automatically released on process death (kernel flock semantics) -- Alternative: check if gRPC socket exists and is responsive before mounting - -**Files**: `musicfs-cli/src/main.rs` - ---- - -### 3.10 Eviction System Has Broken Size Accounting - -**Priority**: Medium (closely related to 3.5) -**Location**: `musicfs-cache/src/eviction.rs`, `musicfs-cas/src/store.rs` - -**Problem**: The LRU eviction system depends on `CasStore::current_size()` to know when to evict. But as identified in 3.5, `calculate_size()` only scans the top level of `chunks_dir`, missing all actual chunks stored in shard subdirectories. The `current_size` is effectively always ~0. - -Additionally, the eviction system operates in-memory only — `LruEviction` stores access times in a `BTreeMap`. On daemon restart, all access history is lost. Every chunk has equal eviction priority, and the most recently accessed (hot) chunks are just as likely to be evicted as cold ones. - -**Current code** (`store.rs`): -```rust -async fn calculate_size(dir: &Path) -> u64 { - let mut size = 0u64; - if let Ok(mut entries) = fs::read_dir(dir).await { - while let Ok(Some(entry)) = entries.next_entry().await { - if let Ok(meta) = entry.metadata().await { - if meta.is_file() { - size += meta.len(); - } - } - } - } - size -} -``` - -This only reads direct children of `chunks_dir`. Chunks are stored as `chunks_dir/aa/bb/` (2 shard levels deep). So `calculate_size` returns the size of `index.sled` files at best. - -**Impact**: -- Cache grows unbounded — eviction never triggers because size appears to be ~0 -- Disk fills up → CAS writes fail → FUSE read errors -- After restart, hot data has no protection from eviction - -**Required**: -- Fix `calculate_size()`: recursive walk through shard directories, or calculate from sled index (sum of all `ChunkLocation.size`) -- Better: maintain size atomically during put/delete (current code does `fetch_add`/`fetch_sub` but seeds from broken `calculate_size`) -- Persist access times: add `last_accessed` column to sled index entries, or a separate SQLite table -- On startup: reconstruct LRU order from persisted access times -- Trigger eviction proactively: when `current_size > 0.9 * max_size`, start background eviction - -**Files**: `musicfs-cas/src/store.rs`, `musicfs-cache/src/eviction.rs` - ---- - -## 4. Network Fault Tolerance Analysis - -### 4.1 Scenario: Source Machine Dies - -**Full failure chain analysis:** - -When the machine hosting origin storage (NFS server, SMB share, S3 bucket, SFTP host) dies: - -| Phase | Timing | Current Behavior | Gap? | -|-------|--------|-----------------|------| -| **Immediate** (0-5s) | First read attempt | `Origin.read()` hangs or returns error | ⚠️ No timeout on FUSE read path (gap 3.8) | -| **Detection** (5-90s) | Health check cycle | NFS/SMB: 5s timeout per check, threshold=3 → marked Unhealthy after 3 intervals | ✅ Works | -| **Failover** (0-3s) | On next read | FailoverExecutor tries next origin, retries 100ms→500ms→2s | ✅ Works | -| **Degraded mode** | Ongoing | Cache-first: CAS serves cached chunks, ENOENT for uncached | ⚠️ Partial | -| **Recovery** | Origin comes back | Health monitor detects healthy, router re-enables | ✅ Works | - -**What works well:** -- **Multi-origin failover**: `FailoverExecutor.read_with_failover()` iterates all origins by priority, falls through on failure -- **Health monitoring**: `HealthMonitor.check_one()` with 5s timeout on health checks, consecutive failure tracking, per-type thresholds (Local=1, Remote=3) -- **Graceful degradation**: `Router.select_with_fallback()` falls through Healthy→Degraded→least-bad Unhealthy -- **Event notification**: `AllOriginsUnhealthy` event emitted when all origins are down, `OriginHealthChanged` on transitions -- **NFS-specific**: `retry_on_stale()` handles ESTALE (stale NFS file handle) with retry -- **SMB-specific**: `retry_on_disconnect()` handles ENOTCONN (SMB session drop) with retry - -### 4.2 Network-Specific Gaps Not Yet Covered - -#### 4.2.1 No Health Check Timeout on Local Origin - -**Priority**: Medium -**Location**: `musicfs-origins/src/local.rs` - -**Problem**: Local origin health check uses `fs::try_exists(&self.root)` with NO timeout. If the local path is actually an NFS/CIFS automount (common in NAS setups), this can hang indefinitely when the remote server dies. NFS and SMB origins wrap their health checks in `tokio::time::timeout(5s)` — local origin does not. - -**Current code**: -```rust -// local.rs - NO timeout -async fn health(&self) -> HealthStatus { - match fs::try_exists(&self.root).await { - Ok(true) => HealthStatus::Healthy, - Ok(false) => HealthStatus::Unhealthy, - Err(_) => HealthStatus::Unhealthy, - } -} - -// nfs.rs - HAS 5s timeout -async fn health(&self) -> HealthStatus { - let health_timeout = Duration::from_secs(5); - match tokio::time::timeout(health_timeout, self.inner.stat(Path::new("/"))).await { - Ok(Ok(_)) => HealthStatus::Healthy, - Ok(Err(_)) | Err(_) => HealthStatus::Unhealthy, - } -} -``` - -**Impact**: If a "local" origin is actually a mounted network share (extremely common — `/mnt/nas/music`), the health check hangs forever when the NAS dies. The health monitor task blocks on this one check and can't check any other origins either (checks are sequential in `check_all()`). - -**Required**: -- Add timeout to local origin health check: `tokio::time::timeout(Duration::from_secs(5), fs::try_exists(...))` -- Better: move the timeout into `HealthMonitor.check_one()` so ALL origin types get a universal timeout regardless of their implementation -- Make health checks parallel (currently sequential `for origin in origins { check_one(...).await }`) - ---- - -#### 4.2.2 Sequential Health Checks Block on Dead Origins - -**Priority**: Medium -**Location**: `musicfs-origins/src/health.rs` - -**Problem**: `check_all()` checks origins sequentially: -```rust -async fn check_all(&self) { - let origins: Vec<_> = self.origins.iter() - .map(|e| (e.key().clone(), e.value().clone())) - .collect(); - for (id, origin) in origins { - self.check_one(&id, &origin).await; // Sequential! - } -} -``` - -If 3 origins are configured and the first one's health check hangs (network timeout), the other 2 origins won't be checked until the first one finishes/times out. With a 5s timeout per check and 3 origins, a single dead origin delays all health updates by 5s. - -**Impact**: Health detection for all origins is delayed by the slowest (dead) origin. With check_interval=30s and 3 origins, worst case: healthy origin shows stale state for 30s + 5s×3 = 45s. - -**Required**: -- Change `check_all()` to use `futures::future::join_all()` or `tokio::spawn` per origin -- Each check runs concurrently with its own timeout -- All origins checked within one timeout window (5s), not N×5s - ---- - -#### 4.2.3 No "Offline Mode" State Machine - -**Priority**: Medium -**Location**: Architecture gap (no current implementation) - -**Problem**: When ALL origins are down and all cached data has been served, the daemon has no concept of "offline mode". It continues trying reads, getting errors, retrying — wasting resources. There's no: -- Backoff on health checks when all origins are down (still checks every 30s) -- User-visible state indicating "all origins offline, serving from cache only" -- Suppression of repeated error logs (every failed read logs warnings) -- Proactive notification that data may be stale - -The gRPC `MountState` enum already has `MOUNT_STATE_DEGRADED` ("Some origins unavailable") but no code sets it. - -**Impact**: Log spam during extended outage, wasted CPU on retries, no clear signal to monitoring systems. - -**Required**: -- Track global mount state: Ready → Degraded (some origins down) → Offline (all origins down) -- In Offline mode: increase health check interval to 60s (reduce wasted probes) -- In Offline mode: suppress per-read error logging, emit periodic summary instead ("Still offline, N reads served from cache in last 60s, M reads failed") -- Set `MountState` in gRPC `StatusResponse` based on health snapshot -- Emit event: `MountStateChanged { from, to }` for monitoring integration -- When first origin recovers: log "Origin {id} recovered, exiting offline mode", trigger immediate sync to catch up - ---- - -#### 4.2.4 No Automatic Origin Re-registration After Transient Failure - -**Priority**: Low -**Location**: `musicfs-origins/src/registry.rs`, `musicfs-sync/src/watcher.rs` - -**Problem**: When a local origin's directory is temporarily unavailable (NAS reboot, USB drive unmounted briefly), the inotify watcher in `watcher.rs` may die with an error. The `OriginWatcher` logs the error and the task ends: -```rust -if let Err(e) = Self::watch_loop(&origin_id, &root, &event_bus, &mut stop_rx).await { - error!("Watcher error: {}", e); -} -// Task exits silently — no restart -``` - -When the origin comes back, the watcher is dead. No new file change events are detected until the daemon is restarted. - -Similarly, if an NFS mount is re-established, the watcher created with the old file descriptors won't work on the new mount. - -**Impact**: After origin recovery, file changes are not detected. Users see stale data until manual restart. - -**Required**: -- Watcher should auto-restart on failure (ties into task supervisor from gap 2.6) -- On origin health transition Unhealthy→Healthy: restart watcher for that origin -- On watcher failure: retry with backoff (1s, 5s, 30s), max 5 attempts -- Log state: "Watcher for origin {id} failed, will retry in {delay}s" - ---- - -#### 4.2.5 No DNS Resolution Failure Handling - -**Priority**: Low -**Location**: Future S3/SFTP implementations - -**Problem**: Remote origins (S3, SFTP) depend on DNS resolution. DNS failures are a common transient network issue. The health check may fail not because the origin is down, but because DNS is temporarily unavailable. Currently, DNS failure = origin marked Unhealthy with same threshold as actual origin death. - -**Impact**: Transient DNS glitch causes unnecessary failover, cache misses, and degraded experience for 90+ seconds (3 failures × 30s check interval). - -**Required**: -- Distinguish DNS errors from connection errors in health checks -- DNS failure → Degraded (not Unhealthy), with faster re-check (5s instead of 30s) -- Cache DNS results internally (TTL 60s) to survive brief DNS outages -- Log DNS failures separately: `warn!("DNS resolution failed for {origin}, using cached IP")` -- Note: NFS and SMB origins (mounted locally) don't have this issue — DNS is resolved at mount time by the kernel - ---- - -#### 4.2.6 No Network Partition Detection (Split-Brain) - -**Priority**: Low -**Location**: Architecture gap - -**Problem**: If the daemon can reach some origins but not others (network partition), it may serve inconsistent data — e.g., origin A has version 1 of a file, origin B has version 2, and only B is reachable. The daemon happily serves version 2 without noting that the file's origin of record (A, higher priority) is unavailable. - -Currently, `FailoverExecutor` just tries origins in priority order and uses whoever responds first. There's no concept of "this file's authoritative origin is A, and A is down — we're serving from backup B which may be stale." - -**Impact**: Subtle inconsistency — user may hear an old version of a re-tagged file without knowing it. Low severity for a music filesystem, but matters for correctness. - -**Required**: -- Track per-file "authoritative origin" (the origin with highest priority that has the file) -- When serving from non-authoritative origin: set a flag, log at debug level -- When authoritative origin recovers: trigger delta sync for files served from backup -- Optional: expose "served from backup" as extended attribute or in gRPC events -- This is P3 / nice-to-have — the read-only nature of MusicFS makes this low-risk - ---- - -### 4.3 Network Fault Summary - -| Failure Type | Detection Time | Recovery | Gaps | -|---|---|---|---| -| **Source machine death** | 15-90s (health check cycles) | Automatic failover to backup origin | Health check on local origin has no timeout; checks are sequential | -| **Network partition** | 5-15s (first failed read + health) | Failover to reachable origin | No stale-data awareness for files served from backup | -| **Transient NFS stale handle** | Immediate (on read attempt) | Automatic retry in NFS origin | ✅ Handled | -| **SMB session drop** | Immediate (on read attempt) | Automatic retry in SMB origin | ✅ Handled | -| **All origins down** | 15-90s | Serve from cache (CAS) | No offline mode state machine, log spam | -| **Origin recovery** | 30s (next health check) | Auto-detected, routing restored | Watcher not restarted, no catch-up sync | -| **DNS failure** | 5-15s (health check timeout) | None — treated as origin death | No distinction from real failure | -| **Slow network (not dead)** | Not detected | Reads succeed but slowly | No latency-based degradation threshold | - ---- - -## 5. Additional Critical Issues - -These are failure modes not covered by the network, shutdown, or crash-recovery categories above. They deal with resource exhaustion, runtime deadlocks, and data loss scenarios specific to a FUSE daemon. - -### 5.1 FUSE↔Tokio Deadlock Risk (block_on inside sync callback) - -**Priority**: Critical -**Location**: `musicfs-fuse/src/filesystem.rs` (read method) - -**Problem**: The `fuser` crate requires the `Filesystem` trait to be implemented synchronously — all callbacks (`lookup`, `getattr`, `readdir`, `read`) run on fuser's internal thread. But all of MusicFS's I/O is async (tokio). The current bridge is: - -```rust -fn read(&mut self, ...) { - let result = std::thread::scope(|_| { - handle.block_on(async { - reader.read(file_id, offset as u64, size).await - }) - }); -} -``` - -`handle.block_on()` from inside a non-tokio thread blocks that thread until the future completes. This is generally fine. **But** if the tokio runtime's thread pool is saturated (all worker threads are busy), the `block_on` call will deadlock — it's waiting for a tokio worker to pick up the task, but all workers are busy (possibly also doing `block_on` calls from other FUSE requests if `fuser` uses multiple threads internally, or doing heavy CAS I/O). - -Specific deadlock scenario: -1. Multiple FUSE reads arrive simultaneously (Plex scanning library) -2. Each calls `handle.block_on()` which enqueues work on the tokio runtime -3. The tokio runtime workers are busy with CAS I/O, health checks, prefetching, watcher events -4. `block_on` waits for a free worker → FUSE thread blocks -5. If fuser processes requests on a single thread (which `mount2` does by default): **all FUSE operations hang** -6. Even `ls` and `stat` are blocked because they share the same fuser thread - -**Impact**: Complete filesystem hang under moderate load. Users see `ls /mnt/music` hang indefinitely. The daemon is alive, systemd thinks it's fine, but the filesystem is frozen. - -**Required**: -- Use `fuser::spawn_mount2()` instead of `mount2()` — this runs FUSE in a background thread and returns a `BackgroundSession`, freeing the main thread for async work -- Consider using `tokio::task::spawn_blocking()` for FUSE reads instead of `std::thread::scope` + `block_on` — this uses tokio's dedicated blocking thread pool which auto-grows -- Set `tokio::runtime::Builder::max_blocking_threads()` appropriately (default 512, should be sufficient) -- Add metrics: track FUSE callback latency, tokio task queue depth -- Alternatively: use `fuser`'s `Session::run_custom()` with a custom thread pool, or implement `Filesystem` with async support if fuser supports it - -**Architecture ref**: NFR-2.4 (>1000 concurrent file handles), NFR-1.3 (<5ms open cached) - ---- - -### 5.2 Tantivy Index Corruption on Crash - -**Priority**: High -**Location**: `musicfs-search/src/index.rs`, `musicfs-search/src/indexer.rs` - -**Problem**: The tantivy `IndexWriter` buffers documents in memory and only flushes to disk on `commit()`. The indexer commits every 5 seconds (via `commit_timer`). If the daemon crashes between commits, all indexed documents since the last commit are lost. - -Worse: if a crash occurs **during** a `commit()` call, the tantivy index files on disk may be in an inconsistent state. Tantivy uses a segment-based architecture — a commit writes new segment files and updates a `meta.json` manifest. If the process dies between writing segments and updating the manifest, the index may reference files that don't exist or miss files that do. - -**Current code** (indexer.rs): -```rust -_ = commit_timer.tick() => { - if pending_commit { - if let Err(e) = self.index.commit() { - error!("Index commit error: {}", e); - } - pending_commit = false; - } -} -``` - -The `IndexWriter` is allocated with 50MB heap (`index.writer(50_000_000)`). In a heavy indexing scenario (origin rescan of 100K files), up to 50MB of uncommitted document data can be lost. - -On the `index.commit()` error path: the indexer logs the error and continues. But a failed commit may leave the writer in an inconsistent state — subsequent `add_document` or `commit` calls may also fail. - -**Impact**: After crash recovery, search results are incomplete or empty. Users search for a song they know exists and get no results. - -**Required**: -- On startup: attempt to open tantivy index. If `Index::open_in_dir()` fails with corruption, delete the index directory and rebuild from SQLite metadata -- Add a "rebuild search index" CLI command: `musicfs search rebuild` -- Reduce commit interval to 1-2 seconds for lower data loss window (tradeoff: more I/O) -- On `commit()` failure: try `writer.rollback()` to restore consistent state, then retry -- On persistent commit failures: stop the indexer, log critical error, flag for rebuild on restart -- Add integrity check on startup: run a simple search query — if it panics or errors, rebuild - -**Architecture ref**: FR-14.1 (index metadata for full-text search — index must be recoverable) - ---- - -### 5.3 File Descriptor Exhaustion - -**Priority**: High -**Location**: System-wide, `dist/musicfs.service` - -**Problem**: MusicFS holds open many file descriptors simultaneously: -- 1 for FUSE `/dev/fuse` -- 1 for SQLite database (+ WAL + SHM = 3 total) -- 1 for sled (multiple internal files, typically 5-10) -- 1 per tantivy segment (grows with index size, typically 10-50) -- 1 per inotify watch (1 per watched directory — can be thousands) -- N for CAS chunk reads during cache misses (concurrent fetcher operations) -- N for gRPC connections (1 per connected client) -- N for origin file reads (local origin opens files via tokio::fs) - -The default Linux `ulimit -n` is 1024. A music library with 10K directories being watched could exhaust this easily (inotify allocates one fd per watch on the directory, plus the inotify fd itself). - -The systemd service has **no `LimitNOFILE` directive**. - -**Impact**: Once fd limit is hit, every operation fails: FUSE reads return EIO, SQLite queries fail, new inotify watches fail silently, gRPC connections are rejected. The daemon is technically alive but completely non-functional. - -**Required**: -- Add `LimitNOFILE=65536` to `dist/musicfs.service` -- Track open fd count via `/proc/self/fd` periodically (every 60s), export as metric -- Set high/critical watermarks: at 80% of limit, log warning; at 95%, stop accepting new gRPC connections and pause prefetching -- For inotify specifically: Linux has `fs.inotify.max_user_watches` (default 8192 on some distros, 524288 on others). Document the requirement: `sysctl fs.inotify.max_user_watches=524288` -- Consider: for very large libraries (100K+ directories), inotify is not viable — switch to polling-based change detection (already mentioned in architecture for remote origins, but needed for large local origins too) - -**Architecture ref**: NFR-3.1 (handle 1M+ files), NFR-3.2 (handle 100K+ directory entries) - ---- - -### 5.4 inotify Unreliable for NFS/SMB Watches - -**Priority**: Medium -**Location**: `musicfs-sync/src/watcher.rs`, `musicfs-origins/src/nfs.rs`, `musicfs-origins/src/smb.rs` - -**Problem**: The `OriginWatcher` uses `notify::RecommendedWatcher` (which uses inotify on Linux) for ALL origin types. But inotify does NOT work across NFS or SMB mounts — the NFS/SMB server doesn't send change notifications to the client kernel. The code already acknowledges this: - -```rust -// nfs.rs -debug!("NFS watch - inotify may be unreliable over NFS, consider polling"); - -// smb.rs -warn!("SMB watch using inotify - may be unreliable. Consider polling for remote mounts."); -``` - -But then proceeds to set up inotify anyway. Changes made on the NFS/SMB server (or by other clients) will NEVER be detected by the watcher. - -**Impact**: Files added/modified/deleted on the NFS/SMB server are invisible to MusicFS until manual rescan. Users add music to their NAS and wonder why it doesn't appear. - -**Required**: -- Implement polling-based watcher for remote origin types (NFS, SMB, S3, SFTP) -- Polling interval: configurable per origin, default 300s (5 minutes) -- Polling strategy: walk directory tree, compare mtime against cached mtime -- Optimization: only walk directories whose parent mtime changed (directory mtime changes when files are added/removed) -- Keep inotify for local origins (reliable and instant) -- Hybrid mode for "local" origins that might be network mounts: start with inotify, fall back to polling if no events detected after initial changes - -**Architecture ref**: FR-10.3 (use polling for remote origins without push support) - ---- - -### 5.5 Memory Growth from Virtual Tree - -**Priority**: Medium -**Location**: `musicfs-cache/src/tree.rs` - -**Problem**: The `VirtualTree` holds the entire directory structure in memory — every directory node, every file node, the inode map, and the path map. For 1M files with average path length of 100 bytes: -- `inode_map`: 1M entries × ~100 bytes = ~100MB -- `path_map`: 1M entries × ~150 bytes (path + overhead) = ~150MB -- `DirNode.children`: BTreeMap overhead per directory -- Total: ~300-400MB for 1M files, approaching the NFR-4.3 peak limit of 500MB - -The tree is wrapped in `Arc>` and kept fully in memory for the entire daemon lifetime. There's no pagination, no lazy loading of deep subtrees, and no eviction of rarely-accessed branches. - -**Current code**: `TreeBuilder::build()` constructs the entire tree upfront during mount. For 10M files (stretch goal NFR-3.5), this would require 3-4GB of RAM — well beyond limits. - -**Impact**: Memory usage scales linearly with library size. At 10M files, the daemon either OOMs or is killed by systemd MemoryMax. - -**Required**: -- Short term: add `MemoryMax=2G` to systemd service as safety net (prevents OOM-killing other services) -- Short term: track RSS via `/proc/self/statm`, export as metric, warn at 80% of limit -- Medium term: lazy subtree loading — only load the first 2 levels of the tree on mount, load deeper levels on first `readdir()` -- Medium term: evict cold subtrees after configurable timeout (30 minutes no access) -- Long term: move tree to SQLite/sled-backed structure with in-memory LRU cache for hot paths — this is a significant architectural change - -**Architecture ref**: NFR-4.1 (idle <50MB), NFR-4.3 (peak <500MB), NFR-3.5 (10M files stretch goal) - ---- - -### 5.6 System Clock Jump Breaks Mtime Comparison - -**Priority**: Low -**Location**: `musicfs-sync/src/delta.rs`, `musicfs-cache/src/db.rs` - -**Problem**: Delta detection compares `origin_mtime` (stored as unix timestamp in SQLite) against the current file's mtime. If the system clock jumps (NTP correction, VM suspend/resume, manual adjustment), files may appear changed (clock jumped forward — everything looks "modified") or unchanged (clock jumped backward — new files look "old"). - -Additionally, `last_sync` in the database uses `strftime('%s', 'now')` which is based on wall-clock time. A clock jump can make sync timing calculations wrong — e.g., "sync all files changed in the last hour" could miss files if the clock jumped forward. - -**Impact**: Unnecessary full re-sync after NTP correction (wastes bandwidth), or missed changes after backward clock jump (stale data served). - -**Required**: -- Use monotonic clock (`Instant`) for internal timing (health checks, intervals) — already done in health.rs -- For mtime comparison: use content hash as secondary check when mtime is "suspicious" (within 5 seconds of a known clock jump) -- Track clock jumps: compare `SystemTime::now()` against monotonic progression, log if jump >5s detected -- For `last_sync`: store both wall-clock time and a monotonic sequence number -- Note: this is inherent to any mtime-based system. Even git has this problem. Low priority because NTP corrections are typically <1s on well-configured systems - ---- - -### 5.7 CAS Chunk Write Not Atomic - -**Priority**: Medium -**Location**: `musicfs-cas/src/store.rs` - -**Problem**: CAS `put()` writes a chunk in two steps: -```rust -fs::write(&path, data).await?; // Step 1: write chunk file -self.index.insert(hash, location)?; // Step 2: update sled index -``` - -If the process crashes between step 1 and step 2: orphaned chunk file on disk (wastes space, but harmless). If the process crashes during step 1: partially written chunk file on disk. On next startup, `calculate_size()` counts this partial file, and if someone tries to read it, `verify_integrity()` will catch the hash mismatch — but only on read, not proactively. - -More subtle: `fs::write()` in tokio is NOT atomic. It calls `write_all()` which may do multiple syscalls. If the kernel OOM-kills the process or power is lost during write, the file contains partial data. - -**Impact**: After crash: orphaned or partial chunk files. Partial chunks cause integrity errors on read, which currently propagate as `CasError::IntegrityError` and cause FUSE to return EIO. - -**Required**: -- Write to temporary file first: `{path}.tmp` -- Call `fsync()` on the temporary file (ensures data is on disk, not just in page cache) -- Rename temporary to final path: `rename()` is atomic on Linux for same-filesystem renames -- Then update sled index -- This guarantees: either the chunk is fully written and indexed, or it doesn't exist -- On startup: scan for `.tmp` files in chunks directory, delete them (incomplete writes from previous crash) -- Cost: one extra `rename()` syscall per chunk write — negligible - ---- - -### 5.8 No Resource Limits in systemd Service - -**Priority**: Medium -**Location**: `dist/musicfs.service` - -**Problem**: The systemd service has security hardening (`NoNewPrivileges`, `ProtectSystem`, `PrivateTmp`) but no resource limits. A bug causing infinite allocation (memory leak, unbounded cache, runaway indexing) will consume all system resources before anything stops it. - -**Current service has NO**: -- `LimitNOFILE` — fd limit (default 1024, way too low) -- `MemoryMax` — memory ceiling -- `MemoryHigh` — memory pressure notification threshold -- `TasksMax` — thread/task limit -- `CPUQuota` — CPU limit (prevents background tasks from starving other services) -- `IOWeight` — I/O priority -- `WatchdogSec` — liveness check (covered in gap 3.1) - -**Impact**: Resource leak → system destabilization. OOM killer picks random victim (might kill sshd or Plex instead of musicfs). - -**Required**: -```ini -# Resource limits -LimitNOFILE=65536 -MemoryMax=4G -MemoryHigh=2G -TasksMax=4096 -CPUQuota=200% - -# I/O priority (lower than media playback, higher than backups) -IOSchedulingClass=best-effort -IOSchedulingPriority=4 - -# OOM handling - prefer killing musicfs over other services -OOMScoreAdjust=200 -``` - -- `MemoryHigh=2G` triggers kernel memory pressure reclaim before hitting hard limit — gives the daemon a chance to evict cache -- `MemoryMax=4G` is the hard kill limit -- `TasksMax=4096` prevents thread/task bomb from runaway spawn loops -- `OOMScoreAdjust=200` makes the kernel prefer killing musicfs over other daemons (it can recover via restart, others may not) - ---- - -## 6. Cache/Database Sudden Death Analysis - -### 6.1 Data Flow Map: What Touches What - -Understanding which storage layer each operation depends on is critical for failure analysis: - -``` -FUSE hot path (every file access): - lookup/getattr/readdir/open → VirtualTree (in-memory only) ← NO disk dependency - read (cache hit) → CasStore.get() ← sled index + chunk files - read (cache miss) → ContentFetcher → Origin.read() ← sled + chunk files + origin - -Background tasks: - Search indexer → tantivy index (disk) - Pattern recording → PatternStore (SQLite, separate DB) - Collection queries → CollectionStore (SQLite, separate DB) - Health monitor → in-memory only (DashMap) - File watcher → in-memory + EventBus - -Startup only (not runtime): - scan_music_files() → origin filesystem - TreeBuilder::build() → builds in-memory VirtualTree - Database is used for metadata caching but NOT in FUSE hot path currently -``` - -### 6.2 Scenario: SQLite Metadata Database Dies - -**How it can die**: File deleted by user/script, filesystem corruption, disk bad sector, `rm ~/.cache/musicfs/metadata.db` by mistake, permissions changed. - -**What happens NOW**: -- **FUSE browsing (lookup/readdir/stat)**: **Unaffected** — VirtualTree is entirely in-memory. Users can browse and see all files. -- **FUSE read**: **Unaffected** — FileReader uses in-memory manifests + CAS. SQLite is not in the read path. -- **Search indexer**: **Unaffected** — uses tantivy, not SQLite. -- **Pattern recording**: **FAILS** — PatternStore has its own SQLite connection. If the pattern DB file is deleted, `record()` returns `PatternError::Database`. The prefetch engine catches this: `warn!("Failed to record access pattern: {}")` and continues. **Gracefully degraded**. -- **Collection queries**: **FAILS** — CollectionStore operations fail with `Error::Database`. Smart collections stop working. -- **Delta sync**: **FAILS** — DeltaDetector queries SQLite for mtime comparisons. Sync operations fail. -- **On restart**: **FATAL** — `Database::open()` re-creates the schema on empty DB, but all metadata is lost. The initial scan repopulates from origin, but this means O(N) startup again + complete re-index. - -**Gap**: No detection of SQLite corruption during runtime. No mechanism to reconstruct SQLite from origin files without full restart. No backup/snapshot of metadata DB. - -**Required**: -- Periodic SQLite health check: `PRAGMA quick_check` every 5 minutes (lightweight, checks page integrity) -- If SQLite becomes inaccessible during runtime: log error, flag for rebuild on restart, continue serving from in-memory tree + CAS -- On startup with missing/corrupt DB: auto-trigger full rescan from origins (already happens implicitly since scan_music_files doesn't use DB, but should log clearly) -- Consider: periodic SQLite backup via `VACUUM INTO '/path/metadata.db.bak'` (atomic backup while DB is open, available since SQLite 3.27) -- Document: `metadata.db` can always be rebuilt from origins — it's a cache, not source of truth - ---- - -### 6.3 Scenario: sled Chunk Index Dies - -**How it can die**: Disk corruption, `rm -rf ~/.cache/musicfs/chunks/index.sled/`, sled internal corruption (rare but documented), unclean shutdown leaving sled in bad state. - -**What happens NOW**: -- **sled::open() on startup**: Returns `sled::Error::Corruption` → propagated as `CasError::Sled` → daemon **crashes on startup**. There is no recovery attempt. -- **sled operation during runtime** (if files deleted under sled): sled will panic or return errors. `CasStore.get()` calls `self.index.insert()` / `self.index.get()` — these errors propagate to `ReaderError::Cas` → FUSE returns EIO. -- **Orphaned chunks**: If sled index is gone but chunk files remain on disk, chunks are invisible. They waste disk space but aren't harmful. -- **Missing chunks with valid index**: If chunk files are deleted but sled still has entries, `CasStore.get()` reads the file → `CasError::Io(NotFound)` → EIO. - -**Critical issue**: sled corruption = **daemon cannot start**. No recovery, no repair attempt. - -**Current code** (store.rs): -```rust -let index = sled::open(&index_path)?; // Panics or errors on corruption -``` - -**Required**: -- On `sled::open()` failure: attempt `sled::Config::new().path(&index_path).repair(true).open()` — sled has built-in repair mode -- If repair fails: delete the sled directory, recreate empty index, and rebuild by scanning chunk files on disk (walk shard directories, compute hash of each file, re-insert into index) -- During runtime: catch sled errors in `put()`/`get()` paths, don't propagate as panics -- Add CLI command: `musicfs cache repair` — rebuilds sled index from chunk files - ---- - -### 6.4 Scenario: CAS Chunk Files Deleted/Corrupted - -**How it can die**: User deletes chunks directory, disk failure, bitrot on cache drive, filesystem corruption, `rm -rf ~/.cache/musicfs/chunks/` by mistake. - -**What happens NOW**: -- **All chunks deleted**: Every `CasStore.get()` returns `CasError::NotFound`. Every FUSE `read()` returns EIO. The filesystem is "browsable" (tree is in memory) but no file can be read. -- **Some chunks deleted**: Affected files return EIO on read. Other files work fine. **No detection** — corruption is only discovered when a specific chunk is requested. -- **Corrupted chunk** (bitrot): `verify_integrity()` catches hash mismatch → `CasError::IntegrityError` → EIO. The corrupted chunk is NOT auto-deleted or re-fetched. -- **Chunk directory permissions changed**: `fs::read()` returns permission error → `CasError::Io` → EIO. - -**Critical gaps**: -1. **No automatic re-fetch on integrity error**: When `verify_integrity()` fails, the daemon returns EIO but doesn't try to re-fetch the chunk from origin. The user is stuck with a corrupt chunk until cache is cleared. -2. **No proactive corruption scanning**: Bitrot can sit undetected for months until a specific file is played. -3. **No distinction between "chunk missing" and "origin down"**: When a read fails, the user sees EIO either way. No hint about whether clearing cache would fix it. -4. **Size tracking is wrong**: (as noted in 3.10) — `current_size` doesn't reflect reality, so eviction doesn't work. - -**Required**: -- On `CasError::IntegrityError`: delete the corrupt chunk, re-fetch from origin automatically, return data to FUSE caller (transparent repair) -- On `CasError::NotFound` with fetcher available: attempt to fetch from origin before returning EIO (this may already work via `get_or_fetch_manifest` but not for individual chunks — the manifest is fetched, but if a chunk file was deleted after manifest creation, only EIO is returned) -- Background scrubber: periodically (daily, configurable) verify N random chunks' integrity. Report corruption rate. If >1% corrupt, trigger full scan. -- On startup with empty/missing chunks directory: create it, log warning, treat all files as cache misses (origin fetch on demand) -- `musicfs cache verify` CLI command: full integrity scan with progress and repair option - ---- - -### 6.5 Scenario: tantivy Search Index Dies - -**How it can die**: Disk corruption, directory deleted, crash during `commit()` (as discussed in 5.2), `meta.json` corrupted, segment files truncated. - -**What happens NOW**: -- **Index deleted/corrupt on startup**: `SearchIndex::open()` calls `Index::open_in_dir()` → tantivy returns error → `SearchError::Tantivy` → daemon crashes (if search is required) or search is unavailable. -- **Current open logic** (index.rs): -```rust -let index = if index_path.exists() && index_path.join("meta.json").exists() { - Index::open_in_dir(index_path)? // Can fail with corruption -} else { - std::fs::create_dir_all(index_path)?; - Index::create_in_dir(index_path, schema_obj.schema.clone())? -}; -``` -- **Commit failure during runtime**: Indexer logs error, sets `pending_commit = false`, continues. But uncommitted documents are lost, and the writer may be in an inconsistent state. -- **Reader reload failure**: After a bad commit, `self.reader.reload()` may fail → subsequent searches return stale results or errors. - -**Impact**: -- Startup crash if index is corrupt and code doesn't handle the error -- Search returns no results or stale results after crash recovery -- `/.search/` virtual directory is broken - -**Required**: -- On `Index::open_in_dir()` failure: log error, delete index directory, create fresh index, trigger re-index from SQLite metadata or in-memory tree -- On `commit()` failure: attempt `writer.rollback()`, log error, schedule retry -- On persistent commit failures (3+ consecutive): mark indexer as degraded, stop attempting commits, flag for rebuild -- Re-index capability: `musicfs search rebuild` CLI command -- On startup: verify index health with simple query before declaring ready - ---- - -### 6.6 Scenario: Cache Disk Hardware Failure - -**How it can die**: SSD wear-out, HDD bad sectors, NVMe controller failure, filesystem goes read-only (ext4 remounts read-only on errors). - -**What happens NOW**: -- **Disk goes read-only**: All writes fail (CAS put, sled insert, SQLite upsert, tantivy commit). Reads continue working for cached data. No detection — each component reports IO errors independently with no correlation. -- **Disk completely dead**: All cache operations fail. The daemon is effectively a broken pipe — tree in memory but every read() returns EIO. -- **Partial failure (bad sectors)**: Random IO errors on specific files. Some chunks work, others don't. Unpredictable behavior. - -**Critical gap**: There is no centralized "cache health" check. Each component (SQLite, sled, CAS, tantivy) handles IO errors independently. There's no detection of "the entire cache disk is gone." - -**Required**: -- Centralized cache health monitor: - - Periodically (every 60s): attempt to write a small test file to cache directory, read it back, delete it - - If write fails: cache disk is read-only or dead → enter "passthrough mode" - - Track consecutive IO errors across all components → if >N in M seconds, declare cache unhealthy -- **Passthrough mode** (cache disk dead, origins still alive): - - Serve reads directly from origin (bypass CAS entirely) - - Disable prefetching, pattern recording, search indexing - - Log: `error!("Cache disk failure detected, operating in passthrough mode")` - - Set MountState to Degraded - - This is the "graceful degradation" the architecture requires (NFR-7.2) -- **Recovery**: When cache disk comes back (e.g., ext4 remount-rw after fsck): - - Detect via periodic health check - - Run integrity checks on all stores - - Resume normal operation - - Log: `info!("Cache disk recovered, resuming cached operation")` - ---- - -### 6.7 Scenario: Cache Directory Permissions Changed - -**How it can die**: Security hardening script, SELinux/AppArmor policy change, user accidentally `chmod 000 ~/.cache/musicfs/`, ownership change. - -**What happens NOW**: Every cache operation fails with permission denied. Each component logs its own error. No centralized detection. The daemon appears to work (tree in memory) but every `read()` fails. - -**Required**: -- On startup: verify write permissions on cache directory, chunks directory, and DB files -- If permissions are wrong: log clear error message with exact path and expected permissions -- During runtime: permission errors should trigger the same cache health check as disk failure → enter passthrough mode if origins are available -- systemd service already has `ReadWritePaths=/var/cache/musicfs` — but this doesn't help if permissions on the directory itself are wrong - ---- - -### 6.8 Cache Failure Summary - -| Component | Dies on Startup | Dies During Runtime | Recovery | -|---|---|---|---| -| **SQLite metadata.db** | Recreates empty DB, full rescan needed | In-memory tree + CAS unaffected, patterns/collections fail | Rebuild from origin rescan | -| **sled chunk index** | **DAEMON CRASHES** — no repair attempt | Chunk reads fail (EIO) | Repair mode or rebuild from chunk files | -| **CAS chunk files** | Cache dir recreated, all files are cache misses | Affected reads fail (EIO), no auto re-fetch | Re-fetch from origins on demand | -| **tantivy index** | May crash or create empty index | Search returns stale/no results | Rebuild from SQLite/tree metadata | -| **Pattern DB** | Recreated empty, predictions reset | Prefetch degrades gracefully (warn + continue) | Naturally repopulates from access patterns | -| **Cache disk (hardware)** | Daemon cannot start | All cache ops fail, EIO on reads | Passthrough mode (serve from origins) | - -**The biggest gap**: No "passthrough mode." If the cache disk dies but origins are alive, MusicFS should still serve files. Currently it just returns EIO everywhere. This violates NFR-7.2 (graceful degradation) — the cache is supposed to be an optimization, not a hard dependency. - ---- - -## 7. Critical Architecture Gap: No Persistent State Used on Restart - -**Full analysis moved to**: [persistent-state.md](persistent-state.md) - -**Summary**: Every mount is a full cold start — O(N × origin_latency). SQLite, tantivy, patterns, and manifests all persist on disk but none are opened during mount. The 4 critical in-memory structures (VirtualTree ~400MB, ContentFetcher.file_meta ~200MB, FileReader.manifests ~100MB, LruEviction ~50MB) are rebuilt from scratch on every restart. This violates G1 (O(1) mount time), NFR-1.7 (<500ms mount), and FR-7.1 (cache persists across restarts). - -**This blocks all resilience work** — persistent state must be wired up before graceful shutdown, crash recovery, or cache integrity checks have meaning. - ---- - -## 8. Requirements Coverage - -| Requirement | Description | Status | -|-------------|-------------|--------| -| NFR-7.1 | Serve cached data when origin unavailable | ✅ Via failover | -| NFR-7.2 | Graceful degradation on network failure | ⚠️ Partial (failover yes, no graceful shutdown) | -| NFR-7.3 | Retry with exponential backoff (100ms, 500ms, 2s) | ✅ In failover.rs | -| NFR-7.4 | Don't crash on malformed audio | ✅ parse_file returns Result | -| NFR-8.1 | Verify chunk integrity via checksums | ❌ Missing | -| NFR-8.2 | ACID transactions for cache DB | ✅ SQLite WAL | -| NFR-8.3 | Recover from interrupted synchronization | ❌ Missing | -| NFR-8.4 | Detect and report cache corruption | ❌ Missing | -| FR-1.4 | Release all resources on unmount | ❌ No graceful unmount | -| FR-17.5 | Graceful shutdown with drain | ❌ Stub only | -| FR-25.3 | Zero-downtime upgrades | ❌ Missing | -| FR-25.5 | Validate cache integrity on startup | ❌ Missing | - ---- - -## 5. Implementation Priority - -### Phase 0: Wire Up Persistent State (Foundational — Unblocks Everything) - -**See [persistent-state.md](persistent-state.md)** — ~8 days, storage engine decision pending. - -Must be completed before Phase A. Without persistent state, graceful shutdown has nothing to flush, crash recovery has nothing to recover, and integrity checks have nothing to check. - -### Phase A: Stop Dying (Critical — Must Ship First) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| A1 | Signal handling (SIGTERM/SIGINT) + CancellationToken | 1 day | Everything | -| A2 | Graceful shutdown orchestration (ordered teardown) | 1 day | A1 | -| A3 | Panic hook (log before death) | 0.5 day | — | -| A4 | RwLock poison recovery (or switch to parking_lot) | 0.5 day | — | -| A5 | FUSE cleanup on exit + ExecStopPost in systemd | 0.5 day | A1, A2 | -| A6 | sd_notify integration (READY/STOPPING/WATCHDOG) | 0.5 day | A1 | - -### Phase B: Recover From Crashes (High — Required for Production) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| B1 | Task supervisor (monitor + restart background tasks) | 1 day | A1 | -| B2 | Startup integrity checks (SQLite + tantivy + CAS) | 1 day | — | -| B3 | Stale mountpoint detection + auto-cleanup on startup | 0.5 day | — | -| B4 | Disk space monitoring + watermark eviction | 1 day | — | - -### Phase C: Resilient Operations (Medium — Production Hardening) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| C1 | Interrupted sync recovery (checkpoint/resume) | 1.5 days | — | -| C2 | CAS chunk integrity verification + sled recovery check | 1 day | — | -| C3 | systemd watchdog integration | 0.5 day | A6 | -| C4 | SIGHUP config reload | 1 day | A1 | -| C5 | Connection pooling for remote origins (SFTP/S3) | 1 day | — | -| C6 | Fix CAS size accounting + persistent eviction LRU | 1 day | — | -| C7 | FUSE read timeout enforcement | 0.5 day | — | -| C8 | Event bus backpressure + capacity config | 0.5 day | — | -| C9 | PID file / flock to prevent concurrent mounts | 0.5 day | — | -| C10 | FUSE session recovery (detect disconnect + remount) | 1 day | A1, A2 | - -### Phase D: Network Resilience (Medium — Hardening for Real-World Networks) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| D1 | Add timeout to local origin health check | 0.25 day | — | -| D2 | Parallelize health checks (join_all instead of sequential) | 0.5 day | — | -| D3 | Offline mode state machine (Ready→Degraded→Offline) | 1 day | — | -| D4 | Auto-restart watcher on origin recovery (Unhealthy→Healthy) | 0.5 day | B1 | -| D5 | DNS failure handling for remote origins | 0.5 day | C5 | -| D6 | Network partition / stale-data awareness | 0.5 day | — | - -### Phase E: Runtime Robustness (High/Medium — Prevents Silent Degradation Under Load) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| E1 | Fix FUSE↔tokio deadlock: switch to spawn_mount2 + spawn_blocking | 1 day | — | -| E2 | Tantivy crash recovery: detect corruption, rebuild from SQLite | 1 day | — | -| E3 | Atomic CAS chunk writes (write-to-tmp + rename + fsync) | 0.5 day | — | -| E4 | systemd resource limits (LimitNOFILE, MemoryMax, TasksMax, OOM) | 0.25 day | — | -| E5 | fd exhaustion monitoring + inotify watch limit documentation | 0.5 day | — | -| E6 | Polling-based watcher for NFS/SMB origins | 1.5 days | — | -| E7 | Memory tracking + metrics for virtual tree growth | 0.5 day | — | - -### Phase F: Cache Resilience (High — Prevents Total Failure on Cache Corruption) - -| # | Task | Effort | Blocks | -|---|------|--------|--------| -| F1 | sled corruption recovery (repair mode + rebuild from chunk files) | 1 day | — | -| F2 | CAS auto re-fetch on integrity error (transparent repair) | 0.5 day | — | -| F3 | Passthrough mode (bypass cache, serve from origins when cache disk dies) | 1.5 days | — | -| F4 | Centralized cache health monitor (write test + IO error correlation) | 1 day | — | -| F5 | tantivy index corruption recovery (detect + rebuild from metadata) | 1 day | E2 | -| F6 | Background chunk scrubber (periodic integrity verification) | 0.5 day | — | -| F7 | SQLite periodic backup (VACUUM INTO) + startup permission check | 0.5 day | — | -| F8 | `musicfs cache verify` + `musicfs cache repair` + `musicfs search rebuild` CLI | 1 day | F1, F2, F5 | - -**Total estimate**: ~30.5 days across phases A-F (Phase 0 tracked separately in [persistent-state.md](persistent-state.md) — ~8 days, storage decision pending) - ---- - -## 6. Key Design Decisions Needed - -1. **parking_lot vs std RwLock**: `parking_lot::RwLock` never poisons (simpler), but loses panic detection. Recommended: use `parking_lot` — panics are caught by the task supervisor, not by lock poisoning. - -2. **CancellationToken propagation**: Every component that spawns tasks needs access to the token. Options: (a) pass through constructors, (b) global static. Recommended: pass through constructors for explicit dependency. - -3. **Integrity check depth on startup**: Full check (verify every chunk hash) vs quick check (SQLite integrity + spot-check 100 random chunks). Recommended: quick check by default, `--full-integrity-check` flag for thorough mode. - -4. **Task restart policy**: Immediate restart vs exponential backoff. Recommended: immediate first restart, then 1s→5s→30s backoff, max 5 restarts before marking task as permanently failed. - ---- - -## 7. Files That Need Changes - -### Phase 0 (Foundational) — see [persistent-state.md](persistent-state.md) - -### Phase A (Critical) -- `musicfs-cli/src/main.rs` — Signal handling, shutdown orchestration, sd_notify -- `musicfs-cli/Cargo.toml` — Add `tokio-util`, `sd-notify` deps -- `musicfs-fuse/src/filesystem.rs` — RwLock poison recovery -- `musicfs-cas/src/reader.rs` — RwLock poison recovery -- `musicfs-origins/src/registry.rs` — RwLock poison recovery -- `musicfs-cas/src/fetcher.rs` — RwLock poison recovery -- `musicfs-cache/src/eviction.rs` — RwLock poison recovery -- `musicfs-core/src/metrics.rs` — RwLock poison recovery -- `dist/musicfs.service` — ExecStopPost, WatchdogSec - -### Phase B (High) -- `musicfs-core/src/lib.rs` — TaskSupervisor, new module -- `musicfs-cache/src/db.rs` — Integrity check on open -- `musicfs-cas/src/store.rs` — Disk space checks -- `musicfs-cli/src/main.rs` — Stale mount detection - -### Phase C (Medium) -- `musicfs-sync/src/delta.rs` — Checkpoint/resume -- `musicfs-cache/src/schema.sql` — sync_progress table -- `musicfs-core/src/config.rs` — Config reload support, event_bus_capacity -- `musicfs-cas/src/store.rs` — Fix calculate_size() recursion, sled recovery verification -- `musicfs-cache/src/eviction.rs` — Persistent LRU access times -- `musicfs-core/src/events.rs` — Lag metrics, configurable capacity -- `musicfs-fuse/src/filesystem.rs` — Read timeout, FUSE session recovery -- `musicfs-origins/src/sftp.rs` — Connection pool (deadpool) -- `musicfs-origins/src/s3.rs` — Explicit hyper pool config, request timeouts -- `musicfs-origins/Cargo.toml` — Add deadpool dependency - -### Phase D (Network Resilience) -- `musicfs-origins/src/local.rs` — Add timeout to health check -- `musicfs-origins/src/health.rs` — Parallel health checks, universal timeout wrapper -- `musicfs-core/src/lib.rs` — MountState enum, offline mode state machine -- `musicfs-origins/src/registry.rs` — Watcher restart on origin recovery -- `musicfs-sync/src/watcher.rs` — Auto-restart support -- `musicfs-grpc/src/server.rs` — MountState in StatusResponse - -### Phase E (Runtime Robustness) -- `musicfs-fuse/src/filesystem.rs` — Switch to spawn_mount2, use spawn_blocking for reads -- `musicfs-search/src/index.rs` — Corruption detection, rebuild capability -- `musicfs-search/src/indexer.rs` — Commit failure recovery (rollback + retry) -- `musicfs-cas/src/store.rs` — Atomic write (tmp + rename + fsync), .tmp cleanup on startup -- `musicfs-sync/src/watcher.rs` — Polling-based watcher variant for remote origins -- `musicfs-cache/src/tree.rs` — Memory tracking, lazy subtree loading (future) -- `dist/musicfs.service` — LimitNOFILE, MemoryMax, MemoryHigh, TasksMax, OOMScoreAdjust, IOSchedulingClass - -### Phase F (Cache Resilience) -- `musicfs-cas/src/store.rs` — sled repair on open failure, rebuild from chunk scan, integrity re-fetch, passthrough mode -- `musicfs-cas/src/reader.rs` — Auto re-fetch on chunk integrity error instead of returning EIO -- `musicfs-search/src/index.rs` — Corruption detection, delete + recreate on open failure -- `musicfs-cache/src/db.rs` — PRAGMA quick_check, VACUUM INTO backup, permission check on open -- `musicfs-core/src/lib.rs` — CacheHealthMonitor, passthrough mode flag -- `musicfs-cli/src/main.rs` — `cache verify`, `cache repair`, `search rebuild` CLI commands -- `musicfs-fuse/src/filesystem.rs` — Passthrough read path (bypass CAS, go to origin directly) diff --git a/docs/v2/plans/resilience-testing.md b/docs/v2/plans/resilience-testing.md deleted file mode 100644 index bff4aa0..0000000 --- a/docs/v2/plans/resilience-testing.md +++ /dev/null @@ -1,1003 +0,0 @@ -# MusicFS Resilience Testing: Design Doc - -**Authors:** AI-assisted -**Status:** Draft -**Last Updated:** 2026-05-13 -**Reviewers:** TBD -**Approvers:** TBD -**Prerequisites:** [resilience-fault-tolerance.md](resilience-fault-tolerance.md), [architecture.md](../architecture.md) - ---- - -[TOC] - ---- - -## 1. Abstract - -MusicFS has 162 unit/integration tests but zero fault injection, crash recovery, or resilience tests. This design doc defines the test infrastructure, tooling, and test cases needed to verify that MusicFS survives the 34 failure modes identified in the [resilience audit](resilience-fault-tolerance.md). - -The approach uses three testing layers: trait-based mocks with failpoints for fast unit-level verification, fork-kill process tests for crash and signal recovery, and Toxiproxy with Docker for real-protocol network fault injection. A new `musicfs-test-utils` crate centralizes shared test helpers that are currently duplicated across 29 files. - ---- - -## 2. Background - -### 2.1 Current Test State - -| Metric | Value | -|--------|-------| -| Total tests | 162 | -| Test files with `#[cfg(test)]` | 43 | -| Async tests (`#[tokio::test]`) | 44 | -| Fault injection tests | 0 | -| Crash recovery tests | 0 | -| Signal handling tests | 0 | -| CI pipeline | None | -| Mocking framework | None (real components + TempDir) | - -### 2.2 What Exists - -- **Unit tests**: Per-crate `#[cfg(test)]` modules using real implementations with `TempDir` isolation -- **Integration tests**: `crates/musicfs-cas/tests/integration.rs` — CAS + fetcher + reader pipeline -- **E2E tests**: `tests/e2e/e2e_players.rs` — mpv/VLC playback over mounted FUSE (`#[ignore]`, manual) -- **Test helpers**: `make_file_meta()`, `mock_health()` — duplicated across modules, not centralized -- **Test tooling**: `cargo-nextest` and `cargo-criterion` available in Nix flake - -### 2.3 What's Missing - -The [resilience audit](resilience-fault-tolerance.md) identified 34 failure modes across 6 phases. None have test coverage. The audit covers: -- Signal handling and graceful shutdown (Phase A) -- Crash recovery and cache integrity (Phase B) -- Network fault tolerance and origin failover (Phase C-D) -- Runtime deadlocks and resource exhaustion (Phase E) -- Cache/database sudden death and passthrough mode (Phase F) - -### 2.4 Why "Doing Nothing" Is Not an Option - -MusicFS is designed as a critical filesystem daemon. Untested failure paths mean: -- Crashes that corrupt SQLite, sled, or tantivy go undetected until production -- Signal handling code (once implemented) has no regression tests -- Origin failover logic is tested for correctness but not for actual failure scenarios -- No confidence that the daemon survives real-world conditions (disk full, NAS reboot, OOM) - ---- - -## 3. Goals & Non-Goals - -### 3.1 Goals - -- **Every resilience issue gets a test** — all 34 failure modes from the audit mapped to concrete test cases -- **Tests run without root** — no kernel modules, no privileged containers for Layer 1 and Layer 2 -- **Tests run fast** — Layer 1 tests complete in <1 second each; full resilience suite in <60 seconds -- **Failpoints are zero-cost** — conditional compilation via Cargo features; no runtime overhead in release builds -- **Test helpers are centralized** — `musicfs-test-utils` crate eliminates duplication across 29 files - -### 3.2 Non-Goals - -- **Full chaos engineering platform** — this is not Jepsen; we test known failure modes, not random exploration -- **Performance benchmarking** — covered separately by `cargo-criterion`; this doc is about correctness under failure -- **CI pipeline setup** — pipeline configuration (GitHub Actions, Nix CI) is a separate task; this doc defines what to run, not where -- **FUSE kernel-level testing** — testing kernel FUSE module behavior or `/dev/fuse` edge cases is out of scope - ---- - -## 4. Proposed Design - -### 4.1 Testing Layers - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Layer 3: Toxiproxy + Docker │ -│ Real protocols, real latency, real connection drops │ -│ ~5 tests, seconds each, requires docker-compose │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 2: Fork-Kill Process Tests │ -│ Spawn daemon, send signals, kill -9, verify recovery │ -│ ~5 tests, seconds each, cargo test │ -├─────────────────────────────────────────────────────────────┤ -│ Layer 1: Trait Mocks + Failpoints │ -│ FaultyOrigin, FaultyCasStore, fail_point! macros │ -│ ~25 tests, milliseconds each, cargo test │ -└─────────────────────────────────────────────────────────────┘ -``` - -**Rule**: Every resilience issue gets Layer 1 coverage at minimum. Critical issues (signal handling, crash recovery, FUSE unmount) additionally get Layer 2. Network-specific issues (origin failover, latency, connection drops) additionally get Layer 3. - -### 4.2 New Dependencies - -```toml -# Cargo.toml [workspace.dependencies] -fail = "0.5" # TiKV failpoints — conditional fault injection -rlimit = "0.10" # Resource limit manipulation (fd, memory) -nix = "0.29" # Signal sending, process control - -# Cargo.toml [workspace.features] -failpoints = ["fail/failpoints"] # Zero-cost when disabled - -# dev-dependencies only (not shipped in release binary) -wiremock = "0.6" # HTTP mock server (S3 origin tests) -assert_cmd = "2.0" # CLI integration testing -``` - -**Why these choices:** -- **`fail`** (TiKV failpoints): Production-proven by TiKV (distributed KV store). Zero overhead when `failpoints` feature is disabled. Supports deterministic failure injection with counter/probability controls. -- **`rlimit`**: Test fd exhaustion and memory limits without root. Wraps `setrlimit`/`getrlimit` syscalls. -- **`nix`**: Send signals to child processes (`kill(pid, SIGTERM)`). Already a transitive dependency via `fuser`. -- **`wiremock`**: Pure-Rust HTTP mock server for S3 origin testing. No external process needed. - -### 4.3 Test Infrastructure Crate - -**`crates/musicfs-test-utils/`** — new workspace crate providing shared test helpers. - -#### 4.3.1 FaultyOrigin - -Wraps any `Origin` implementation with configurable failure injection: - -```rust -pub struct FaultyOrigin { - inner: Arc, - fail_mode: Arc>, - call_count: AtomicUsize, -} - -pub enum FailMode { - Healthy, // Pass through to inner - FailEveryNth(usize), // Fail on every Nth call - FailAfterN(usize), // Succeed N times, then always fail - TimeoutMs(u64), // Sleep then fail (simulates hung NFS) - PartialRead { max_bytes: usize }, // Return truncated data - ReturnError(io::ErrorKind), // Return specific error -} -``` - -Implements `Origin` trait. `fail_mode` is `Arc>` so tests can change behavior mid-test (e.g., origin "recovers" after health check). - -#### 4.3.2 FaultyCasStore - -Wraps `CasStore` with injectable disk errors: - -```rust -pub struct FaultyCasStore { - inner: CasStore, - inject_enospc: AtomicBool, // put() fails with ENOSPC - inject_eio_on_read: AtomicBool, // get() fails with EIO - inject_corruption: AtomicBool, // get() returns bad data -} -``` - -#### 4.3.3 Centralized Fixtures - -Currently duplicated across 29 test modules: - -```rust -pub fn make_file_meta(id: i64, vpath: &str, size: u64) -> FileMeta; -pub fn make_audio_meta(artist: &str, album: &str, title: &str) -> AudioMeta; -pub async fn setup_test_cas(dir: &Path) -> Arc; -pub fn setup_test_tree(files: &[FileMeta]) -> Arc>; -``` - -### 4.4 Failpoints Instrumentation - -Production code locations that need `fail_point!` macros: - -| Location | Failpoint Name | Simulates | -|----------|---------------|-----------| -| `musicfs-cas/src/store.rs` `put()` | `cas-put-before-write` | ENOSPC before chunk write | -| `musicfs-cas/src/store.rs` `put()` | `cas-put-after-write-before-index` | Crash between write and sled insert | -| `musicfs-cas/src/reader.rs` `get_or_fetch_manifest()` | `reader-manifest-fetch` | Manifest fetch failure | -| `musicfs-sync/src/delta.rs` `detect_changes()` | `delta-sync-after-batch` | Crash mid-sync | -| `musicfs-origins/src/health.rs` `check_one()` | `health-check-hang` | Health check hangs forever | -| `musicfs-cache/src/db.rs` `open()` | `db-open-corrupt` | Database corruption on open | - -All guarded by `#[cfg(feature = "failpoints")]` — zero-cost in release builds. - -### 4.5 Test File Organization - -``` -musicfs/ -├── crates/ -│ └── musicfs-test-utils/ # NEW — shared test helpers -│ ├── Cargo.toml -│ └── src/ -│ ├── lib.rs -│ ├── faulty_origin.rs # FaultyOrigin with FailMode -│ ├── faulty_cas.rs # FaultyCasStore -│ ├── fixtures.rs # make_file_meta, setup_test_cas, etc. -│ └── assertions.rs # Custom assertions -├── tests/ -│ ├── resilience/ # NEW — resilience test suite -│ │ ├── mod.rs -│ │ ├── signal_handling.rs # SIGTERM/SIGINT/double-signal -│ │ ├── crash_recovery.rs # Fork-kill + state verification -│ │ ├── cache_corruption.rs # SQLite/sled/tantivy/CAS corruption -│ │ ├── disk_failure.rs # ENOSPC, permissions, passthrough mode -│ │ ├── resource_limits.rs # fd exhaustion, memory limits -│ │ └── lock_poisoning.rs # RwLock poison recovery -│ ├── failpoints/ # NEW — failpoint-gated tests -│ │ ├── mod.rs -│ │ ├── origin_failures.rs # Injected origin errors -│ │ ├── sync_interruption.rs # Delta sync crash/resume -│ │ └── cas_failures.rs # CAS write failures -│ ├── integration/ # NEW — network integration (Docker) -│ │ ├── docker-compose.yml -│ │ ├── network_faults.rs # Toxiproxy: latency, drops, bandwidth -│ │ └── origin_failover.rs # Multi-origin failover integration -│ └── e2e/ -│ └── e2e_players.rs # Existing (unchanged) -``` - -**Running**: -```bash -# Layer 1: Fast resilience tests (no Docker, no FUSE) -cargo test --lib --tests resilience - -# Layer 1: Failpoint tests (sequential, feature-gated) -cargo test --features failpoints --test failpoints -- --test-threads 1 - -# Layer 2: Process-level tests (included in resilience/) -cargo test --test resilience - -# Layer 3: Network integration (requires docker-compose up) -cargo test --test integration -- --ignored - -# All layers -cargo nextest run --features failpoints -``` - -### 4.6 Integration Test Docker Setup - -For Layer 3 network fault testing: - -```yaml -# tests/integration/docker-compose.yml -services: - toxiproxy: - image: ghcr.io/shopify/toxiproxy:2.9.0 - ports: - - "8474:8474" # Toxiproxy API - - "20000-20010:20000-20010" # Proxy ports - - minio: - image: minio/minio - command: server /data - ports: - - "9000:9000" - environment: - MINIO_ROOT_USER: test - MINIO_ROOT_PASSWORD: testtest - - sftp: - image: atmoz/sftp - ports: - - "2222:22" - command: test:test:::music -``` - -Tests use `noxious-client` crate to configure Toxiproxy faults at runtime (latency injection, connection drops, bandwidth throttling). - ---- - -## 5. Cross-Cutting Concerns - -### 5.1 Security & Privacy - -- Tests run without root — no kernel modules, no privileged containers for Layer 1/2 -- Layer 3 Docker tests use ephemeral containers with test credentials only -- No real music files or user data in tests — synthetic `make_file_meta()` fixtures -- `rlimit` tests restore original limits after test (cleanup in all code paths) - -### 5.2 Observability - -- Failpoint tests log injected faults via `tracing` — test failures include full trace context -- Layer 2 (fork-kill) tests capture daemon stdout/stderr for failure diagnosis -- Test coverage tracked per resilience issue (coverage matrix in Section 7) - -### 5.3 Scalability & Performance - -- Layer 1 tests: <10ms each, ~25 tests = <1s total -- Layer 2 tests: ~2-5s each (process spawn + signal + verify), ~5 tests = <30s total -- Layer 3 tests: ~5-10s each (Docker network), ~5 tests = <60s total -- Full suite: <2 minutes including failpoint tests (sequential `--test-threads 1`) -- Failpoint global state requires `--test-threads 1` for failpoint tests; all other tests parallelize normally - -### 5.4 Testing the Tests - -- Corruption tests self-validate: create known-good state → corrupt → verify detection -- FaultyOrigin has mode assertions: `assert_eq!(origin.call_count(), expected)` to verify injection triggered -- Failpoint tests verify both the error path AND the happy path (remove failpoint, retry, verify success) -- Resource limit tests always restore original limits (even on panic — use scopeguard or Drop impl) - ---- - -## 6. Alternatives Considered - -### 6.1 Jepsen / Full Chaos Engineering Framework - -**Rejected.** Jepsen tests distributed consensus under network partitions. MusicFS is a single-daemon filesystem — its failure modes are local (disk, signals, panics), not distributed. The 3-layer approach covers our actual failure surface with 10x less complexity. - -### 6.2 proptest / Property-Based Testing - -**Deferred.** Property-based testing (random input generation) is valuable for finding edge cases in path resolution, CDC chunking, and search queries. But it's orthogonal to resilience testing — it tests correctness under random input, not correctness under infrastructure failure. Can be added later without affecting this design. - -### 6.3 loom (Concurrency Model Checker) - -**Deferred.** loom exhaustively checks all possible thread interleavings for data races and deadlocks. It would be useful for the FUSE↔tokio deadlock issue (5.1) and RwLock poison issue (2.9). However, loom requires rewriting code to use `loom::sync` primitives and is very slow. Not practical for initial resilience coverage. Consider for Phase E hardening. - -### 6.4 In-Process Failure Injection Without Failpoints - -**Rejected.** Alternative: inject failures via trait-method overrides or runtime flags instead of the `fail` crate. This avoids a new dependency but requires modifying every function signature to accept an error injection parameter. Failpoints are cleaner — they're invisible in production (compiled out) and don't pollute the API surface. - -### 6.5 Mock Framework (mockall) - -**Rejected for now.** The codebase uses real implementations with TempDir isolation — this pattern is well-established across 43 test files. Introducing `mockall` would split the test codebase into two incompatible patterns. `FaultyOrigin` wrapper achieves the same result while staying consistent with existing patterns. - ---- - -## 7. Implementation Plan - -### Phase 1: Test Infrastructure (Days 1-2.5) - -| Task | Effort | Deliverable | -|------|--------|-------------| -| Create `musicfs-test-utils` crate | 1 day | `FaultyOrigin`, `FaultyCasStore`, centralized fixtures | -| Add `fail` crate, instrument 10 failpoints | 1 day | Failpoint macros in store.rs, reader.rs, delta.rs, health.rs, db.rs | -| Setup test directory structure | 0.5 day | `tests/resilience/`, `tests/failpoints/`, `tests/integration/` | - -### Phase 2: Layer 1 Tests (Days 3-5.5) - -| Test Group | Tests | Effort | Can Write Now? | -|------------|-------|--------|----------------| -| Cache corruption (SQLite, sled, tantivy, CAS) | 4 | 0.5 day | ✅ Yes | -| RwLock poison recovery | 2 | 0.25 day | ✅ Yes | -| Health check timeout + parallel checks | 2 | 0.25 day | ✅ Yes | -| tantivy crash recovery | 2 | 0.25 day | ✅ Yes | -| fd exhaustion | 1 | 0.25 day | ✅ Yes | -| Disk space / ENOSPC | 2 | 0.25 day | ✅ Yes | -| Origin failover (FaultyOrigin) | 3 | 0.5 day | ✅ Yes | -| Panic hook + task supervisor | 3 | 0.5 day | ❌ Needs implementation | -| Shutdown orchestration | 3 | 0.5 day | ❌ Needs implementation | -| sd_notify mock socket | 1 | 0.25 day | ❌ Needs implementation | -| Passthrough mode | 1 | 0.25 day | ❌ Needs implementation | -| Systemd service file assertions | 1 | 0.1 day | ✅ Yes | - -### Phase 3: Layer 2 Tests (Days 6-7) - -| Test | Effort | Requires | -|------|--------|----------| -| SIGTERM triggers clean shutdown | 0.25 day | Signal handler implementation | -| SIGINT triggers clean shutdown | 0.1 day | Signal handler implementation | -| Double-signal forces immediate exit | 0.1 day | Signal handler implementation | -| Kill -9 + stale mount detection | 0.25 day | Stale mount detection implementation | -| 100 concurrent FUSE reads (deadlock) | 0.25 day | FUSE mount in test (Docker or privileged) | - -### Phase 4: Layer 3 Tests (Days 8-9) - -| Task | Effort | Requires | -|------|--------|----------| -| Docker Compose setup (Toxiproxy + MinIO + SFTP) | 0.5 day | Docker | -| S3 latency spike test | 0.25 day | S3 origin implementation | -| S3 connection drop + failover | 0.25 day | S3 origin implementation | -| SFTP connection drop + failover | 0.25 day | SFTP origin implementation | -| Origin recovery after partition heal | 0.25 day | Docker | - -### Rollout - -1. **Phase 1 first** — test infrastructure is prerequisite for everything else -2. **Phase 2 "write now" tests** — 11 tests that can be written before resilience implementation; they document expected behavior as executable specs (currently failing) -3. **Phase 2 remaining** — written alongside resilience implementation (test-first development) -4. **Phase 3** — after signal handling and shutdown are implemented -5. **Phase 4** — after S3/SFTP origins are implemented; deferred if origins remain stubs - ---- - -## 8. Coverage Matrix - -### 8.1 Issue → Test → Layer Mapping - -| Issue | Description | Layer | Test Approach | Write Now? | -|-------|-------------|-------|--------------|------------| -| 2.1 | Signal handling | 2 | Fork daemon + send SIGTERM/SIGINT | ❌ | -| 2.2 | Panic hook | 1 | `catch_unwind` + log capture | ❌ | -| 2.3 | Shutdown orchestration | 1+2 | CancellationToken + ordered teardown | ❌ | -| 2.4 | Cache integrity on startup | 1 | Corrupt file bytes + reopen | ✅ | -| 2.5 | Interrupted sync | 1 | Failpoint `delta-sync-after-batch` | ❌ | -| 2.6 | Task supervisor | 1 | Spawn panicking task + verify restart | ❌ | -| 2.7 | FUSE unmount on crash | 2 | Fork + kill -9 + check /proc/mounts | ❌ | -| 2.8 | Disk space | 1 | Small `max_size` + oversized write | ✅ | -| 2.9 | RwLock poison | 1 | Panic in writer thread + verify read | ✅ | -| 2.10 | sd_notify | 1 | Mock Unix datagram socket | ❌ | -| 3.1 | Watchdog | 1 | Mock sd_notify + verify WATCHDOG=1 | ❌ | -| 3.5 | sled recovery | 1 | Corrupt sled files + reopen | ✅ | -| 3.7 | ExecStop stub | 1 | Assert service file contains fusermount | ✅ | -| 3.8 | FUSE read timeout | 1 | FaultyOrigin with TimeoutMs + verify EIO | ✅ | -| 4.2.1 | Health check timeout | 1 | FaultyOrigin with 30s hang + timer | ✅ | -| 4.2.2 | Parallel health checks | 1 | 3 origins (2 fast, 1 slow) + timer | ✅ | -| 4.2.3 | Offline mode | 1 | All origins fail + verify state machine | ❌ | -| 5.1 | FUSE↔tokio deadlock | 2 | 100 concurrent reads with timeout | ✅ | -| 5.2 | tantivy crash | 1 | Write + `mem::forget` + reopen | ✅ | -| 5.3 | fd exhaustion | 1 | `rlimit` NOFILE=64 + CAS operations | ✅ | -| 5.7 | CAS atomic write | 1 | Failpoint between write and index | ❌ | -| 6.3 | sled dies at runtime | 1 | Corrupt sled + verify EIO not panic | ✅ | -| 6.4 | CAS chunk corruption | 1 | Overwrite chunk file + verify auto-repair | ✅ | -| 6.6 | Passthrough mode | 1 | Read-only cache dir + verify origin read | ❌ | -| Network | Origin failover | 1+3 | FaultyOrigin + Toxiproxy | ✅ (L1) | - -### 8.2 Summary - -- **Total test cases**: ~35 -- **Can write now** (before resilience implementation): 15 -- **Need implementation first**: 12 -- **Need Docker** (Layer 3 only): 5 -- **Need FUSE mount** (Layer 2): 3 - ---- - -## 9. Glossary / References - -### 9.1 Libraries - -| Library | Link | Purpose | -|---------|------|---------| -| `fail` (TiKV failpoints) | [github.com/tikv/fail-rs](https://github.com/tikv/fail-rs) | Conditional fault injection | -| `rlimit` | [docs.rs/rlimit](https://docs.rs/rlimit) | Resource limit manipulation | -| `nix` | [docs.rs/nix](https://docs.rs/nix) | POSIX signal sending | -| `wiremock` | [docs.rs/wiremock](https://docs.rs/wiremock) | HTTP mock server | -| `assert_cmd` | [docs.rs/assert_cmd](https://docs.rs/assert_cmd) | CLI process testing | -| Toxiproxy | [github.com/Shopify/toxiproxy](https://github.com/Shopify/toxiproxy) | Network fault injection proxy | -| `noxious-client` | [docs.rs/noxious-client](https://docs.rs/noxious-client) | Async Toxiproxy Rust client | - -### 9.2 References - -| Document | Path | -|----------|------| -| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) | -| Persistent state plan | [persistent-state.md](persistent-state.md) | -| Architecture | [architecture.md](../architecture.md) | -| Requirements | [requirements.md](../requirements.md) | - -### 9.3 Glossary - -| Term | Definition | -|------|------------| -| **Failpoint** | A conditional injection point in production code, compiled out in release builds | -| **FaultyOrigin** | Test wrapper around `Origin` trait that injects configurable errors | -| **Layer 1** | In-process tests (trait mocks, failpoints) — fastest, no external deps | -| **Layer 2** | Process-level tests (fork, signal, kill) — tests daemon lifecycle | -| **Layer 3** | Network-level tests (Toxiproxy, Docker) — tests real protocol behavior | -| **Passthrough mode** | Operating mode where cache is bypassed; reads go directly to origin | - ---- - -## Appendix A: Test Code Examples - -Reference implementations for each test case. These serve as executable specifications — tests can be written before the resilience features are implemented (they will fail until the feature lands). - -### A.1 Signal Handling (Issue 2.1) - -```rust -// tests/resilience/signal_handling.rs - -#[tokio::test] -async fn test_sigterm_triggers_shutdown() { - let mut child = Command::new(env!("CARGO_BIN_EXE_musicfs")) - .args(["mount", "--origin", &test_dir, &mount_dir]) - .spawn().unwrap(); - - wait_for_mount(&mount_dir).await; - - nix::sys::signal::kill( - nix::unistd::Pid::from_raw(child.id() as i32), - nix::sys::signal::Signal::SIGTERM, - ).unwrap(); - - let status = tokio::time::timeout( - Duration::from_secs(10), child.wait() - ).await.unwrap().unwrap(); - assert!(status.success() || status.code() == Some(0)); - assert!(!is_mounted(&mount_dir)); -} - -#[tokio::test] -async fn test_double_signal_forces_immediate_exit() { - // Send SIGTERM, then SIGTERM again within 1s - // Verify daemon exits immediately on second signal -} -``` - -### A.2 Panic Hook (Issue 2.2) - -```rust -#[tokio::test] -async fn test_panic_in_background_task_is_logged() { - let (subscriber, logs) = test_subscriber(); - - let handle = tokio::spawn(async { - panic!("test panic in background task"); - }); - - let result = handle.await; - assert!(result.is_err()); - assert!(logs.contains("test panic in background task")); -} - -#[test] -fn test_panic_hook_includes_backtrace() { - install_panic_hook(); - let result = std::panic::catch_unwind(|| { - panic!("deliberate test panic"); - }); - assert!(result.is_err()); -} -``` - -### A.3 Graceful Shutdown Orchestration (Issue 2.3) - -```rust -#[tokio::test] -async fn test_shutdown_order() { - let events = Arc::new(Mutex::new(Vec::::new())); - let token = CancellationToken::new(); - - let watcher_events = events.clone(); - let watcher_token = token.clone(); - tokio::spawn(async move { - watcher_token.cancelled().await; - watcher_events.lock().unwrap().push("watcher_stopped".into()); - }); - - let indexer_events = events.clone(); - let indexer_token = token.clone(); - tokio::spawn(async move { - indexer_token.cancelled().await; - indexer_events.lock().unwrap().push("indexer_stopped".into()); - }); - - token.cancel(); - tokio::time::sleep(Duration::from_millis(100)).await; - - let order = events.lock().unwrap(); - assert!(order.contains(&"watcher_stopped".to_string())); - assert!(order.contains(&"indexer_stopped".to_string())); -} - -#[tokio::test] -async fn test_shutdown_flushes_tantivy() { - let dir = TempDir::new().unwrap(); - let index = SearchIndex::open(dir.path()).unwrap(); - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - index.commit().unwrap(); - - let index2 = SearchIndex::open(dir.path()).unwrap(); - let results = index2.search("a", 10).unwrap(); - assert_eq!(results.len(), 1); -} -``` - -### A.4 Cache Integrity on Startup (Issue 2.4) - -```rust -#[tokio::test] -async fn test_sqlite_integrity_check_detects_corruption() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("test.db"); - - { - let db = Database::open(&db_path).unwrap(); - db.upsert_file(/* ... */).unwrap(); - } - - let mut data = std::fs::read(&db_path).unwrap(); - if data.len() > 200 { data[100..200].fill(0xFF); } - std::fs::write(&db_path, &data).unwrap(); - - let result = Database::open_with_integrity_check(&db_path); - assert!(matches!(result, Err(Error::DatabaseCorrupted(_)))); -} - -#[tokio::test] -async fn test_tantivy_corruption_triggers_rebuild() { - let dir = TempDir::new().unwrap(); - { - let index = SearchIndex::open(dir.path()).unwrap(); - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - index.commit().unwrap(); - } - - std::fs::write(dir.path().join("meta.json"), b"corrupted").unwrap(); - - let index = SearchIndex::open_with_recovery(dir.path()).unwrap(); - let results = index.search("a", 10).unwrap(); - assert_eq!(results.len(), 0); // Rebuilt empty but functional -} - -#[tokio::test] -async fn test_sled_corruption_triggers_repair() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { chunks_dir: dir.path().join("chunks"), ..Default::default() }; - - { - let store = CasStore::open(config.clone()).await.unwrap(); - store.put(b"test data").await.unwrap(); - } - - for entry in std::fs::read_dir(dir.path().join("chunks/index.sled")).unwrap() { - let entry = entry.unwrap(); - if entry.path().extension().is_some() { - std::fs::write(entry.path(), b"corrupted").unwrap(); - } - } - - let result = CasStore::open(config).await; - // Either succeeds with repair, or returns clear error -} -``` - -### A.5 Interrupted Sync Recovery (Issue 2.5) - -```rust -#[tokio::test] -#[cfg(feature = "failpoints")] -async fn test_sync_resumes_after_crash() { - let dir = TempDir::new().unwrap(); - - fail::cfg("delta-sync-after-batch", "50*off->return").unwrap(); - let detector = DeltaDetector::new(dir.path()); - let result = detector.detect_changes(&origin).await; - assert!(result.is_err()); - - fail::remove("delta-sync-after-batch"); - let result = detector.detect_changes(&origin).await; - assert!(result.is_ok()); -} -``` - -### A.6 Task Supervisor (Issue 2.6) - -```rust -#[tokio::test] -async fn test_task_supervisor_detects_panic() { - let supervisor = TaskSupervisor::new(); - supervisor.spawn_supervised("test_task", async { - tokio::time::sleep(Duration::from_millis(100)).await; - panic!("deliberate task panic"); - }); - - tokio::time::sleep(Duration::from_millis(200)).await; - let status = supervisor.task_status("test_task"); - assert!(matches!(status, TaskStatus::Failed { .. })); -} - -#[tokio::test] -async fn test_task_supervisor_restarts_critical_task() { - let call_count = Arc::new(AtomicU32::new(0)); - let count = call_count.clone(); - - let supervisor = TaskSupervisor::new(); - supervisor.spawn_critical("health_monitor", move || { - let count = count.clone(); - async move { - count.fetch_add(1, Ordering::SeqCst); - if count.load(Ordering::SeqCst) == 1 { - panic!("first run fails"); - } - loop { tokio::time::sleep(Duration::from_secs(60)).await; } - } - }); - - tokio::time::sleep(Duration::from_secs(2)).await; - assert_eq!(call_count.load(Ordering::SeqCst), 2); - assert!(matches!(supervisor.task_status("health_monitor"), TaskStatus::Running)); -} -``` - -### A.7 FUSE Unmount on Crash (Issue 2.7) - -```rust -#[test] -fn test_systemd_service_has_execstoppost() { - let service = std::fs::read_to_string("dist/musicfs.service").unwrap(); - assert!(service.contains("ExecStopPost")); - assert!(service.contains("fusermount")); -} -``` - -### A.8 Disk Space Handling (Issue 2.8) - -```rust -#[tokio::test] -async fn test_cas_put_handles_enospc() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - max_size: 1024, - ..Default::default() - }; - let store = CasStore::open(config).await.unwrap(); - - let big_data = vec![0u8; 2048]; - let result = store.put(&big_data).await; - assert!(result.is_err() || store.current_size() <= 1024); -} -``` - -### A.9 RwLock Poison Recovery (Issue 2.9) - -```rust -#[test] -fn test_poisoned_tree_lock_returns_eio_not_panic() { - let tree = Arc::new(std::sync::RwLock::new(VirtualTree::new())); - - let tree_clone = tree.clone(); - let _ = std::thread::spawn(move || { - let _guard = tree_clone.write().unwrap(); - panic!("poisoning the lock"); - }).join(); - - assert!(tree.read().is_err()); -} - -#[test] -fn test_parking_lot_rwlock_survives_panic() { - let tree = Arc::new(parking_lot::RwLock::new(VirtualTree::new())); - - let tree_clone = tree.clone(); - let _ = std::thread::spawn(move || { - let _guard = tree_clone.write(); - panic!("writer panic"); - }).join(); - - let guard = tree.read(); - assert!(guard.get(ROOT_INODE).is_some()); -} -``` - -### A.10 sd_notify Integration (Issue 2.10) - -```rust -#[test] -fn test_sd_notify_ready_sent() { - let dir = TempDir::new().unwrap(); - let socket_path = dir.path().join("notify.sock"); - std::env::set_var("NOTIFY_SOCKET", &socket_path); - - let listener = std::os::unix::net::UnixDatagram::bind(&socket_path).unwrap(); - sd_notify::notify(false, &[sd_notify::NotifyState::Ready]).unwrap(); - - let mut buf = [0u8; 256]; - let n = listener.recv(&mut buf).unwrap(); - let msg = std::str::from_utf8(&buf[..n]).unwrap(); - assert!(msg.contains("READY=1")); -} -``` - -### A.11 Origin Failover (Issues 4.2.1, 4.2.2) - -```rust -#[tokio::test] -async fn test_failover_on_primary_death() { - let primary = Arc::new(FaultyOrigin::new( - LocalOrigin::new("primary", &primary_dir), - FailMode::ReturnError(io::ErrorKind::ConnectionRefused), - )); - let secondary = Arc::new(LocalOrigin::new("secondary", &secondary_dir)); - - let registry = OriginRegistry::new(/* ... */); - registry.register(primary, 1); - registry.register(secondary, 2); - - let executor = FailoverExecutor::new(registry, RetryConfig::default()); - let result = executor.read_with_failover(&path, 0, 100).await; - assert!(result.is_ok()); -} - -#[tokio::test] -async fn test_origin_recovery_resumes_routing() { - let origin = Arc::new(FaultyOrigin::new( - LocalOrigin::new("test", &dir), - FailMode::FailAfterN(0), - )); - - monitor.add_origin(origin.clone()); - monitor.check_now(&id).await; - assert!(monitor.snapshot().is_unhealthy(&id)); - - origin.set_mode(FailMode::Healthy); - monitor.check_now(&id).await; - assert!(monitor.snapshot().is_healthy(&id)); -} - -#[tokio::test] -async fn test_local_origin_health_check_has_timeout() { - let origin = Arc::new(FaultyOrigin::new( - LocalOrigin::new("slow", &dir), - FailMode::TimeoutMs(30_000), - )); - - let monitor = HealthMonitor::new(Duration::from_secs(30)); - monitor.add_origin(origin); - - let start = Instant::now(); - monitor.check_now(&OriginId::from("slow")).await; - assert!(start.elapsed() < Duration::from_secs(10)); - assert!(monitor.snapshot().is_unhealthy(&OriginId::from("slow"))); -} - -#[tokio::test] -async fn test_health_checks_run_in_parallel() { - let monitor = HealthMonitor::new(Duration::from_secs(30)); - monitor.add_origin(healthy_origin_1); - monitor.add_origin(healthy_origin_2); - monitor.add_origin(dead_origin); - - let start = Instant::now(); - monitor.check_all().await; - assert!(start.elapsed() < Duration::from_secs(8)); - - let snapshot = monitor.snapshot(); - assert!(snapshot.is_healthy(&healthy_1_id)); - assert!(snapshot.is_healthy(&healthy_2_id)); -} -``` - -### A.12 FUSE↔tokio Deadlock (Issue 5.1) - -```rust -#[tokio::test] -async fn test_concurrent_fuse_reads_dont_deadlock() { - let mount_dir = TempDir::new().unwrap(); - let session = spawn_test_mount(mount_dir.path()).await; - - let handles: Vec<_> = (0..100).map(|i| { - let path = mount_dir.path().join(format!("Artist/Album/{:02} - Track.flac", i)); - tokio::spawn(async move { - tokio::time::timeout(Duration::from_secs(30), tokio::fs::read(&path)).await - }) - }).collect(); - - for handle in handles { - let result = handle.await.unwrap(); - assert!(result.is_ok(), "read timed out — possible deadlock"); - } - drop(session); -} -``` - -### A.13 tantivy Crash Recovery (Issue 5.2) - -```rust -#[test] -fn test_tantivy_survives_uncommitted_crash() { - let dir = TempDir::new().unwrap(); - - { - let index = SearchIndex::open(dir.path()).unwrap(); - index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap(); - index.commit().unwrap(); - index.index_file(&make_file_meta(2, "/b.flac", 1000)).unwrap(); - std::mem::forget(index); // Simulate crash - } - - let index = SearchIndex::open(dir.path()).unwrap(); - assert_eq!(index.search("a", 10).unwrap().len(), 1); // Committed survives - assert_eq!(index.search("b", 10).unwrap().len(), 0); // Uncommitted lost -} -``` - -### A.14 File Descriptor Exhaustion (Issue 5.3) - -```rust -#[test] -#[cfg(target_os = "linux")] -fn test_fd_exhaustion_handling() { - use rlimit::{Resource, setrlimit, getrlimit}; - - let (orig_soft, orig_hard) = getrlimit(Resource::NOFILE).unwrap(); - setrlimit(Resource::NOFILE, 64, 64).unwrap(); - - let dir = TempDir::new().unwrap(); - // Attempt CAS operations under tight fd limit - // Should fail gracefully, not panic - - setrlimit(Resource::NOFILE, orig_soft, orig_hard).unwrap(); -} -``` - -### A.15 CAS Chunk Corruption + Auto-Repair (Issue 6.4) - -```rust -#[tokio::test] -async fn test_corrupt_chunk_auto_refetched() { - let dir = TempDir::new().unwrap(); - let store = Arc::new(CasStore::open(/* ... */).await.unwrap()); - - let data = b"valid audio data"; - let hash = store.put(data).await.unwrap(); - - let chunk_path = store.chunk_path(&hash); - std::fs::write(&chunk_path, b"corrupted garbage").unwrap(); - - let reader = FileReader::with_fetcher(store, fetcher); - let result = reader.read(file_id, 0, data.len() as u32).await; - assert!(result.is_ok()); - assert_eq!(&result.unwrap()[..], data); -} - -#[tokio::test] -async fn test_missing_chunk_triggers_origin_fetch() { - let hash = store.put(b"data").await.unwrap(); - std::fs::remove_file(store.chunk_path(&hash)).unwrap(); - - let result = reader.read(file_id, 0, 4).await; - assert!(result.is_ok()); -} -``` - -### A.16 Passthrough Mode (Issue 6.6) - -```rust -#[tokio::test] -async fn test_passthrough_mode_when_cache_disk_dead() { - let cache_dir = TempDir::new().unwrap(); - let origin_dir = TempDir::new().unwrap(); - std::fs::write(origin_dir.path().join("test.flac"), b"audio data").unwrap(); - - let store = CasStore::open(/* cache_dir */).await.unwrap(); - - std::fs::set_permissions( - cache_dir.path(), - std::fs::Permissions::from_mode(0o444), - ).unwrap(); - - let result = reader.read(file_id, 0, 10).await; - assert!(result.is_ok()); - assert_eq!(&result.unwrap()[..], b"audio data"); - - std::fs::set_permissions( - cache_dir.path(), - std::fs::Permissions::from_mode(0o755), - ).unwrap(); -} -``` - -### A.17 Toxiproxy Network Fault Tests (Layer 3) - -```rust -// tests/integration/network_faults.rs - -#[tokio::test] -#[ignore] // Requires docker-compose up -async fn test_s3_origin_survives_latency_spike() { - let toxi = noxious_client::Client::new("http://localhost:8474"); - - let proxy = toxi.create_proxy("minio", "0.0.0.0:20000", "minio:9000").await.unwrap(); - - let origin = S3Origin::new("http://localhost:20000", "test-bucket"); - let data = origin.read(Path::new("/test.flac"), 0, 100).await.unwrap(); - assert!(!data.is_empty()); - - proxy.add_toxic(&Toxic { - name: "latency".into(), - kind: ToxicKind::Latency { latency: 5000, jitter: 0 }, - direction: StreamDirection::Downstream, - toxicity: 1.0, - }).await.unwrap(); - - let start = Instant::now(); - let result = origin.read(Path::new("/test.flac"), 0, 100).await; - assert!(start.elapsed() < Duration::from_secs(35)); - - proxy.remove_toxic("latency").await.unwrap(); - let data = origin.read(Path::new("/test.flac"), 0, 100).await.unwrap(); - assert!(!data.is_empty()); -} - -#[tokio::test] -#[ignore] -async fn test_origin_connection_drop_triggers_failover() { - // Setup toxiproxy for primary origin - // Inject "down" toxic → connection refused - // Verify: requests routed to secondary origin - // Remove toxic → verify: primary re-enabled on next health check -} -``` diff --git a/docs/v2/plans/week-01-foundation.md b/docs/v2/plans/week-01-foundation.md deleted file mode 100644 index 8d5394d..0000000 --- a/docs/v2/plans/week-01-foundation.md +++ /dev/null @@ -1,1126 +0,0 @@ -# Week 1: Foundation - -**Phase**: 1 (MVP) -**Prerequisites**: None -**Estimated effort**: 5 days - ---- - -## Objective - -Set up Rust workspace, define core types, create FUSE skeleton, implement local origin plugin. - ---- - -## Deliverables - -| Task | Crate | Files | Done | -|------|-------|-------|------| -| Workspace setup | root | `Cargo.toml`, `.cargo/config.toml` | [ ] | -| Core types | musicfs-core | `lib.rs`, `error.rs`, `types.rs` | [ ] | -| Event Bus | musicfs-core | `events.rs` | [ ] | -| FUSE skeleton | musicfs-fuse | `lib.rs`, `filesystem.rs` | [ ] | -| Local origin | musicfs-origins | `lib.rs`, `local.rs`, `traits.rs` | [ ] | -| Nix flake | root | `flake.nix` | [ ] | - ---- - -## Task 1: Workspace Setup - -### 1.1 Create directory structure - -```bash -mkdir -p musicfs -cd musicfs -mkdir -p crates/{musicfs-core,musicfs-fuse,musicfs-cache,musicfs-cas,musicfs-sync,musicfs-origins,musicfs-metadata,musicfs-search,musicfs-plugins,musicfs-grpc,musicfs-cli} -mkdir -p proto tests/{integration,e2e} benches -``` - -### 1.2 Create root `Cargo.toml` - -```toml -[workspace] -resolver = "2" -members = ["crates/*"] - -[workspace.package] -version = "0.1.0" -edition = "2021" -license = "MIT OR Apache-2.0" -rust-version = "1.75" -authors = ["MusicFS Contributors"] -repository = "https://github.com/user/musicfs" - -[workspace.dependencies] -# Async runtime -tokio = { version = "1", features = ["full"] } -async-trait = "0.1" - -# Error handling -thiserror = "1" -anyhow = "1" - -# Serialization -serde = { version = "1", features = ["derive"] } -serde_json = "1" -rmp-serde = "1" # msgpack - -# Logging -tracing = "0.1" -tracing-subscriber = { version = "0.3", features = ["env-filter"] } - -# FUSE -fuser = "0.14" - -# Database -rusqlite = { version = "0.31", features = ["bundled"] } -sled = "0.34" - -# Hashing (per architecture 8.3) -xxhash-rust = { version = "0.8", features = ["xxh64"] } - -# Testing -tempfile = "3" -``` - -### 1.3 Create `.cargo/config.toml` - -```toml -[build] -rustflags = ["-C", "link-arg=-fuse-ld=lld"] - -[target.x86_64-unknown-linux-gnu] -linker = "clang" - -[alias] -t = "test" -c = "check" -b = "build" -``` - ---- - -## Task 2: Core Types (`musicfs-core`) - -### 2.1 Initialize crate - -```bash -cd crates/musicfs-core -cargo init --lib -``` - -### 2.2 Create `Cargo.toml` - -```toml -[package] -name = "musicfs-core" -version.workspace = true -edition.workspace = true - -[dependencies] -thiserror.workspace = true -serde.workspace = true -tokio = { workspace = true, features = ["sync"] } -xxhash-rust.workspace = true -``` - -### 2.3 Create `src/lib.rs` - -```rust -pub mod error; -pub mod types; -pub mod events; - -pub use error::{Error, Result}; -pub use types::*; -pub use events::{Event, EventBus}; -``` - -### 2.4 Create `src/error.rs` - -```rust -use thiserror::Error; - -#[derive(Error, Debug)] -pub enum Error { - #[error("I/O error: {0}")] - Io(#[from] std::io::Error), - - #[error("Origin not found: {0}")] - OriginNotFound(String), - - #[error("File not found: {0}")] - FileNotFound(String), - - #[error("Path resolution failed: {0}")] - PathResolution(String), - - #[error("Cache error: {0}")] - Cache(String), - - #[error("Database error: {0}")] - Database(String), - - #[error("NFS stale file handle")] - NfsStaleHandle, - - #[error("Operation not permitted (read-only filesystem)")] - ReadOnly, -} - -pub type Result = std::result::Result; -``` - -### 2.5 Create `src/types.rs` - -```rust -use serde::{Deserialize, Serialize}; -use std::path::PathBuf; -use std::time::SystemTime; - -/// Unique identifier for an origin -#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct OriginId(pub String); - -impl From<&str> for OriginId { - fn from(s: &str) -> Self { - Self(s.to_string()) - } -} - -/// Unique identifier for a file in the database -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct FileId(pub i64); - -/// Virtual path in metadata-organized tree -#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct VirtualPath(pub PathBuf); - -impl VirtualPath { - pub fn new(path: impl Into) -> Self { - Self(path.into()) - } - - pub fn as_path(&self) -> &std::path::Path { - &self.0 - } - - pub fn as_str(&self) -> &str { - self.0.to_str().unwrap_or("") - } -} - -/// Real path on origin storage -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -pub struct RealPath { - pub origin_id: OriginId, - pub path: PathBuf, -} - -/// Content-addressable hash (xxHash64 per architecture 8.3) -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct ContentHash(pub [u8; 8]); - -impl ContentHash { - pub fn from_bytes(data: &[u8]) -> Self { - use xxhash_rust::xxh64::xxh64; - Self(xxh64(data, 0).to_le_bytes()) - } -} - -/// Chunk-level hash (xxHash64) -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct ChunkHash(pub [u8; 8]); - -impl ChunkHash { - pub fn from_bytes(data: &[u8]) -> Self { - use xxhash_rust::xxh64::xxh64; - Self(xxh64(data, 0).to_le_bytes()) - } -} - -/// Audio format enumeration -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -pub enum AudioFormat { - Flac, - Mp3, - Opus, - Vorbis, - Aac, - Alac, - Wav, - Unknown, -} - -impl AudioFormat { - pub fn from_extension(ext: &str) -> Self { - match ext.to_lowercase().as_str() { - "flac" => Self::Flac, - "mp3" => Self::Mp3, - "opus" => Self::Opus, - "ogg" => Self::Vorbis, - "m4a" | "aac" => Self::Aac, - "wav" => Self::Wav, - _ => Self::Unknown, - } - } -} - -/// Audio metadata extracted from files -#[derive(Debug, Clone, Default, Serialize, Deserialize)] -pub struct AudioMeta { - pub title: Option, - pub artist: Option, - pub album: Option, - pub album_artist: Option, - pub genre: Option, - pub year: Option, - pub track: Option, // "track" per architecture 4.3.6 - pub disc: Option, // "disc" per architecture 4.3.6 - pub duration_ms: Option, - pub bitrate: Option, - pub sample_rate: Option, - pub format: AudioFormat, -} - -/// Complete file metadata -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct FileMeta { - pub id: FileId, - pub virtual_path: VirtualPath, - pub real_path: RealPath, - pub size: u64, - pub mtime: SystemTime, - pub content_hash: Option, - pub audio: Option, -} - -/// Directory entry for readdir -#[derive(Debug, Clone)] -pub struct DirEntry { - pub name: String, - pub is_dir: bool, - pub size: u64, - pub mtime: SystemTime, -} - -/// File stat information -#[derive(Debug, Clone)] -pub struct FileStat { - pub size: u64, - pub mtime: SystemTime, - pub is_dir: bool, -} - -/// Origin health status -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum HealthStatus { - Healthy, - Degraded, - Unhealthy, - Unknown, -} -``` - -### 2.6 Create `src/events.rs` - -```rust -use crate::types::{OriginId, VirtualPath}; -use tokio::sync::broadcast; - -/// Central event bus for system-wide notifications (per architecture 4.2) -pub struct EventBus { - sender: broadcast::Sender, -} - -impl EventBus { - pub fn new(capacity: usize) -> Self { - let (sender, _) = broadcast::channel(capacity); - Self { sender } - } - - pub fn publish(&self, event: Event) { - // Ignore error if no receivers - let _ = self.sender.send(event); - } - - pub fn subscribe(&self) -> broadcast::Receiver { - self.sender.subscribe() - } -} - -impl Default for EventBus { - fn default() -> Self { - Self::new(1024) - } -} - -/// System events -#[derive(Clone, Debug)] -pub enum Event { - FileAdded { - path: VirtualPath, - origin_id: OriginId, - }, - FileRemoved { - path: VirtualPath, - }, - FileModified { - path: VirtualPath, - }, - FileAccessed { - path: VirtualPath, - origin_id: OriginId, - offset: u64, - size: u32, - }, - OriginConnected { - origin_id: OriginId, - }, - OriginDisconnected { - origin_id: OriginId, - }, - SyncStarted { - origin_id: OriginId, - }, - SyncCompleted { - origin_id: OriginId, - files_changed: u64, - }, - CacheEviction { - bytes_freed: u64, - }, -} -``` - ---- - -## Task 3: FUSE Skeleton (`musicfs-fuse`) - -### 3.1 Create `Cargo.toml` - -```toml -[package] -name = "musicfs-fuse" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -fuser.workspace = true -tokio.workspace = true -tracing.workspace = true -``` - -### 3.2 Create `src/lib.rs` - -```rust -mod filesystem; - -pub use filesystem::MusicFs; -``` - -### 3.3 Create `src/filesystem.rs` - -```rust -use fuser::{ - FileAttr, FileType, Filesystem, ReplyAttr, ReplyData, ReplyDirectory, - ReplyEntry, ReplyOpen, Request, FUSE_ROOT_ID, -}; -use musicfs_core::{Error, Result}; -use std::ffi::OsStr; -use std::time::{Duration, SystemTime, UNIX_EPOCH}; -use tracing::{debug, error, info}; - -const TTL: Duration = Duration::from_secs(1); - -/// Main FUSE filesystem implementation -pub struct MusicFs { - // Will be populated in later weeks: - // origins: Arc, - // cache: Arc, - // tree: Arc>, -} - -impl MusicFs { - pub fn new() -> Self { - Self {} - } - - /// Mount the filesystem - pub fn mount(self, mountpoint: &std::path::Path) -> Result<()> { - info!("Mounting MusicFS at {:?}", mountpoint); - - let options = vec![ - fuser::MountOption::RO, // Read-only - fuser::MountOption::FSName("musicfs".to_string()), - fuser::MountOption::AutoUnmount, - fuser::MountOption::AllowOther, - ]; - - fuser::mount2(self, mountpoint, &options) - .map_err(|e| Error::Io(e))?; - - Ok(()) - } - - fn root_attr(&self) -> FileAttr { - FileAttr { - ino: FUSE_ROOT_ID, - size: 0, - blocks: 0, - atime: UNIX_EPOCH, - mtime: UNIX_EPOCH, - ctime: UNIX_EPOCH, - crtime: UNIX_EPOCH, - kind: FileType::Directory, - perm: 0o755, - nlink: 2, - uid: unsafe { libc::getuid() }, - gid: unsafe { libc::getgid() }, - rdev: 0, - blksize: 512, - flags: 0, - } - } -} - -impl Default for MusicFs { - fn default() -> Self { - Self::new() - } -} - -impl Filesystem for MusicFs { - fn init( - &mut self, - _req: &Request<'_>, - _config: &mut fuser::KernelConfig, - ) -> std::result::Result<(), libc::c_int> { - info!("MusicFS initialized"); - Ok(()) - } - - fn destroy(&mut self) { - info!("MusicFS destroyed"); - } - - fn lookup(&mut self, _req: &Request, parent: u64, name: &OsStr, reply: ReplyEntry) { - debug!("lookup(parent={}, name={:?})", parent, name); - - // TODO: Implement in Week 3 - reply.error(libc::ENOENT); - } - - fn getattr(&mut self, _req: &Request, ino: u64, reply: ReplyAttr) { - debug!("getattr(ino={})", ino); - - if ino == FUSE_ROOT_ID { - reply.attr(&TTL, &self.root_attr()); - } else { - // TODO: Implement in Week 3 - reply.error(libc::ENOENT); - } - } - - fn readdir( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - offset: i64, - mut reply: ReplyDirectory, - ) { - debug!("readdir(ino={}, offset={})", ino, offset); - - if ino == FUSE_ROOT_ID { - // Root directory with . and .. - if offset == 0 { - let _ = reply.add(FUSE_ROOT_ID, 1, FileType::Directory, "."); - } - if offset <= 1 { - let _ = reply.add(FUSE_ROOT_ID, 2, FileType::Directory, ".."); - } - // TODO: Add actual entries in Week 3 - reply.ok(); - } else { - // TODO: Implement in Week 3 - reply.error(libc::ENOENT); - } - } - - fn open(&mut self, _req: &Request, ino: u64, flags: i32, reply: ReplyOpen) { - debug!("open(ino={}, flags={})", ino, flags); - - // Check for write flags - we're read-only (FR-4.1) - let write_flags = libc::O_WRONLY | libc::O_RDWR | libc::O_APPEND | libc::O_TRUNC; - if flags & write_flags != 0 { - reply.error(libc::EROFS); - return; - } - - // TODO: Implement in Week 3 - reply.error(libc::ENOENT); - } - - fn read( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - offset: i64, - size: u32, - _flags: i32, - _lock_owner: Option, - reply: ReplyData, - ) { - debug!("read(ino={}, offset={}, size={})", ino, offset, size); - - // TODO: Implement in Week 3 - reply.error(libc::ENOENT); - } - - fn release( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - _flags: i32, - _lock_owner: Option, - _flush: bool, - reply: fuser::ReplyEmpty, - ) { - debug!("release(ino={})", ino); - reply.ok(); - } - - // Write operations - always return EROFS (FR-4.1-4.4) - - fn write( - &mut self, - _req: &Request, - _ino: u64, - _fh: u64, - _offset: i64, - _data: &[u8], - _write_flags: u32, - _flags: i32, - _lock_owner: Option, - reply: fuser::ReplyWrite, - ) { - reply.error(libc::EROFS); - } - - fn mkdir( - &mut self, - _req: &Request, - _parent: u64, - _name: &OsStr, - _mode: u32, - _umask: u32, - reply: ReplyEntry, - ) { - reply.error(libc::EROFS); - } - - fn unlink(&mut self, _req: &Request, _parent: u64, _name: &OsStr, reply: fuser::ReplyEmpty) { - reply.error(libc::EROFS); - } - - fn rmdir(&mut self, _req: &Request, _parent: u64, _name: &OsStr, reply: fuser::ReplyEmpty) { - reply.error(libc::EROFS); - } - - fn rename( - &mut self, - _req: &Request, - _parent: u64, - _name: &OsStr, - _newparent: u64, - _newname: &OsStr, - _flags: u32, - reply: fuser::ReplyEmpty, - ) { - reply.error(libc::EROFS); - } - - fn create( - &mut self, - _req: &Request, - _parent: u64, - _name: &OsStr, - _mode: u32, - _umask: u32, - _flags: i32, - reply: fuser::ReplyCreate, - ) { - reply.error(libc::EROFS); - } - - // Additional EROFS handlers (FR-4.5) - - fn setattr( - &mut self, - _req: &Request, - _ino: u64, - _mode: Option, - _uid: Option, - _gid: Option, - _size: Option, - _atime: Option, - _mtime: Option, - _ctime: Option, - _fh: Option, - _crtime: Option, - _chgtime: Option, - _bkuptime: Option, - _flags: Option, - reply: ReplyAttr, - ) { - reply.error(libc::EROFS); - } - - fn symlink( - &mut self, - _req: &Request, - _parent: u64, - _name: &OsStr, - _link: &std::path::Path, - reply: ReplyEntry, - ) { - reply.error(libc::EROFS); - } - - fn link( - &mut self, - _req: &Request, - _ino: u64, - _newparent: u64, - _newname: &OsStr, - reply: ReplyEntry, - ) { - reply.error(libc::EROFS); - } - - fn mknod( - &mut self, - _req: &Request, - _parent: u64, - _name: &OsStr, - _mode: u32, - _umask: u32, - _rdev: u32, - reply: ReplyEntry, - ) { - reply.error(libc::EROFS); - } -} -``` - ---- - -## Task 4: Local Origin (`musicfs-origins`) - -### 4.1 Create `Cargo.toml` - -```toml -[package] -name = "musicfs-origins" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -async-trait.workspace = true -tokio = { workspace = true, features = ["fs"] } -tracing.workspace = true -``` - -### 4.2 Create `src/lib.rs` - -```rust -mod traits; -mod local; - -pub use traits::Origin; -pub use local::LocalOrigin; -``` - -### 4.3 Create `src/traits.rs` - -```rust -use async_trait::async_trait; -use musicfs_core::{DirEntry, FileStat, HealthStatus, OriginId, Result}; -use std::path::Path; -use tokio::io::AsyncRead; - -/// Origin type enumeration -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum OriginType { - Local, - Nfs, - Smb, - S3, - Sftp, -} - -/// Origin plugin interface (per architecture 4.3.4) -#[async_trait] -pub trait Origin: Send + Sync { - /// Unique identifier for this origin - fn id(&self) -> &OriginId; - - /// Origin type - fn origin_type(&self) -> OriginType; - - /// Human-readable display name - fn display_name(&self) -> &str; - - /// List entries in directory - async fn readdir(&self, path: &Path) -> Result>; - - /// Get file/directory metadata - async fn stat(&self, path: &Path) -> Result; - - /// Read file content at offset - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result>; - - /// Check if path exists - async fn exists(&self, path: &Path) -> Result; - - /// Health check - async fn health(&self) -> HealthStatus; - - /// Get a reader for streaming large files - async fn open_read(&self, path: &Path) -> Result>; - - /// Watch path for changes (per architecture 4.3.4) - /// Returns handle that cancels watch on drop - async fn watch(&self, path: &Path, callback: WatchCallback) -> Result; -} - -/// Callback for file change notifications -pub type WatchCallback = Box; - -/// Handle to cancel a watch - cancels on drop -pub struct WatchHandle { - _cancel: tokio::sync::oneshot::Sender<()>, -} - -/// Watch event types -#[derive(Debug, Clone)] -pub enum WatchEvent { - Created(PathBuf), - Modified(PathBuf), - Deleted(PathBuf), -} -``` - -### 4.4 Create `src/local.rs` - -```rust -use crate::traits::{Origin, OriginType}; -use async_trait::async_trait; -use musicfs_core::{DirEntry, Error, FileStat, HealthStatus, OriginId, Result}; -use std::path::{Path, PathBuf}; -use tokio::fs; -use tokio::io::AsyncRead; -use tracing::debug; - -/// Local filesystem origin (FR-12.1) -pub struct LocalOrigin { - id: OriginId, - root: PathBuf, - display_name: String, -} - -impl LocalOrigin { - pub fn new(id: impl Into, root: impl Into) -> Self { - let root = root.into(); - let display_name = format!("Local: {}", root.display()); - Self { - id: id.into(), - root, - display_name, - } - } - - fn full_path(&self, path: &Path) -> PathBuf { - self.root.join(path) - } -} - -#[async_trait] -impl Origin for LocalOrigin { - fn id(&self) -> &OriginId { - &self.id - } - - fn origin_type(&self) -> OriginType { - OriginType::Local - } - - fn display_name(&self) -> &str { - &self.display_name - } - - async fn readdir(&self, path: &Path) -> Result> { - let full_path = self.full_path(path); - debug!("LocalOrigin::readdir({:?})", full_path); - - let mut entries = Vec::new(); - let mut dir = fs::read_dir(&full_path).await?; - - while let Some(entry) = dir.next_entry().await? { - let metadata = entry.metadata().await?; - let name = entry.file_name().to_string_lossy().into_owned(); - - entries.push(DirEntry { - name, - is_dir: metadata.is_dir(), - size: metadata.len(), - mtime: metadata.modified().unwrap_or(std::time::UNIX_EPOCH), - }); - } - - Ok(entries) - } - - async fn stat(&self, path: &Path) -> Result { - let full_path = self.full_path(path); - debug!("LocalOrigin::stat({:?})", full_path); - - let metadata = fs::metadata(&full_path).await?; - - Ok(FileStat { - size: metadata.len(), - mtime: metadata.modified().unwrap_or(std::time::UNIX_EPOCH), - is_dir: metadata.is_dir(), - }) - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result> { - use tokio::io::{AsyncReadExt, AsyncSeekExt}; - - let full_path = self.full_path(path); - debug!("LocalOrigin::read({:?}, offset={}, size={})", full_path, offset, size); - - let mut file = fs::File::open(&full_path).await?; - file.seek(std::io::SeekFrom::Start(offset)).await?; - - let mut buffer = vec![0u8; size as usize]; - let bytes_read = file.read(&mut buffer).await?; - buffer.truncate(bytes_read); - - Ok(buffer) - } - - async fn exists(&self, path: &Path) -> Result { - let full_path = self.full_path(path); - Ok(fs::try_exists(&full_path).await?) - } - - async fn health(&self) -> HealthStatus { - match fs::try_exists(&self.root).await { - Ok(true) => HealthStatus::Healthy, - Ok(false) => HealthStatus::Unhealthy, - Err(_) => HealthStatus::Unhealthy, - } - } - - async fn open_read(&self, path: &Path) -> Result> { - let full_path = self.full_path(path); - let file = fs::File::open(&full_path).await?; - Ok(Box::new(file)) - } - - async fn watch(&self, path: &Path, callback: WatchCallback) -> Result { - // Stub implementation for Week 1 - // Full inotify/notify implementation deferred to Week 5 (FR-10.2) - let (tx, mut rx) = tokio::sync::oneshot::channel(); - - // In Week 5: Use notify crate for real filesystem watching - // For now, just return a handle that does nothing - debug!("LocalOrigin::watch({:?}) - stub implementation", path); - - Ok(WatchHandle { _cancel: tx }) - } -} -``` - ---- - -## Task 5: Nix Flake - -### 5.1 Create `flake.nix` - -```nix -{ - description = "MusicFS - FUSE filesystem for music libraries"; - - inputs = { - nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; - rust-overlay.url = "github:oxalica/rust-overlay"; - flake-utils.url = "github:numtide/flake-utils"; - }; - - outputs = { self, nixpkgs, rust-overlay, flake-utils }: - flake-utils.lib.eachDefaultSystem (system: - let - overlays = [ (import rust-overlay) ]; - pkgs = import nixpkgs { inherit system overlays; }; - - rustToolchain = pkgs.rust-bin.stable.latest.default.override { - extensions = [ "rust-src" "rust-analyzer" ]; - }; - in - { - devShells.default = pkgs.mkShell { - buildInputs = with pkgs; [ - rustToolchain - pkg-config - fuse3 - sqlite - openssl - - # Development tools - cargo-watch - cargo-nextest - cargo-criterion - - # gRPC - protobuf - grpcurl - ]; - - RUST_BACKTRACE = 1; - RUST_LOG = "debug"; - }; - - packages.default = pkgs.rustPlatform.buildRustPackage { - pname = "musicfs"; - version = "0.1.0"; - src = ./.; - cargoLock.lockFile = ./Cargo.lock; - - nativeBuildInputs = [ pkgs.pkg-config ]; - buildInputs = [ pkgs.fuse3 pkgs.sqlite pkgs.openssl ]; - }; - } - ); -} -``` - ---- - -## Tests - -### Unit Tests - -Create `crates/musicfs-core/src/lib.rs` test module: - -```rust -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_content_hash() { - let data = b"hello world"; - let hash1 = ContentHash::from_bytes(data); - let hash2 = ContentHash::from_bytes(data); - assert_eq!(hash1, hash2); - - let hash3 = ContentHash::from_bytes(b"different"); - assert_ne!(hash1, hash3); - } - - #[test] - fn test_audio_format_from_extension() { - assert_eq!(AudioFormat::from_extension("flac"), AudioFormat::Flac); - assert_eq!(AudioFormat::from_extension("MP3"), AudioFormat::Mp3); - assert_eq!(AudioFormat::from_extension("unknown"), AudioFormat::Unknown); - } - - #[tokio::test] - async fn test_event_bus() { - let bus = EventBus::new(16); - let mut rx = bus.subscribe(); - - bus.publish(Event::SyncStarted { - origin_id: OriginId::from("test"), - }); - - let event = rx.recv().await.unwrap(); - assert!(matches!(event, Event::SyncStarted { .. })); - } -} -``` - -### Integration Tests - -Create `tests/integration/basic_mount.rs`: - -```rust -use musicfs_fuse::MusicFs; -use std::process::Command; -use tempfile::TempDir; - -#[test] -fn test_fuse_mount_unmount() { - let mount_dir = TempDir::new().unwrap(); - let mount_path = mount_dir.path(); - - // Fork and mount in child process - let fs = MusicFs::new(); - - // For now, just verify construction works - // Full mount test requires FUSE permissions - drop(fs); -} - -#[test] -fn test_read_only_enforcement() { - // Verify write operations return EROFS - // This will be expanded in Week 3 -} -``` - ---- - -## Exit Criteria - -- [ ] `cargo build` succeeds for all crates -- [ ] `cargo test` passes -- [ ] `nix develop` enters shell with all dependencies -- [ ] FUSE skeleton compiles with all required trait methods -- [ ] LocalOrigin can list files in a test directory -- [ ] Write operations return EROFS -- [ ] EventBus publishes and receives events - ---- - -## Verification Commands - -```bash -# Build all crates -cargo build --workspace - -# Run all tests -cargo test --workspace - -# Check for warnings -cargo clippy --workspace -- -D warnings - -# Enter nix shell -nix develop - -# Test local origin (manual) -cargo run --example local_origin_test -``` - ---- - -## Next Week - -Week 2 will implement metadata extraction using symphonia and create the SQLite schema. diff --git a/docs/v2/plans/week-02-metadata.md b/docs/v2/plans/week-02-metadata.md deleted file mode 100644 index a5611ef..0000000 --- a/docs/v2/plans/week-02-metadata.md +++ /dev/null @@ -1,771 +0,0 @@ -# Week 2: Metadata Extraction - -**Phase**: 1 (MVP) -**Prerequisites**: Week 1 (Foundation) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement audio metadata extraction using symphonia and create SQLite schema for metadata cache. - ---- - -## Deliverables - -| Task | Crate | Files | Done | -|------|-------|-------|------| -| Audio parsing | musicfs-metadata | `lib.rs`, `parser.rs` | [ ] | -| Format handlers | musicfs-metadata | `formats/*.rs` | [ ] | -| SQLite schema | musicfs-cache | `schema.sql`, `db.rs` | [ ] | -| Metadata cache | musicfs-cache | `metadata.rs` | [ ] | - ---- - -## Task 0: Extend AudioMeta in `musicfs-core` - -Add `lyrics` and `composer` fields to `AudioMeta` struct (FR-6.4): - -```rust -// In musicfs-core/src/types.rs, add to AudioMeta: -pub struct AudioMeta { - // ... existing fields ... - pub lyrics: Option, - pub composer: Option, -} -``` - ---- - -## Task 1: Metadata Parser (`musicfs-metadata`) - -### 1.1 Create `Cargo.toml` - -```toml -[package] -name = "musicfs-metadata" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -symphonia = { version = "0.5", features = ["all"] } -thiserror.workspace = true -tracing.workspace = true -``` - -### 1.2 Create `src/lib.rs` - -```rust -mod parser; - -pub use parser::MetadataParser; -``` - -### 1.3 Create `src/parser.rs` - -```rust -use musicfs_core::{AudioFormat, AudioMeta, Result, Error}; -use std::io::{Read, Seek}; -use std::path::Path; -use symphonia::core::codecs::CODEC_TYPE_NULL; -use symphonia::core::formats::FormatOptions; -use symphonia::core::io::MediaSourceStream; -use symphonia::core::meta::MetadataOptions; -use symphonia::core::probe::Hint; -use tracing::debug; - -/// Metadata extraction using symphonia (FR-6.1-6.5) -pub struct MetadataParser; - -impl MetadataParser { - pub fn new() -> Self { - Self - } - - /// Extract metadata from audio file - pub fn parse_file(&self, path: &Path) -> Result { - let file = std::fs::File::open(path)?; - let ext = path.extension() - .and_then(|e| e.to_str()) - .unwrap_or(""); - self.parse_reader(file, ext) - } - - /// Extract metadata from reader - pub fn parse_reader( - &self, - reader: R, - extension: &str, - ) -> Result { - let mss = MediaSourceStream::new(Box::new(reader), Default::default()); - - let mut hint = Hint::new(); - if !extension.is_empty() { - hint.with_extension(extension); - } - - let format_opts = FormatOptions { - enable_gapless: false, - ..Default::default() - }; - - let metadata_opts = MetadataOptions::default(); - - let probed = symphonia::default::get_probe() - .format(&hint, mss, &format_opts, &metadata_opts) - .map_err(|e| Error::Cache(format!("Failed to probe format: {}", e)))?; - - let mut format = probed.format; - let mut audio_meta = AudioMeta { - format: AudioFormat::from_extension(extension), - ..Default::default() - }; - - // Extract metadata from container - if let Some(metadata) = format.metadata().current() { - self.extract_tags(&mut audio_meta, metadata); - } - - // Also check probed metadata - if let Some(metadata) = probed.metadata.current() { - self.extract_tags(&mut audio_meta, metadata); - } - - // Get duration and codec info from track - if let Some(track) = format.tracks().iter().find(|t| t.codec_params.codec != CODEC_TYPE_NULL) { - let params = &track.codec_params; - - if let Some(n_frames) = params.n_frames { - if let Some(sample_rate) = params.sample_rate { - audio_meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64); - audio_meta.sample_rate = Some(sample_rate); - } - } - - if let Some(bits_per_sample) = params.bits_per_sample { - if let Some(sample_rate) = params.sample_rate { - if let Some(channels) = params.channels { - audio_meta.bitrate = Some( - bits_per_sample * sample_rate * channels.count() as u32 / 1000 - ); - } - } - } - } - - debug!("Parsed metadata: {:?}", audio_meta); - Ok(audio_meta) - } - - fn extract_tags(&self, meta: &mut AudioMeta, metadata: &symphonia::core::meta::MetadataRevision) { - use symphonia::core::meta::StandardTagKey; - - for tag in metadata.tags() { - if let Some(std_key) = tag.std_key { - let value = tag.value.to_string(); - match std_key { - StandardTagKey::TrackTitle => meta.title = Some(value), - StandardTagKey::Artist => meta.artist = Some(value), - StandardTagKey::Album => meta.album = Some(value), - StandardTagKey::AlbumArtist => meta.album_artist = Some(value), - StandardTagKey::Genre => meta.genre = Some(value), - StandardTagKey::TrackNumber => { - meta.track = value.split('/').next() - .and_then(|s| s.parse().ok()); - } - StandardTagKey::DiscNumber => { - meta.disc = value.split('/').next() - .and_then(|s| s.parse().ok()); - } - StandardTagKey::Date | StandardTagKey::ReleaseDate => { - meta.year = value.chars().take(4).collect::() - .parse().ok(); - } - StandardTagKey::Lyrics => { - meta.lyrics = Some(value); - } - StandardTagKey::Composer => { - meta.composer = Some(value); - } - _ => {} - } - } - } - } -} - -impl Default for MetadataParser { - fn default() -> Self { - Self::new() - } -} -``` - ---- - -## Task 2: Cache Database (`musicfs-cache`) - -### 2.1 Create `Cargo.toml` - -```toml -[package] -name = "musicfs-cache" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -rusqlite = { workspace = true, features = ["bundled"] } -sled.workspace = true -tokio.workspace = true -tracing.workspace = true -thiserror.workspace = true -serde.workspace = true -rmp-serde.workspace = true -``` - -### 2.2 Create `src/lib.rs` - -```rust -mod db; -mod metadata; - -pub use db::Database; -pub use metadata::MetadataCache; -``` - -### 2.3 Create `src/schema.sql` - -```sql --- MusicFS Metadata Cache Schema --- Per architecture.md section 4.3.6 --- NOTE: Chunk index stored in sled (chunks.sled/), NOT SQLite - -PRAGMA journal_mode = WAL; -PRAGMA foreign_keys = ON; -PRAGMA synchronous = NORMAL; - -CREATE TABLE IF NOT EXISTS files ( - id INTEGER PRIMARY KEY, - origin_id TEXT NOT NULL, - real_path TEXT NOT NULL, - virtual_path TEXT NOT NULL, - - -- Audio metadata (FR-6.1-6.5) - title TEXT, - artist TEXT, - album TEXT, - album_artist TEXT, - genre TEXT, - year INTEGER, - track INTEGER, - disc INTEGER, - duration_ms INTEGER, - bitrate INTEGER, - sample_rate INTEGER, - format TEXT, - - -- Sync state - origin_mtime INTEGER NOT NULL, - origin_size INTEGER NOT NULL, - content_hash TEXT, -- hex-encoded xxHash64 - chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)] - last_sync INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), - - UNIQUE(origin_id, real_path) -); - -CREATE TABLE IF NOT EXISTS artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE, - art_type TEXT NOT NULL, -- 'front', 'back', 'disc' - chunk_hash TEXT NOT NULL, -- hex-encoded reference to CAS - width INTEGER, - height INTEGER, - mime_type TEXT, - UNIQUE(file_id, art_type) -); - -CREATE TABLE IF NOT EXISTS collections ( - id INTEGER PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - query_json TEXT NOT NULL, -- smart collection query - created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), - updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')) -); - --- Indexes for performance (NFR-1.1, NFR-1.2) -CREATE INDEX IF NOT EXISTS idx_files_virtual ON files(virtual_path); -CREATE INDEX IF NOT EXISTS idx_files_artist_album ON files(artist, album); -CREATE INDEX IF NOT EXISTS idx_files_content_hash ON files(content_hash); -CREATE INDEX IF NOT EXISTS idx_files_real ON files(origin_id, real_path); -- FR-7.3 -CREATE INDEX IF NOT EXISTS idx_files_origin ON files(origin_id); -CREATE INDEX IF NOT EXISTS idx_files_last_sync ON files(last_sync); -CREATE INDEX IF NOT EXISTS idx_artwork_file ON artwork(file_id); -``` - -### 2.4 Create `src/db.rs` - -```rust -use musicfs_core::{AudioMeta, ContentHash, Error, FileId, FileMeta, OriginId, RealPath, Result, VirtualPath}; -use rusqlite::{params, Connection, OptionalExtension}; -use std::path::{Path, PathBuf}; -use std::sync::{Arc, Mutex}; -use std::time::{SystemTime, UNIX_EPOCH}; -use tracing::{debug, info}; - -const SCHEMA: &str = include_str!("schema.sql"); - -/// SQLite database connection manager -pub struct Database { - conn: Arc>, -} - -impl Database { - /// Open or create database at path - pub fn open(path: &Path) -> Result { - info!("Opening database at {:?}", path); - - let conn = Connection::open(path) - .map_err(|e| Error::Database(e.to_string()))?; - - // Execute schema - conn.execute_batch(SCHEMA) - .map_err(|e| Error::Database(e.to_string()))?; - - Ok(Self { - conn: Arc::new(Mutex::new(conn)), - }) - } - - /// Open in-memory database (for testing) - pub fn open_memory() -> Result { - let conn = Connection::open_in_memory() - .map_err(|e| Error::Database(e.to_string()))?; - - conn.execute_batch(SCHEMA) - .map_err(|e| Error::Database(e.to_string()))?; - - Ok(Self { - conn: Arc::new(Mutex::new(conn)), - }) - } - - /// Insert or update file metadata - pub fn upsert_file( - &self, - origin_id: &OriginId, - real_path: &Path, - virtual_path: &VirtualPath, - audio_meta: &AudioMeta, - origin_mtime: SystemTime, - origin_size: u64, - ) -> Result { - let conn = self.conn.lock().unwrap(); - - let mtime_secs = origin_mtime - .duration_since(UNIX_EPOCH) - .unwrap_or_default() - .as_secs() as i64; - - conn.execute( - r#" - INSERT INTO files ( - origin_id, real_path, virtual_path, - title, artist, album, album_artist, genre, - year, track, disc, - duration_ms, bitrate, sample_rate, format, - origin_mtime, origin_size - ) VALUES ( - ?1, ?2, ?3, - ?4, ?5, ?6, ?7, ?8, - ?9, ?10, ?11, - ?12, ?13, ?14, ?15, - ?16, ?17 - ) - ON CONFLICT(origin_id, real_path) DO UPDATE SET - virtual_path = excluded.virtual_path, - title = excluded.title, - artist = excluded.artist, - album = excluded.album, - album_artist = excluded.album_artist, - genre = excluded.genre, - year = excluded.year, - track = excluded.track, - disc = excluded.disc, - duration_ms = excluded.duration_ms, - bitrate = excluded.bitrate, - sample_rate = excluded.sample_rate, - format = excluded.format, - origin_mtime = excluded.origin_mtime, - origin_size = excluded.origin_size, - last_sync = strftime('%s', 'now') - "#, - params![ - &origin_id.0, - real_path.to_string_lossy(), - virtual_path.as_str(), - &audio_meta.title, - &audio_meta.artist, - &audio_meta.album, - &audio_meta.album_artist, - &audio_meta.genre, - &audio_meta.year, - &audio_meta.track, - &audio_meta.disc, - &audio_meta.duration_ms.map(|d| d as i64), - &audio_meta.bitrate, - &audio_meta.sample_rate, - format!("{:?}", audio_meta.format), - mtime_secs, - origin_size as i64, - ], - ).map_err(|e| Error::Database(e.to_string()))?; - - let id = conn.last_insert_rowid(); - debug!("Upserted file {} with id {}", virtual_path.as_str(), id); - - Ok(FileId(id)) - } - - /// Get file by virtual path - pub fn get_file_by_virtual_path(&self, path: &VirtualPath) -> Result> { - let conn = self.conn.lock().unwrap(); - - conn.query_row( - r#" - SELECT id, origin_id, real_path, virtual_path, - title, artist, album, album_artist, genre, - year, track, disc, - duration_ms, bitrate, sample_rate, format, - origin_mtime, origin_size, content_hash - FROM files - WHERE virtual_path = ?1 - "#, - params![path.as_str()], - |row| { - Ok(FileMeta { - id: FileId(row.get(0)?), - real_path: RealPath { - origin_id: OriginId(row.get(1)?), - path: PathBuf::from(row.get::<_, String>(2)?), - }, - virtual_path: VirtualPath::new(row.get::<_, String>(3)?), - audio: Some(AudioMeta { - title: row.get(4)?, - artist: row.get(5)?, - album: row.get(6)?, - album_artist: row.get(7)?, - genre: row.get(8)?, - year: row.get(9)?, - track: row.get(10)?, - disc: row.get(11)?, - duration_ms: row.get::<_, Option>(12)?.map(|d| d as u64), - bitrate: row.get(13)?, - sample_rate: row.get(14)?, - format: musicfs_core::AudioFormat::Unknown, // TODO: parse - }), - size: row.get::<_, i64>(17)? as u64, - mtime: UNIX_EPOCH + std::time::Duration::from_secs(row.get::<_, i64>(16)? as u64), - content_hash: row.get::<_, Option>>(18)? - .map(|b| ContentHash(b.try_into().unwrap_or([0; 8]))), - }) - }, - ) - .optional() - .map_err(|e| Error::Database(e.to_string())) - } - - /// Get file by ID - pub fn get_file_by_id(&self, id: FileId) -> Result> { - let conn = self.conn.lock().unwrap(); - - conn.query_row( - "SELECT virtual_path FROM files WHERE id = ?1", - params![id.0], - |row| row.get::<_, String>(0), - ) - .optional() - .map_err(|e| Error::Database(e.to_string()))? - .map(|vp| self.get_file_by_virtual_path(&VirtualPath::new(vp))) - .transpose() - .map(|o| o.flatten()) - } - - /// List all files for an origin - pub fn list_files(&self, origin_id: &OriginId) -> Result> { - let conn = self.conn.lock().unwrap(); - - let mut stmt = conn.prepare( - "SELECT virtual_path FROM files WHERE origin_id = ?1" - ).map_err(|e| Error::Database(e.to_string()))?; - - let paths: Vec = stmt - .query_map(params![&origin_id.0], |row| row.get(0)) - .map_err(|e| Error::Database(e.to_string()))? - .filter_map(|r| r.ok()) - .collect(); - - drop(stmt); - drop(conn); - - paths - .into_iter() - .filter_map(|p| self.get_file_by_virtual_path(&VirtualPath::new(p)).ok().flatten()) - .collect::>() - .pipe(Ok) - } - - /// Delete file by ID - pub fn delete_file(&self, id: FileId) -> Result<()> { - let conn = self.conn.lock().unwrap(); - conn.execute("DELETE FROM files WHERE id = ?1", params![id.0]) - .map_err(|e| Error::Database(e.to_string()))?; - Ok(()) - } - - /// Get file count - pub fn file_count(&self) -> Result { - let conn = self.conn.lock().unwrap(); - conn.query_row("SELECT COUNT(*) FROM files", [], |row| row.get::<_, i64>(0)) - .map(|c| c as u64) - .map_err(|e| Error::Database(e.to_string())) - } -} - -trait Pipe: Sized { - fn pipe(self, f: impl FnOnce(Self) -> T) -> T { - f(self) - } -} - -impl Pipe for T {} -``` - -### 2.5 Create `src/metadata.rs` - -```rust -use crate::db::Database; -use musicfs_core::{AudioMeta, FileMeta, OriginId, Result, VirtualPath}; -use std::path::Path; -use std::sync::Arc; -use std::time::SystemTime; - -/// High-level metadata cache interface -pub struct MetadataCache { - db: Arc, -} - -impl MetadataCache { - pub fn new(db: Arc) -> Self { - Self { db } - } - - /// Store file metadata - pub fn store( - &self, - origin_id: &OriginId, - real_path: &Path, - virtual_path: &VirtualPath, - audio_meta: &AudioMeta, - origin_mtime: SystemTime, - origin_size: u64, - ) -> Result<()> { - self.db.upsert_file( - origin_id, - real_path, - virtual_path, - audio_meta, - origin_mtime, - origin_size, - )?; - Ok(()) - } - - /// Lookup by virtual path - pub fn lookup(&self, path: &VirtualPath) -> Result> { - self.db.get_file_by_virtual_path(path) - } - - /// Check if file exists and is fresh - pub fn is_fresh( - &self, - origin_id: &OriginId, - real_path: &Path, - current_mtime: SystemTime, - ) -> Result { - // TODO: Compare mtime with cached value - Ok(false) - } -} -``` - ---- - -## Tests - -### Unit Tests (`musicfs-metadata`) - -```rust -#[cfg(test)] -mod tests { - use super::*; - use std::io::Cursor; - - #[test] - fn test_parse_flac_metadata() { - // Use a real FLAC file for testing - // For CI, embed a small test file or use a fixture - let parser = MetadataParser::new(); - - // This would need a real file path - // let meta = parser.parse_file(Path::new("test.flac")).unwrap(); - // assert!(meta.title.is_some()); - } - - #[test] - fn test_audio_format_detection() { - assert_eq!(AudioFormat::from_extension("flac"), AudioFormat::Flac); - assert_eq!(AudioFormat::from_extension("mp3"), AudioFormat::Mp3); - assert_eq!(AudioFormat::from_extension("opus"), AudioFormat::Opus); - } -} -``` - -### Unit Tests (`musicfs-cache`) - -```rust -#[cfg(test)] -mod tests { - use super::*; - use musicfs_core::{AudioFormat, AudioMeta, OriginId, VirtualPath}; - use std::time::UNIX_EPOCH; - - #[test] - fn test_database_creation() { - let db = Database::open_memory().unwrap(); - assert_eq!(db.file_count().unwrap(), 0); - } - - #[test] - fn test_upsert_and_retrieve() { - let db = Database::open_memory().unwrap(); - - let origin_id = OriginId::from("local"); - let real_path = Path::new("/music/test.flac"); - let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac"); - let audio_meta = AudioMeta { - title: Some("Track".to_string()), - artist: Some("Artist".to_string()), - album: Some("Album".to_string()), - track: Some(1), - format: AudioFormat::Flac, - ..Default::default() - }; - - let id = db.upsert_file( - &origin_id, - real_path, - &virtual_path, - &audio_meta, - UNIX_EPOCH, - 1000, - ).unwrap(); - - let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap(); - assert_eq!(retrieved.id, id); - assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Track".to_string())); - } - - #[test] - fn test_upsert_updates_existing() { - let db = Database::open_memory().unwrap(); - - let origin_id = OriginId::from("local"); - let real_path = Path::new("/music/test.flac"); - let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac"); - - // First insert - let meta1 = AudioMeta { - title: Some("Original".to_string()), - ..Default::default() - }; - db.upsert_file(&origin_id, real_path, &virtual_path, &meta1, UNIX_EPOCH, 1000).unwrap(); - - // Update - let meta2 = AudioMeta { - title: Some("Updated".to_string()), - ..Default::default() - }; - db.upsert_file(&origin_id, real_path, &virtual_path, &meta2, UNIX_EPOCH, 1000).unwrap(); - - // Should still be 1 file - assert_eq!(db.file_count().unwrap(), 1); - - // Title should be updated - let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap(); - assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Updated".to_string())); - } - - #[test] - fn test_metadata_persistence() { - let dir = tempfile::tempdir().unwrap(); - let db_path = dir.path().join("test.db"); - - // Create and populate - { - let db = Database::open(&db_path).unwrap(); - db.upsert_file( - &OriginId::from("local"), - Path::new("/test.flac"), - &VirtualPath::new("/Test.flac"), - &AudioMeta::default(), - UNIX_EPOCH, - 100, - ).unwrap(); - } - - // Reopen and verify - { - let db = Database::open(&db_path).unwrap(); - assert_eq!(db.file_count().unwrap(), 1); - } - } -} -``` - ---- - -## Exit Criteria - -- [ ] Parse FLAC metadata (title, artist, album, track, duration) -- [ ] Parse MP3 metadata (ID3v2 and ID3v1 fallback) -- [ ] Parse Opus/Vorbis comments -- [ ] Parse M4A/AAC metadata -- [ ] Handle missing metadata gracefully (FR-6.5) -- [ ] SQLite schema creates all tables -- [ ] Metadata persists across daemon restarts (FR-7.4) -- [ ] Upsert correctly updates existing records - ---- - -## Verification Commands - -```bash -# Run metadata tests -cargo test -p musicfs-metadata - -# Run cache tests -cargo test -p musicfs-cache - -# Test with real audio file -cargo run --example parse_metadata -- /path/to/test.flac -``` - ---- - -## Next Week - -Week 3 will implement the virtual path resolver and tree cache, connecting metadata to the FUSE operations. diff --git a/docs/v2/plans/week-03-virtual-tree.md b/docs/v2/plans/week-03-virtual-tree.md deleted file mode 100644 index 9739633..0000000 --- a/docs/v2/plans/week-03-virtual-tree.md +++ /dev/null @@ -1,1069 +0,0 @@ -# Week 3: Virtual Tree & Basic Ops - -**Phase**: 1 (MVP) -**Prerequisites**: Week 2 (Metadata Extraction) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement virtual path resolver, tree cache, and connect to FUSE operations (readdir, stat, read). - ---- - -## Deliverables - -| Task | Crate | Files | Done | -|------|-------|-------|------| -| Virtual path resolver | musicfs-core | `resolver.rs` | [ ] | -| Tree cache | musicfs-cache | `tree.rs` | [ ] | -| readdir implementation | musicfs-fuse | `ops/readdir.rs` | [ ] | -| stat implementation | musicfs-fuse | `ops/stat.rs` | [ ] | -| open/read implementation | musicfs-fuse | `ops/read.rs` | [ ] | - ---- - -## Task 1: Virtual Path Resolver - -### 1.1 Add to `musicfs-core/src/lib.rs` - -```rust -pub mod resolver; -pub use resolver::{PathResolver, PathTemplate}; -``` - -### 1.2 Create `musicfs-core/src/resolver.rs` - -```rust -use crate::{AudioMeta, VirtualPath}; -use std::path::PathBuf; - -/// Path template configuration (per architecture 4.3.1) -/// Uses $variable syntax: $artist, $album, $title, $track, $year, $genre, -/// $format, $format_upper, $disc -#[derive(Debug, Clone)] -pub struct PathTemplate { - /// Template pattern using $var syntax - /// Default: "$artist/$album ($year) [$format_upper]/$track - $title.$format" - pub pattern: String, - /// Fallback for missing artist - pub fallback_artist: String, - /// Fallback for missing album - pub fallback_album: String, - /// Fallback for missing title - pub fallback_title: String, - /// Fallback for missing year - pub fallback_year: String, -} - -impl Default for PathTemplate { - fn default() -> Self { - Self { - // Architecture 4.3.1 default template - pattern: "$artist/$album ($year) [$format_upper]/$track - $title.$format".to_string(), - fallback_artist: "Unknown Artist".to_string(), - fallback_album: "Unknown Album".to_string(), - fallback_title: "Unknown Track".to_string(), - fallback_year: "Unknown".to_string(), - } - } -} - -/// Virtual path resolver (FR-5.1, FR-5.2) -pub struct PathResolver { - template: PathTemplate, -} - -impl PathResolver { - pub fn new(template: PathTemplate) -> Self { - Self { template } - } - - /// Resolve real path + metadata to virtual path - /// Uses $var syntax per architecture 4.3.1 - pub fn resolve(&self, meta: &AudioMeta, extension: &str) -> VirtualPath { - let artist = meta.artist.as_deref() - .unwrap_or(&self.template.fallback_artist); - let album = meta.album.as_deref() - .unwrap_or(&self.template.fallback_album); - let title = meta.title.as_deref() - .unwrap_or(&self.template.fallback_title); - let year = meta.year - .map(|y| y.to_string()) - .unwrap_or_else(|| self.template.fallback_year.clone()); - let track = meta.track.unwrap_or(0); - let disc = meta.disc.unwrap_or(1); - let genre = meta.genre.as_deref().unwrap_or("Unknown"); - let format = extension.to_lowercase(); - let format_upper = extension.to_uppercase(); - - // Sanitize path components - let artist = sanitize_path_component(artist); - let album = sanitize_path_component(album); - let title = sanitize_path_component(title); - let genre = sanitize_path_component(genre); - - // Replace $var patterns (architecture 4.3.1 grammar) - let path = self.template.pattern - .replace("$artist", &artist) - .replace("$album", &album) - .replace("$title", &title) - .replace("$track", &format!("{:02}", track)) - .replace("$disc", &disc.to_string()) - .replace("$year", &year) - .replace("$genre", &genre) - .replace("$format_upper", &format_upper) - .replace("$format", &format); - - VirtualPath::new(path) - } - - /// Parse template pattern to extract directory structure - pub fn get_hierarchy(&self) -> Vec<&str> { - // Returns ["artist", "album"] for default template - self.template.pattern - .split('/') - .filter(|s| s.starts_with('{') && s.ends_with('}')) - .map(|s| s.trim_matches(|c| c == '{' || c == '}')) - .filter(|s| *s == "artist" || *s == "album") - .collect() - } -} - -impl Default for PathResolver { - fn default() -> Self { - Self::new(PathTemplate::default()) - } -} - -/// Sanitize string for use in file path -fn sanitize_path_component(s: &str) -> String { - s.chars() - .map(|c| match c { - '/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_', - '\0' => '_', - c => c, - }) - .collect::() - .trim() - .to_string() -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::AudioFormat; - - #[test] - fn test_resolve_complete_metadata() { - let resolver = PathResolver::default(); - let meta = AudioMeta { - artist: Some("Metallica".to_string()), - album: Some("Master of Puppets".to_string()), - title: Some("Battery".to_string()), - track: Some(1), - year: Some(1986), - format: AudioFormat::Flac, - ..Default::default() - }; - - let path = resolver.resolve(&meta, "flac"); - // Default template: $artist/$album ($year) [$format_upper]/$track - $title.$format - assert_eq!(path.as_str(), "Metallica/Master of Puppets (1986) [FLAC]/01 - Battery.flac"); - } - - #[test] - fn test_resolve_missing_album() { - let resolver = PathResolver::default(); - let meta = AudioMeta { - artist: Some("Artist".to_string()), - title: Some("Track".to_string()), - track: Some(5), - ..Default::default() - }; - - let path = resolver.resolve(&meta, "mp3"); - // Missing year uses fallback - assert_eq!(path.as_str(), "Artist/Unknown Album (Unknown) [MP3]/05 - Track.mp3"); - } - - #[test] - fn test_sanitize_special_chars() { - let resolver = PathResolver::default(); - let meta = AudioMeta { - artist: Some("AC/DC".to_string()), - album: Some("Who Made Who?".to_string()), - title: Some("Test:Track".to_string()), - track: Some(1), - year: Some(1986), - ..Default::default() - }; - - let path = resolver.resolve(&meta, "flac"); - // Should not contain problematic characters - assert!(!path.as_str().contains(':')); - assert!(!path.as_str().contains('?')); - // AC/DC becomes AC_DC - assert!(path.as_str().contains("AC_DC")); - } - - #[test] - fn test_custom_template() { - let template = PathTemplate { - pattern: "$genre/$artist - $album/$track $title.$format".to_string(), - ..Default::default() - }; - let resolver = PathResolver::new(template); - let meta = AudioMeta { - artist: Some("Artist".to_string()), - album: Some("Album".to_string()), - title: Some("Song".to_string()), - genre: Some("Rock".to_string()), - track: Some(3), - ..Default::default() - }; - - let path = resolver.resolve(&meta, "flac"); - assert_eq!(path.as_str(), "Rock/Artist - Album/03 Song.flac"); - } -} -``` - ---- - -## Task 2: Tree Cache - -### 2.1 Add to `musicfs-cache/src/lib.rs` - -```rust -mod tree; -pub use tree::{VirtualTree, VirtualNode, TreeBuilder}; -``` - -### 2.2 Create `musicfs-cache/src/tree.rs` - -```rust -use musicfs_core::{FileId, FileMeta, VirtualPath}; -use std::collections::{BTreeMap, HashMap}; -use std::ffi::{OsStr, OsString}; -use std::sync::atomic::{AtomicU64, Ordering}; -use std::sync::RwLock; -use std::time::{Duration, SystemTime}; - -/// Inode number type -pub type Inode = u64; - -/// Root inode (FUSE convention) -pub const ROOT_INODE: Inode = 1; - -/// Node in the virtual tree -#[derive(Debug)] -pub enum VirtualNode { - Directory(DirNode), - File(FileNode), -} - -impl VirtualNode { - pub fn inode(&self) -> Inode { - match self { - VirtualNode::Directory(d) => d.inode, - VirtualNode::File(f) => f.inode, - } - } - - pub fn name(&self) -> &OsStr { - match self { - VirtualNode::Directory(d) => &d.name, - VirtualNode::File(f) => &f.name, - } - } - - pub fn is_dir(&self) -> bool { - matches!(self, VirtualNode::Directory(_)) - } -} - -/// Directory node -#[derive(Debug)] -pub struct DirNode { - pub inode: Inode, - pub name: OsString, - pub children: BTreeMap, - pub mtime: SystemTime, -} - -/// File node -#[derive(Debug)] -pub struct FileNode { - pub inode: Inode, - pub name: OsString, - pub file_id: FileId, - pub size: u64, - pub mtime: SystemTime, -} - -/// Refresh policy configuration (FR-9.3) -#[derive(Debug, Clone)] -pub struct RefreshPolicy { - /// Time-to-live for cached entries - pub ttl: Duration, - /// Whether to refresh on access - pub refresh_on_access: bool, - /// Background refresh interval (None = disabled) - pub background_interval: Option, -} - -impl Default for RefreshPolicy { - fn default() -> Self { - Self { - ttl: Duration::from_secs(300), // 5 minutes - refresh_on_access: false, - background_interval: None, - } - } -} - -/// Virtual filesystem tree (FR-9.1-9.4) -pub struct VirtualTree { - nodes: HashMap, - path_to_inode: HashMap, - next_inode: AtomicU64, - /// Last refresh timestamp - last_refresh: RwLock, - /// Refresh policy (FR-9.3) - refresh_policy: RefreshPolicy, -} - -impl VirtualTree { - pub fn new() -> Self { - Self::with_policy(RefreshPolicy::default()) - } - - pub fn with_policy(policy: RefreshPolicy) -> Self { - let mut tree = Self { - nodes: HashMap::new(), - path_to_inode: HashMap::new(), - next_inode: AtomicU64::new(ROOT_INODE + 1), - last_refresh: RwLock::new(SystemTime::now()), - refresh_policy: policy, - }; - - // Create root directory - tree.nodes.insert(ROOT_INODE, VirtualNode::Directory(DirNode { - inode: ROOT_INODE, - name: OsString::from(""), - children: BTreeMap::new(), - mtime: SystemTime::now(), - })); - tree.path_to_inode.insert(VirtualPath::new("/"), ROOT_INODE); - - tree - } - - /// Allocate a new inode number - fn alloc_inode(&self) -> Inode { - self.next_inode.fetch_add(1, Ordering::SeqCst) - } - - /// Get node by inode - pub fn get(&self, inode: Inode) -> Option<&VirtualNode> { - self.nodes.get(&inode) - } - - /// Get node by path - pub fn get_by_path(&self, path: &VirtualPath) -> Option<&VirtualNode> { - self.path_to_inode.get(path).and_then(|ino| self.nodes.get(ino)) - } - - /// Lookup child in directory - pub fn lookup(&self, parent_inode: Inode, name: &OsStr) -> Option { - if let Some(VirtualNode::Directory(dir)) = self.nodes.get(&parent_inode) { - dir.children.get(name).copied() - } else { - None - } - } - - /// List directory children - pub fn readdir(&self, inode: Inode) -> Option> { - if let Some(VirtualNode::Directory(dir)) = self.nodes.get(&inode) { - Some(dir.children.iter().map(|(name, &ino)| { - let is_dir = self.nodes.get(&ino).map(|n| n.is_dir()).unwrap_or(false); - (name.clone(), ino, is_dir) - }).collect()) - } else { - None - } - } - - /// Insert a file into the tree - pub fn insert_file(&mut self, meta: &FileMeta) -> Inode { - let path = &meta.virtual_path; - - // Ensure parent directories exist - let parent_inode = self.ensure_parents(path); - - // Create file node - let inode = self.alloc_inode(); - let name = std::path::Path::new(path.as_str()) - .file_name() - .unwrap_or_default() - .to_os_string(); - - let file_node = FileNode { - inode, - name: name.clone(), - file_id: meta.id, - size: meta.size, - mtime: meta.mtime, - }; - - self.nodes.insert(inode, VirtualNode::File(file_node)); - self.path_to_inode.insert(path.clone(), inode); - - // Add to parent - if let Some(VirtualNode::Directory(dir)) = self.nodes.get_mut(&parent_inode) { - dir.children.insert(name, inode); - } - - inode - } - - /// Ensure all parent directories exist for a path - fn ensure_parents(&mut self, path: &VirtualPath) -> Inode { - let path_str = path.as_str(); - let components: Vec<&str> = path_str - .trim_start_matches('/') - .split('/') - .filter(|s| !s.is_empty()) - .collect(); - - if components.len() <= 1 { - return ROOT_INODE; - } - - let mut current_inode = ROOT_INODE; - let mut current_path = String::from("/"); - - // Process all but the last component (which is the file) - for component in &components[..components.len() - 1] { - current_path.push_str(component); - - let vpath = VirtualPath::new(¤t_path); - - if let Some(&existing) = self.path_to_inode.get(&vpath) { - current_inode = existing; - } else { - // Create directory - let new_inode = self.alloc_inode(); - let name = OsString::from(*component); - - let dir_node = DirNode { - inode: new_inode, - name: name.clone(), - children: BTreeMap::new(), - mtime: SystemTime::now(), - }; - - self.nodes.insert(new_inode, VirtualNode::Directory(dir_node)); - self.path_to_inode.insert(vpath, new_inode); - - // Add to parent - if let Some(VirtualNode::Directory(parent)) = self.nodes.get_mut(¤t_inode) { - parent.children.insert(name, new_inode); - } - - current_inode = new_inode; - } - - current_path.push('/'); - } - - current_inode - } - - /// Remove a file from the tree - pub fn remove_file(&mut self, path: &VirtualPath) -> Option { - let inode = self.path_to_inode.remove(path)?; - - if let Some(VirtualNode::File(file)) = self.nodes.remove(&inode) { - // Remove from parent - let parent_path = std::path::Path::new(path.as_str()) - .parent() - .map(|p| VirtualPath::new(p.to_string_lossy())) - .unwrap_or_else(|| VirtualPath::new("/")); - - if let Some(&parent_inode) = self.path_to_inode.get(&parent_path) { - if let Some(VirtualNode::Directory(dir)) = self.nodes.get_mut(&parent_inode) { - dir.children.remove(&file.name); - } - } - - Some(file.file_id) - } else { - None - } - } - - /// Get total file count - pub fn file_count(&self) -> usize { - self.nodes.values().filter(|n| matches!(n, VirtualNode::File(_))).count() - } - - /// Get total directory count - pub fn dir_count(&self) -> usize { - self.nodes.values().filter(|n| matches!(n, VirtualNode::Directory(_))).count() - } - - // ========================================================================= - // Refresh Policy (FR-9.3, FR-9.4) - // ========================================================================= - - /// Check if tree needs refresh based on policy (FR-9.3) - pub fn needs_refresh(&self) -> bool { - let last = *self.last_refresh.read().unwrap(); - last.elapsed().unwrap_or(Duration::MAX) > self.refresh_policy.ttl - } - - /// Force refresh - clears tree for rebuild (FR-9.4) - /// Call this from signal handler or API endpoint - pub fn force_refresh(&mut self) { - // Keep root, clear everything else - self.nodes.retain(|&ino, _| ino == ROOT_INODE); - self.path_to_inode.retain(|p, _| p.as_str() == "/"); - - // Reset root children - if let Some(VirtualNode::Directory(root)) = self.nodes.get_mut(&ROOT_INODE) { - root.children.clear(); - } - - *self.last_refresh.write().unwrap() = SystemTime::now(); - } - - /// Mark tree as refreshed - pub fn mark_refreshed(&self) { - *self.last_refresh.write().unwrap() = SystemTime::now(); - } - - /// Get current refresh policy - pub fn refresh_policy(&self) -> &RefreshPolicy { - &self.refresh_policy - } -} - -impl Default for VirtualTree { - fn default() -> Self { - Self::new() - } -} - -/// Builder for constructing tree from database -pub struct TreeBuilder { - tree: VirtualTree, -} - -impl TreeBuilder { - pub fn new() -> Self { - Self { - tree: VirtualTree::new(), - } - } - - pub fn add_file(&mut self, meta: &FileMeta) { - self.tree.insert_file(meta); - } - - pub fn build(self) -> VirtualTree { - self.tree - } -} - -impl Default for TreeBuilder { - fn default() -> Self { - Self::new() - } -} - -#[cfg(test)] -mod tests { - use super::*; - use musicfs_core::{AudioMeta, OriginId, RealPath}; - use std::path::PathBuf; - - fn make_file_meta(id: i64, vpath: &str) -> FileMeta { - FileMeta { - id: FileId(id), - virtual_path: VirtualPath::new(vpath), - real_path: RealPath { - origin_id: OriginId::from("test"), - path: PathBuf::from("/test"), - }, - size: 1000, - mtime: SystemTime::now(), - content_hash: None, - audio: None, - } - } - - #[test] - fn test_tree_creation() { - let tree = VirtualTree::new(); - assert!(tree.get(ROOT_INODE).is_some()); - } - - #[test] - fn test_insert_file() { - let mut tree = VirtualTree::new(); - let meta = make_file_meta(1, "/Artist/Album/Track.flac"); - tree.insert_file(&meta); - - assert!(tree.get_by_path(&VirtualPath::new("/Artist")).is_some()); - assert!(tree.get_by_path(&VirtualPath::new("/Artist/Album")).is_some()); - assert!(tree.get_by_path(&VirtualPath::new("/Artist/Album/Track.flac")).is_some()); - } - - #[test] - fn test_readdir() { - let mut tree = VirtualTree::new(); - tree.insert_file(&make_file_meta(1, "/Artist/Album/Track1.flac")); - tree.insert_file(&make_file_meta(2, "/Artist/Album/Track2.flac")); - - let root_children = tree.readdir(ROOT_INODE).unwrap(); - assert_eq!(root_children.len(), 1); - assert_eq!(root_children[0].0, "Artist"); - } - - #[test] - fn test_lookup() { - let mut tree = VirtualTree::new(); - tree.insert_file(&make_file_meta(1, "/Artist/Album/Track.flac")); - - let artist_inode = tree.lookup(ROOT_INODE, OsStr::new("Artist")).unwrap(); - assert!(tree.lookup(artist_inode, OsStr::new("Album")).is_some()); - } -} -``` - ---- - -## Task 3: FUSE Operations - -### 3.1 Refactor `musicfs-fuse/src/lib.rs` - -```rust -mod filesystem; -mod ops; - -pub use filesystem::MusicFs; -``` - -### 3.2 Create `musicfs-fuse/src/ops/mod.rs` - -```rust -pub mod readdir; -pub mod stat; -pub mod read; -``` - -### 3.3 Update `musicfs-fuse/src/filesystem.rs` - -```rust -use fuser::{ - FileAttr, FileType, Filesystem, ReplyAttr, ReplyData, ReplyDirectory, - ReplyEntry, ReplyOpen, Request, FUSE_ROOT_ID, -}; -use musicfs_cache::{Database, MetadataCache, VirtualTree, VirtualNode, ROOT_INODE}; -use musicfs_core::{Error, Result, VirtualPath}; -use musicfs_origins::Origin; -use std::collections::HashMap; -use std::ffi::OsStr; -use std::sync::{Arc, RwLock}; -use std::time::{Duration, SystemTime, UNIX_EPOCH}; -use tracing::{debug, error, info, warn}; - -const TTL: Duration = Duration::from_secs(1); -const BLOCK_SIZE: u32 = 512; - -/// Main FUSE filesystem implementation -pub struct MusicFs { - tree: Arc>, - cache: Arc, - origins: Arc>>, - uid: u32, - gid: u32, -} - -impl MusicFs { - pub fn new( - tree: Arc>, - cache: Arc, - origins: Arc>>, - ) -> Self { - Self { - tree, - cache, - origins, - uid: unsafe { libc::getuid() }, - gid: unsafe { libc::getgid() }, - } - } - - fn node_to_attr(&self, node: &VirtualNode) -> FileAttr { - match node { - VirtualNode::Directory(dir) => FileAttr { - ino: dir.inode, - size: 0, - blocks: 0, - atime: dir.mtime, - mtime: dir.mtime, - ctime: dir.mtime, - crtime: dir.mtime, - kind: FileType::Directory, - perm: 0o755, - nlink: 2, - uid: self.uid, - gid: self.gid, - rdev: 0, - blksize: BLOCK_SIZE, - flags: 0, - }, - VirtualNode::File(file) => FileAttr { - ino: file.inode, - size: file.size, - blocks: (file.size + BLOCK_SIZE as u64 - 1) / BLOCK_SIZE as u64, - atime: file.mtime, - mtime: file.mtime, - ctime: file.mtime, - crtime: file.mtime, - kind: FileType::RegularFile, - perm: 0o644, - nlink: 1, - uid: self.uid, - gid: self.gid, - rdev: 0, - blksize: BLOCK_SIZE, - flags: 0, - }, - } - } -} - -impl Filesystem for MusicFs { - fn init( - &mut self, - _req: &Request<'_>, - _config: &mut fuser::KernelConfig, - ) -> std::result::Result<(), libc::c_int> { - info!("MusicFS initialized"); - Ok(()) - } - - fn destroy(&mut self) { - info!("MusicFS destroyed"); - } - - fn lookup(&mut self, _req: &Request, parent: u64, name: &OsStr, reply: ReplyEntry) { - debug!("lookup(parent={}, name={:?})", parent, name); - - let tree = self.tree.read().unwrap(); - - if let Some(inode) = tree.lookup(parent, name) { - if let Some(node) = tree.get(inode) { - let attr = self.node_to_attr(node); - reply.entry(&TTL, &attr, 0); - return; - } - } - - reply.error(libc::ENOENT); - } - - fn getattr(&mut self, _req: &Request, ino: u64, reply: ReplyAttr) { - debug!("getattr(ino={})", ino); - - let tree = self.tree.read().unwrap(); - - if let Some(node) = tree.get(ino) { - let attr = self.node_to_attr(node); - reply.attr(&TTL, &attr); - } else { - reply.error(libc::ENOENT); - } - } - - fn readdir( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - offset: i64, - mut reply: ReplyDirectory, - ) { - debug!("readdir(ino={}, offset={})", ino, offset); - - let tree = self.tree.read().unwrap(); - - if let Some(children) = tree.readdir(ino) { - let mut entries = vec![ - (ino, FileType::Directory, "."), - (if ino == ROOT_INODE { ROOT_INODE } else { ROOT_INODE }, FileType::Directory, ".."), - ]; - - for (name, child_ino, is_dir) in children { - let kind = if is_dir { FileType::Directory } else { FileType::RegularFile }; - entries.push((child_ino, kind, name.to_str().unwrap_or("?"))); - } - - for (i, (inode, kind, name)) in entries.iter().enumerate().skip(offset as usize) { - if reply.add(*inode, (i + 1) as i64, *kind, name) { - break; - } - } - - reply.ok(); - } else { - reply.error(libc::ENOENT); - } - } - - fn open(&mut self, _req: &Request, ino: u64, flags: i32, reply: ReplyOpen) { - debug!("open(ino={}, flags={})", ino, flags); - - // Check for write flags (FR-4.1) - let write_flags = libc::O_WRONLY | libc::O_RDWR | libc::O_APPEND | libc::O_TRUNC; - if flags & write_flags != 0 { - reply.error(libc::EROFS); - return; - } - - let tree = self.tree.read().unwrap(); - - if tree.get(ino).is_some() { - reply.opened(0, 0); - } else { - reply.error(libc::ENOENT); - } - } - - fn read( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - offset: i64, - size: u32, - _flags: i32, - _lock_owner: Option, - reply: ReplyData, - ) { - debug!("read(ino={}, offset={}, size={})", ino, offset, size); - - // Get file info from tree - let file_id = { - let tree = self.tree.read().unwrap(); - if let Some(VirtualNode::File(file)) = tree.get(ino) { - file.file_id - } else { - reply.error(libc::ENOENT); - return; - } - }; - - // TODO: Read from cache/origin - // For now, return empty data - reply.data(&[]); - } - - fn release( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - _flags: i32, - _lock_owner: Option, - _flush: bool, - reply: fuser::ReplyEmpty, - ) { - debug!("release(ino={})", ino); - reply.ok(); - } - - // Write operations - always EROFS (FR-4.1-4.4) - fn write(&mut self, _req: &Request, _ino: u64, _fh: u64, _offset: i64, _data: &[u8], _write_flags: u32, _flags: i32, _lock_owner: Option, reply: fuser::ReplyWrite) { - reply.error(libc::EROFS); - } - - fn mkdir(&mut self, _req: &Request, _parent: u64, _name: &OsStr, _mode: u32, _umask: u32, reply: ReplyEntry) { - reply.error(libc::EROFS); - } - - fn unlink(&mut self, _req: &Request, _parent: u64, _name: &OsStr, reply: fuser::ReplyEmpty) { - reply.error(libc::EROFS); - } - - fn rmdir(&mut self, _req: &Request, _parent: u64, _name: &OsStr, reply: fuser::ReplyEmpty) { - reply.error(libc::EROFS); - } - - fn rename(&mut self, _req: &Request, _parent: u64, _name: &OsStr, _newparent: u64, _newname: &OsStr, _flags: u32, reply: fuser::ReplyEmpty) { - reply.error(libc::EROFS); - } - - fn create(&mut self, _req: &Request, _parent: u64, _name: &OsStr, _mode: u32, _umask: u32, _flags: i32, reply: fuser::ReplyCreate) { - reply.error(libc::EROFS); - } -} -``` - ---- - -## Task 4: Benchmarks - -### 4.1 Create `benches/tree_ops.rs` - -```rust -use criterion::{black_box, criterion_group, criterion_main, Criterion}; -use musicfs_cache::{VirtualTree, TreeBuilder}; -use musicfs_core::{FileId, FileMeta, OriginId, RealPath, VirtualPath}; -use std::path::PathBuf; -use std::time::SystemTime; - -fn make_file_meta(id: i64, artist: &str, album: &str, track: u32) -> FileMeta { - let vpath = format!("/{}/{}/{:02} - Track.flac", artist, album, track); - FileMeta { - id: FileId(id), - virtual_path: VirtualPath::new(&vpath), - real_path: RealPath { - origin_id: OriginId::from("test"), - path: PathBuf::from("/test"), - }, - size: 30_000_000, - mtime: SystemTime::now(), - content_hash: None, - audio: None, - } -} - -fn build_tree(n_artists: usize, albums_per_artist: usize, tracks_per_album: usize) -> VirtualTree { - let mut builder = TreeBuilder::new(); - let mut id = 1i64; - - for a in 0..n_artists { - for b in 0..albums_per_artist { - for t in 0..tracks_per_album { - let meta = make_file_meta( - id, - &format!("Artist {}", a), - &format!("Album {}", b), - t as u32 + 1, - ); - builder.add_file(&meta); - id += 1; - } - } - } - - builder.build() -} - -fn bench_stat_cached(c: &mut Criterion) { - // 100 artists * 10 albums * 12 tracks = 12,000 files - let tree = build_tree(100, 10, 12); - let path = VirtualPath::new("/Artist 50/Album 5/06 - Track.flac"); - - c.bench_function("stat_cached", |b| { - b.iter(|| { - black_box(tree.get_by_path(&path)) - }) - }); - - // Target: <1ms p99 (NFR-1.1) -} - -fn bench_readdir_1000_entries(c: &mut Criterion) { - // Create tree with 1000 artists (1000 entries in root) - let tree = build_tree(1000, 1, 1); - - c.bench_function("readdir_1000", |b| { - b.iter(|| { - black_box(tree.readdir(1)) // ROOT_INODE - }) - }); - - // Target: <10ms p99 (NFR-1.2) -} - -fn bench_lookup(c: &mut Criterion) { - let tree = build_tree(100, 10, 12); - - c.bench_function("lookup", |b| { - b.iter(|| { - let artist = tree.lookup(1, std::ffi::OsStr::new("Artist 50")); - if let Some(a) = artist { - black_box(tree.lookup(a, std::ffi::OsStr::new("Album 5"))); - } - }) - }); -} - -fn bench_mount_time(c: &mut Criterion) { - c.bench_function("tree_build_12k", |b| { - b.iter(|| { - black_box(build_tree(100, 10, 12)) - }) - }); - - // Target: <100ms, Max: <500ms (NFR-1.7) -} - -criterion_group!(benches, bench_stat_cached, bench_readdir_1000_entries, bench_lookup, bench_mount_time); -criterion_main!(benches); -``` - ---- - -## Tests - -### Integration Tests - -```rust -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_fuse_readdir() { - // Build tree and verify readdir returns correct entries - } - - #[test] - fn test_fuse_stat() { - // Verify stat returns correct size/mtime - } - - #[test] - fn test_read_only_enforcement() { - // Verify write ops return EROFS - } -} -``` - ---- - -## Exit Criteria - -- [ ] Virtual tree built from metadata -- [ ] `ls /mnt/music` shows Artist directories -- [ ] `ls /mnt/music/Artist/Album` shows tracks -- [ ] `stat` returns correct size, mtime -- [ ] Write operations return EROFS (FR-4.1) -- [ ] stat benchmark <1ms p99 (NFR-1.1) -- [ ] readdir benchmark <10ms p99 (NFR-1.2) -- [ ] Mount completes in <500ms (NFR-1.7) - ---- - -## Next Week - -Week 4 will implement CAS storage and chunk caching, enabling actual file reads. diff --git a/docs/v2/plans/week-04-cas-caching.md b/docs/v2/plans/week-04-cas-caching.md deleted file mode 100644 index 47ff19b..0000000 --- a/docs/v2/plans/week-04-cas-caching.md +++ /dev/null @@ -1,1089 +0,0 @@ -# Week 4: CAS & Chunk Caching - -**Phase**: 1 (MVP) -**Prerequisites**: Week 3 (Virtual Tree & Basic Ops) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement Content-Addressable Storage (CAS) for chunk deduplication, cache eviction with LRU policy, and connect to FUSE read operations to enable actual file playback. - -**Note**: Week 4 treats whole files as single chunks for simplicity. Week 5 adds CDC (Content-Defined Chunking) via FastCDC for efficient delta sync (FR-8.2, FR-11.2). - ---- - -## Deliverables - -| Task | Crate | Files | Done | -|------|-------|-------|------| -| CAS store implementation | musicfs-cas | `lib.rs`, `store.rs` | [ ] | -| Chunk storage | musicfs-cas | `chunks.rs` | [ ] | -| Cache eviction (LRU) | musicfs-cache | `eviction.rs` | [ ] | -| FUSE read integration | musicfs-fuse | `filesystem.rs` | [ ] | -| Integration tests | tests/integration | `basic_mount.rs` | [ ] | - ---- - -## Task 1: CAS Store - -### 1.1 Update `musicfs-cas/Cargo.toml` - -```toml -[package] -name = "musicfs-cas" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -tokio.workspace = true -tracing.workspace = true -serde.workspace = true -sled = "0.34" -xxhash-rust = { version = "0.8", features = ["xxh64"] } -bytes = "1" -rmp-serde = "1" # msgpack per architecture 4.3.6 -hex = "0.4" -dirs = "5" # For ~/.cache resolution -thiserror.workspace = true -``` - -### 1.2 Create `musicfs-cas/src/lib.rs` - -```rust -mod store; -mod chunks; - -pub use store::{CasStore, CasConfig, CasError, DedupStats}; -pub use chunks::{ChunkHash, ChunkLocation, ChunkRef}; -``` - -### 1.3 Create `musicfs-cas/src/chunks.rs` - -```rust -use serde::{Deserialize, Serialize}; -use std::path::PathBuf; - -/// Chunk hash (xxHash64, 8 bytes) per architecture 8.3 -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] -pub struct ChunkHash(pub [u8; 8]); - -impl ChunkHash { - pub fn from_bytes(bytes: &[u8]) -> Self { - let hash = xxhash_rust::xxh64::xxh64(bytes, 0); - Self(hash.to_le_bytes()) - } - - pub fn as_hex(&self) -> String { - hex::encode(self.0) - } - - pub fn from_hex(s: &str) -> Option { - let bytes = hex::decode(s).ok()?; - if bytes.len() != 8 { - return None; - } - let mut arr = [0u8; 8]; - arr.copy_from_slice(&bytes); - Some(Self(arr)) - } -} - -impl std::fmt::Display for ChunkHash { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - write!(f, "{}", self.as_hex()) - } -} - -/// Location of a chunk in storage -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct ChunkLocation { - pub path: PathBuf, - pub size: u32, -} - -/// Reference to a chunk within a file (per architecture 4.3.6 chunk_manifest format) -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct ChunkRef { - pub hash: ChunkHash, - pub offset: u64, - pub size: u32, -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_chunk_hash_from_bytes() { - let data = b"hello world"; - let hash = ChunkHash::from_bytes(data); - assert_eq!(hash.as_hex().len(), 16); - } - - #[test] - fn test_chunk_hash_deterministic() { - let data = b"test data"; - let hash1 = ChunkHash::from_bytes(data); - let hash2 = ChunkHash::from_bytes(data); - assert_eq!(hash1, hash2); - } - - #[test] - fn test_chunk_hash_hex_roundtrip() { - let data = b"roundtrip test"; - let hash = ChunkHash::from_bytes(data); - let hex = hash.as_hex(); - let restored = ChunkHash::from_hex(&hex).unwrap(); - assert_eq!(hash, restored); - } -} -``` - -### 1.4 Create `musicfs-cas/src/store.rs` - -```rust -use crate::chunks::{ChunkHash, ChunkLocation}; -use bytes::Bytes; -use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicU64, Ordering}; -use tokio::fs; -use tracing::{debug, warn}; - -/// CAS configuration -#[derive(Debug, Clone)] -pub struct CasConfig { - /// Root directory for chunk storage - pub chunks_dir: PathBuf, - /// Maximum cache size in bytes (FR-8.2) - pub max_size: u64, - /// Number of subdirectory levels (for filesystem performance) - pub shard_levels: u8, -} - -impl Default for CasConfig { - fn default() -> Self { - // Per architecture 4.3.2: ~/.cache/musicfs/chunks/ - let cache_dir = dirs::cache_dir() - .unwrap_or_else(|| PathBuf::from(".cache")) - .join("musicfs") - .join("chunks"); - - Self { - chunks_dir: cache_dir, - max_size: 10 * 1024 * 1024 * 1024, // 10 GB per NFR-5.2 - shard_levels: 2, // 256 subdirs per architecture 4.3.2 - } - } -} - -/// Content-Addressable Storage (FR-20.1-20.4) -pub struct CasStore { - config: CasConfig, - index: sled::Db, - current_size: AtomicU64, -} - -impl CasStore { - pub async fn open(config: CasConfig) -> Result { - fs::create_dir_all(&config.chunks_dir).await?; - - let index_path = config.chunks_dir.join("index.sled"); - let index = sled::open(&index_path)?; - - let current_size = Self::calculate_size(&config.chunks_dir).await; - - Ok(Self { - config, - index, - current_size: AtomicU64::new(current_size), - }) - } - - async fn calculate_size(dir: &Path) -> u64 { - let mut size = 0u64; - if let Ok(mut entries) = fs::read_dir(dir).await { - while let Ok(Some(entry)) = entries.next_entry().await { - if let Ok(meta) = entry.metadata().await { - if meta.is_file() { - size += meta.len(); - } - } - } - } - size - } - - /// Store chunk, returns hash (FR-20.1) - /// Deduplicates automatically - same content = same hash (FR-20.2) - pub async fn put(&self, data: &[u8]) -> Result { - let hash = ChunkHash::from_bytes(data); - let path = self.chunk_path(&hash); - - if path.exists() { - debug!("Chunk {} already exists (dedup)", hash); - return Ok(hash); - } - - if let Some(parent) = path.parent() { - fs::create_dir_all(parent).await?; - } - - fs::write(&path, data).await?; - - let location = ChunkLocation { - path: path.clone(), - size: data.len() as u32, - }; - // Use msgpack per architecture 4.3.6 - self.index.insert( - hash.0.as_slice(), - rmp_serde::to_vec(&location).unwrap(), - )?; - - self.current_size.fetch_add(data.len() as u64, Ordering::SeqCst); - - debug!("Stored chunk {} ({} bytes)", hash, data.len()); - Ok(hash) - } - - /// Retrieve chunk by hash (FR-20.1) - pub async fn get(&self, hash: &ChunkHash) -> Result { - let path = self.chunk_path(hash); - - if !path.exists() { - return Err(CasError::NotFound(hash.as_hex())); - } - - let data = fs::read(&path).await?; - - if self.config.max_size > 0 { - self.verify_integrity(hash, &data)?; - } - - Ok(Bytes::from(data)) - } - - /// Check if chunk exists (for dedup check) - pub fn exists(&self, hash: &ChunkHash) -> bool { - self.chunk_path(hash).exists() - } - - /// Verify chunk integrity (FR-20.4) - fn verify_integrity(&self, expected: &ChunkHash, data: &[u8]) -> Result<(), CasError> { - let actual = ChunkHash::from_bytes(data); - if actual != *expected { - warn!("Chunk integrity failure: expected {}, got {}", expected, actual); - return Err(CasError::IntegrityError { - expected: expected.as_hex(), - actual: actual.as_hex(), - }); - } - Ok(()) - } - - /// Get path for a chunk hash (sharded for filesystem performance) - fn chunk_path(&self, hash: &ChunkHash) -> PathBuf { - let hex = hash.as_hex(); - let mut path = self.config.chunks_dir.clone(); - - for i in 0..self.config.shard_levels as usize { - let start = i * 2; - let end = start + 2; - if end <= hex.len() { - path = path.join(&hex[start..end]); - } - } - - path.join(&hex) - } - - /// Delete a chunk - pub async fn delete(&self, hash: &ChunkHash) -> Result<(), CasError> { - let path = self.chunk_path(hash); - - if path.exists() { - let meta = fs::metadata(&path).await?; - fs::remove_file(&path).await?; - self.index.remove(hash.0.as_slice())?; - self.current_size.fetch_sub(meta.len(), Ordering::SeqCst); - debug!("Deleted chunk {}", hash); - } - - Ok(()) - } - - /// Get current cache size - pub fn current_size(&self) -> u64 { - self.current_size.load(Ordering::SeqCst) - } - - /// Get maximum cache size - pub fn max_size(&self) -> u64 { - self.config.max_size - } - - /// List all chunk hashes - pub fn list_chunks(&self) -> impl Iterator + '_ { - self.index.iter().filter_map(|r| { - r.ok().and_then(|(k, _)| { - if k.len() == 8 { - let mut arr = [0u8; 8]; - arr.copy_from_slice(&k); - Some(ChunkHash(arr)) - } else { - None - } - }) - }) - } - - /// Get deduplication statistics (FR-20.3) - pub fn dedup_stats(&self) -> DedupStats { - let chunks_stored = self.index.len() as u64; - let size_bytes = self.current_size(); - - DedupStats { - chunks_stored, - chunks_unique: chunks_stored, // All stored chunks are unique by definition - size_bytes, - size_limit_bytes: self.config.max_size, - } - } -} - -/// Deduplication statistics (FR-20.3) -#[derive(Debug, Clone)] -pub struct DedupStats { - pub chunks_stored: u64, - pub chunks_unique: u64, - pub size_bytes: u64, - pub size_limit_bytes: u64, -} - -impl DedupStats { - /// Calculate dedup ratio (space saved) - pub fn dedup_ratio(&self) -> f64 { - if self.chunks_stored == 0 { - 0.0 - } else { - 1.0 - (self.chunks_unique as f64 / self.chunks_stored as f64) - } - } -} - -#[derive(Debug, thiserror::Error)] -pub enum CasError { - #[error("IO error: {0}")] - Io(#[from] std::io::Error), - - #[error("Sled error: {0}")] - Sled(#[from] sled::Error), - - #[error("Chunk not found: {0}")] - NotFound(String), - - #[error("Integrity error: expected {expected}, got {actual}")] - IntegrityError { expected: String, actual: String }, -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - async fn test_store() -> (CasStore, TempDir) { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - max_size: 1024 * 1024, - shard_levels: 2, - }; - let store = CasStore::open(config).await.unwrap(); - (store, dir) - } - - #[tokio::test] - async fn test_cas_put_get() { - let (store, _dir) = test_store().await; - - let data = b"test chunk data"; - let hash = store.put(data).await.unwrap(); - - let retrieved = store.get(&hash).await.unwrap(); - assert_eq!(&retrieved[..], data); - } - - #[tokio::test] - async fn test_cas_dedup() { - let (store, _dir) = test_store().await; - - let data = b"duplicate data"; - let hash1 = store.put(data).await.unwrap(); - let hash2 = store.put(data).await.unwrap(); - - assert_eq!(hash1, hash2); - } - - #[tokio::test] - async fn test_cas_exists() { - let (store, _dir) = test_store().await; - - let data = b"existence test"; - let hash = store.put(data).await.unwrap(); - - assert!(store.exists(&hash)); - - let fake_hash = ChunkHash::from_bytes(b"nonexistent"); - assert!(!store.exists(&fake_hash)); - } - - #[tokio::test] - async fn test_cas_delete() { - let (store, _dir) = test_store().await; - - let data = b"delete me"; - let hash = store.put(data).await.unwrap(); - - assert!(store.exists(&hash)); - - store.delete(&hash).await.unwrap(); - - assert!(!store.exists(&hash)); - } - - #[tokio::test] - async fn test_cas_integrity() { - let (store, _dir) = test_store().await; - - let data = b"integrity test"; - let hash = store.put(data).await.unwrap(); - - let retrieved = store.get(&hash).await.unwrap(); - assert_eq!(&retrieved[..], data); - } - - #[tokio::test] - async fn test_cas_dedup_stats() { - let (store, _dir) = test_store().await; - - store.put(b"chunk1").await.unwrap(); - store.put(b"chunk2").await.unwrap(); - store.put(b"chunk1").await.unwrap(); // Duplicate - - let stats = store.dedup_stats(); - assert_eq!(stats.chunks_stored, 2); // Only 2 unique - assert_eq!(stats.chunks_unique, 2); - } -} -``` - ---- - -## Task 2: Cache Eviction - -### 2.1 Add to `musicfs-cache/src/lib.rs` - -```rust -mod eviction; -pub use eviction::{LruEviction, EvictionPolicy}; -``` - -### 2.2 Create `musicfs-cache/src/eviction.rs` - -```rust -use musicfs_cas::{CasStore, ChunkHash}; -use std::collections::BTreeMap; -use std::sync::RwLock; -use std::time::Instant; -use tracing::{debug, info}; - -/// Eviction policy trait -pub trait EvictionPolicy: Send + Sync { - fn record_access(&self, hash: ChunkHash); - fn select_victims(&self, count: usize) -> Vec; - fn remove(&self, hash: &ChunkHash); -} - -/// LRU eviction policy (FR-8.2) -pub struct LruEviction { - access_times: RwLock>, - hash_to_time: RwLock>, -} - -impl LruEviction { - pub fn new() -> Self { - Self { - access_times: RwLock::new(BTreeMap::new()), - hash_to_time: RwLock::new(std::collections::HashMap::new()), - } - } - - /// Evict chunks until under target size - pub async fn evict_to_target( - &self, - store: &CasStore, - target_size: u64, - ) -> Result { - let mut bytes_freed = 0u64; - - while store.current_size() > target_size { - let victims = self.select_victims(10); - - if victims.is_empty() { - break; - } - - for hash in victims { - if let Ok(data) = store.get(&hash).await { - bytes_freed += data.len() as u64; - store.delete(&hash).await?; - self.remove(&hash); - } - } - } - - if bytes_freed > 0 { - info!("Evicted {} bytes from cache", bytes_freed); - } - - Ok(bytes_freed) - } -} - -impl Default for LruEviction { - fn default() -> Self { - Self::new() - } -} - -impl EvictionPolicy for LruEviction { - fn record_access(&self, hash: ChunkHash) { - let now = Instant::now(); - let mut times = self.access_times.write().unwrap(); - let mut h2t = self.hash_to_time.write().unwrap(); - - if let Some(old_time) = h2t.remove(&hash) { - times.remove(&old_time); - } - - times.insert(now, hash); - h2t.insert(hash, now); - } - - fn select_victims(&self, count: usize) -> Vec { - let times = self.access_times.read().unwrap(); - times.values().take(count).copied().collect() - } - - fn remove(&self, hash: &ChunkHash) { - let mut times = self.access_times.write().unwrap(); - let mut h2t = self.hash_to_time.write().unwrap(); - - if let Some(time) = h2t.remove(hash) { - times.remove(&time); - } - } -} - -#[derive(Debug, thiserror::Error)] -pub enum EvictionError { - #[error("CAS error: {0}")] - Cas(#[from] musicfs_cas::CasError), -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_lru_access_order() { - let lru = LruEviction::new(); - - let h1 = ChunkHash::from_bytes(b"chunk1"); - let h2 = ChunkHash::from_bytes(b"chunk2"); - let h3 = ChunkHash::from_bytes(b"chunk3"); - - lru.record_access(h1); - std::thread::sleep(std::time::Duration::from_millis(1)); - lru.record_access(h2); - std::thread::sleep(std::time::Duration::from_millis(1)); - lru.record_access(h3); - - let victims = lru.select_victims(2); - assert_eq!(victims.len(), 2); - assert_eq!(victims[0], h1); - assert_eq!(victims[1], h2); - } - - #[test] - fn test_lru_reaccess_updates_order() { - let lru = LruEviction::new(); - - let h1 = ChunkHash::from_bytes(b"chunk1"); - let h2 = ChunkHash::from_bytes(b"chunk2"); - - lru.record_access(h1); - std::thread::sleep(std::time::Duration::from_millis(1)); - lru.record_access(h2); - std::thread::sleep(std::time::Duration::from_millis(1)); - lru.record_access(h1); - - let victims = lru.select_victims(1); - assert_eq!(victims[0], h2); - } - - #[test] - fn test_lru_remove() { - let lru = LruEviction::new(); - - let h1 = ChunkHash::from_bytes(b"chunk1"); - let h2 = ChunkHash::from_bytes(b"chunk2"); - - lru.record_access(h1); - lru.record_access(h2); - lru.remove(&h1); - - let victims = lru.select_victims(10); - assert_eq!(victims.len(), 1); - assert_eq!(victims[0], h2); - } -} -``` - ---- - -## Task 3: File Reader Integration - -### 3.1 Create `musicfs-cas/src/reader.rs` - -```rust -use crate::{ChunkHash, ChunkRef, CasStore}; -use bytes::{Bytes, BytesMut}; -use musicfs_core::FileId; -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; -use std::sync::RwLock; - -/// Chunk manifest for a file (per architecture 4.3.6) -/// Stored as msgpack BLOB in SQLite files.chunk_manifest column -/// Format: [(chunk_hash, offset, size), ...] -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct ChunkManifest { - pub file_id: FileId, - pub total_size: u64, - pub chunks: Vec, -} - -impl ChunkManifest { - /// Serialize chunks to msgpack for database storage (architecture 4.3.6) - pub fn chunks_to_bytes(&self) -> Vec { - rmp_serde::to_vec(&self.chunks).unwrap() - } - - /// Deserialize chunks from database BLOB - pub fn chunks_from_bytes(data: &[u8]) -> Option> { - rmp_serde::from_slice(data).ok() - } - - /// Create manifest from database fields - pub fn from_db(file_id: FileId, total_size: u64, chunk_blob: &[u8]) -> Option { - let chunks = Self::chunks_from_bytes(chunk_blob)?; - Some(Self { file_id, total_size, chunks }) - } -} - -/// File reader using CAS chunks -pub struct FileReader { - store: std::sync::Arc, - manifests: RwLock>, -} - -impl FileReader { - pub fn new(store: std::sync::Arc) -> Self { - Self { - store, - manifests: RwLock::new(HashMap::new()), - } - } - - /// Register a file's chunk manifest - pub fn register_manifest(&self, manifest: ChunkManifest) { - let mut manifests = self.manifests.write().unwrap(); - manifests.insert(manifest.file_id, manifest); - } - - /// Read bytes from a file at offset - pub async fn read( - &self, - file_id: FileId, - offset: u64, - size: u32, - ) -> Result { - let manifest = { - let manifests = self.manifests.read().unwrap(); - manifests.get(&file_id).cloned() - .ok_or(ReaderError::ManifestNotFound(file_id))? - }; - - if offset >= manifest.total_size { - return Ok(Bytes::new()); - } - - let end = std::cmp::min(offset + size as u64, manifest.total_size); - let mut result = BytesMut::with_capacity((end - offset) as usize); - - for chunk_ref in &manifest.chunks { - let chunk_start = chunk_ref.offset; - let chunk_end = chunk_ref.offset + chunk_ref.size as u64; - - if chunk_end <= offset || chunk_start >= end { - continue; - } - - let chunk_data = self.store.get(&chunk_ref.hash).await?; - - let read_start = if offset > chunk_start { - (offset - chunk_start) as usize - } else { - 0 - }; - - let read_end = if end < chunk_end { - (end - chunk_start) as usize - } else { - chunk_ref.size as usize - }; - - result.extend_from_slice(&chunk_data[read_start..read_end]); - } - - Ok(result.freeze()) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum ReaderError { - #[error("Manifest not found for file {0:?}")] - ManifestNotFound(FileId), - - #[error("CAS error: {0}")] - Cas(#[from] crate::CasError), -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::CasConfig; - use tempfile::TempDir; - - #[tokio::test] - async fn test_file_reader_simple() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - ..Default::default() - }; - let store = std::sync::Arc::new(CasStore::open(config).await.unwrap()); - - let data = b"Hello, World!"; - let hash = store.put(data).await.unwrap(); - - let reader = FileReader::new(store); - reader.register_manifest(ChunkManifest { - file_id: FileId(1), - total_size: data.len() as u64, - chunks: vec![ChunkRef { - hash, - offset: 0, - size: data.len() as u32, - }], - }); - - let result = reader.read(FileId(1), 0, data.len() as u32).await.unwrap(); - assert_eq!(&result[..], data); - } - - #[tokio::test] - async fn test_file_reader_partial() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - ..Default::default() - }; - let store = std::sync::Arc::new(CasStore::open(config).await.unwrap()); - - let data = b"ABCDEFGHIJ"; - let hash = store.put(data).await.unwrap(); - - let reader = FileReader::new(store); - reader.register_manifest(ChunkManifest { - file_id: FileId(1), - total_size: data.len() as u64, - chunks: vec![ChunkRef { - hash, - offset: 0, - size: data.len() as u32, - }], - }); - - let result = reader.read(FileId(1), 3, 4).await.unwrap(); - assert_eq!(&result[..], b"DEFG"); - } -} -``` - -### 3.2 Update `musicfs-cas/src/lib.rs` - -```rust -mod store; -mod chunks; -mod reader; - -pub use store::{CasStore, CasConfig, CasError, DedupStats}; -pub use chunks::{ChunkHash, ChunkLocation, ChunkRef}; -pub use reader::{FileReader, ChunkManifest, ReaderError}; -``` - ---- - -## Task 4: FUSE Read Integration - -### 4.1 Update `musicfs-fuse/Cargo.toml` - -```toml -[dependencies] -musicfs-core = { path = "../musicfs-core" } -musicfs-cache = { path = "../musicfs-cache" } -musicfs-cas = { path = "../musicfs-cas" } -musicfs-origins = { path = "../musicfs-origins" } -# ... rest of dependencies -``` - -### 4.2 Update `musicfs-fuse/src/filesystem.rs` read method - -Replace the placeholder `read` implementation: - -```rust -use musicfs_cas::{FileReader, ChunkManifest}; - -pub struct MusicFs { - tree: Arc>, - reader: Arc, - uid: u32, - gid: u32, -} - -impl MusicFs { - pub fn new( - tree: Arc>, - reader: Arc, - ) -> Self { - Self { - tree, - reader, - uid: unsafe { libc::getuid() }, - gid: unsafe { libc::getgid() }, - } - } -} - -// In Filesystem impl: -fn read( - &mut self, - _req: &Request, - ino: u64, - _fh: u64, - offset: i64, - size: u32, - _flags: i32, - _lock_owner: Option, - reply: ReplyData, -) { - debug!("read(ino={}, offset={}, size={})", ino, offset, size); - - let file_id = { - let tree = self.tree.read().unwrap(); - if let Some(VirtualNode::File(file)) = tree.get(ino) { - file.file_id - } else { - reply.error(libc::ENOENT); - return; - } - }; - - // Use tokio runtime for async read - let reader = self.reader.clone(); - let result = tokio::runtime::Handle::current().block_on(async { - reader.read(file_id, offset as u64, size).await - }); - - match result { - Ok(data) => reply.data(&data), - Err(e) => { - warn!("Read error: {}", e); - reply.error(libc::EIO); - } - } -} -``` - ---- - -## Task 5: Integration Tests - -### 5.1 Create `tests/integration/basic_mount.rs` - -```rust -use musicfs_cache::{TreeBuilder, VirtualTree}; -use musicfs_cas::{CasStore, CasConfig, FileReader, ChunkManifest, ChunkRef}; -use musicfs_core::{FileId, FileMeta, OriginId, RealPath, VirtualPath}; -use std::path::PathBuf; -use std::sync::{Arc, RwLock}; -use std::time::SystemTime; -use tempfile::TempDir; - -fn make_file_meta(id: i64, vpath: &str, size: u64) -> FileMeta { - FileMeta { - id: FileId(id), - virtual_path: VirtualPath::new(vpath), - real_path: RealPath { - origin_id: OriginId::from("test"), - path: PathBuf::from("/test"), - }, - size, - mtime: SystemTime::now(), - content_hash: None, - audio: None, - } -} - -#[tokio::test] -async fn test_cas_and_tree_integration() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - ..Default::default() - }; - let store = Arc::new(CasStore::open(config).await.unwrap()); - - let file_data = b"This is test audio file content for testing."; - let chunk_hash = store.put(file_data).await.unwrap(); - - let mut builder = TreeBuilder::new(); - builder.add_file(&make_file_meta(1, "/Artist/Album/Track.flac", file_data.len() as u64)); - let tree = Arc::new(RwLock::new(builder.build())); - - let reader = Arc::new(FileReader::new(store.clone())); - reader.register_manifest(ChunkManifest { - file_id: FileId(1), - total_size: file_data.len() as u64, - chunks: vec![ChunkRef { - hash: chunk_hash, - offset: 0, - size: file_data.len() as u32, - }], - }); - - let result = reader.read(FileId(1), 0, file_data.len() as u32).await.unwrap(); - assert_eq!(&result[..], file_data); -} - -#[tokio::test] -async fn test_cache_persistence() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - ..Default::default() - }; - - let data = b"persistent data"; - let hash = { - let store = CasStore::open(config.clone()).await.unwrap(); - store.put(data).await.unwrap() - }; - - let store = CasStore::open(config).await.unwrap(); - let retrieved = store.get(&hash).await.unwrap(); - assert_eq!(&retrieved[..], data); -} - -#[tokio::test] -async fn test_deduplication() { - let dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: dir.path().join("chunks"), - ..Default::default() - }; - let store = CasStore::open(config).await.unwrap(); - - let data = b"duplicate this content"; - - let hash1 = store.put(data).await.unwrap(); - let size_after_first = store.current_size(); - - let hash2 = store.put(data).await.unwrap(); - let size_after_second = store.current_size(); - - assert_eq!(hash1, hash2); - assert_eq!(size_after_first, size_after_second); -} -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_cas_put_get` | Unit | Basic store/retrieve (FR-20.1) | -| `test_cas_dedup` | Unit | Same content → same hash (FR-20.2) | -| `test_cas_dedup_stats` | Unit | Dedup statistics reported (FR-20.3) | -| `test_cas_integrity` | Unit | Verify chunk hash (FR-20.4) | -| `test_lru_access_order` | Unit | LRU ordering correct | -| `test_lru_reaccess_updates_order` | Unit | Re-access moves to end | -| `test_cache_eviction` | Unit | LRU eviction works (FR-8.4) | -| `test_cache_persistence` | Integration | Survives restart (FR-8.4) | -| `test_file_reader_simple` | Unit | Full file read | -| `test_file_reader_partial` | Unit | Offset/size read | -| `test_cas_and_tree_integration` | Integration | End-to-end read | -| `test_deduplication` | Integration | Dedup saves space | - ---- - -## Exit Criteria - -- [ ] Chunks stored in CAS with deduplication (FR-20.1, FR-20.2) -- [ ] Deduplication statistics reported via `dedup_stats()` (FR-20.3) -- [ ] Chunk integrity verified on read (FR-20.4) -- [ ] Cache size limit enforced via LRU eviction (FR-8.4) -- [ ] Cache persists across daemon restarts (FR-8.4) -- [ ] FUSE `read()` returns actual file content -- [ ] Audio playback works through mounted filesystem -- [ ] All Phase 1 requirements pass acceptance tests - ---- - -## Dependencies to Add - -### Workspace `Cargo.toml` - -```toml -[workspace.dependencies] -# ... existing ... -sled = "0.34" -xxhash-rust = { version = "0.8", features = ["xxh64"] } -bytes = "1" -rmp-serde = "1" # msgpack per architecture 4.3.6 -hex = "0.4" -dirs = "5" # For ~/.cache resolution -tempfile = "3" -``` - ---- - -## Next Week - -Week 5 will implement CDC chunking and delta detection for efficient synchronization. diff --git a/docs/v2/plans/week-04b-origin-connector.md b/docs/v2/plans/week-04b-origin-connector.md deleted file mode 100644 index de2b0f2..0000000 --- a/docs/v2/plans/week-04b-origin-connector.md +++ /dev/null @@ -1,501 +0,0 @@ -# Week 4b: Origin-CAS Connector - -**Phase**: 1 (MVP) -**Prerequisites**: Week 4 (CAS & Chunk Caching) -**Estimated effort**: 1 day - ---- - -## Objective - -Bridge the gap between Origin (source files) and CAS (chunk cache) to enable actual file reads through FUSE. This implements the "cache miss" flow from architecture section 4.3.5. - -**Problem**: Week 4 implemented CAS storage and FileReader, but there's no code that: -1. Detects when requested chunks aren't cached -2. Fetches data from Origin -3. Stores chunks in CAS -4. Creates ChunkManifest for the file - -**Solution**: Create `ContentFetcher` that orchestrates Origin → CAS data flow on cache miss. - ---- - -## Architecture Reference - -From architecture.md section 4.3.5 (Read Operation Activity): - -``` -|CAS| -:compute chunk range for [offset, offset+size]; -if (all chunks cached?) then (yes) - :read from local chunk files; -else (no) - |OriginFederation| - :select healthy origin by priority; - :fetch missing byte range; - |CAS| - :chunk fetched data (CDC); - :store chunks by hash; - :update chunk manifest; -endif -``` - ---- - -## Deliverables - -| Task | Crate | Files | Done | -|------|-------|-------|------| -| ContentFetcher implementation | musicfs-cas | `fetcher.rs` | [ ] | -| FileId → FileMeta resolver | musicfs-cas | `fetcher.rs` | [ ] | -| Update FileReader for cache-miss | musicfs-cas | `reader.rs` | [ ] | -| Update FUSE with fetcher | musicfs-fuse | `filesystem.rs` | [ ] | -| E2E test: cat file through FUSE | tests | `integration.rs` | [ ] | - ---- - -## Task 1: ContentFetcher - -### 1.1 Create `musicfs-cas/src/fetcher.rs` - -```rust -use crate::{CasStore, ChunkManifest, ChunkRef}; -use musicfs_core::{Event, EventBus, FileId, FileMeta, OriginId, RealPath}; -use musicfs_origins::Origin; -use std::collections::HashMap; -use std::path::Path; -use std::sync::{Arc, RwLock}; -use tracing::{debug, info}; - -pub struct ContentFetcher { - store: Arc, - origins: RwLock>>, - file_meta: RwLock>, - event_bus: Option>, -} - -impl ContentFetcher { - pub fn new(store: Arc) -> Self { - Self { - store, - origins: RwLock::new(HashMap::new()), - file_meta: RwLock::new(HashMap::new()), - event_bus: None, - } - } - - pub fn with_event_bus(store: Arc, event_bus: Arc) -> Self { - Self { - store, - origins: RwLock::new(HashMap::new()), - file_meta: RwLock::new(HashMap::new()), - event_bus: Some(event_bus), - } - } - - pub fn register_origin(&self, origin: Arc) { - let id = origin.id().clone(); - self.origins.write().unwrap().insert(id, origin); - } - - pub fn register_file(&self, meta: FileMeta) { - self.file_meta.write().unwrap().insert(meta.id, meta); - } - - pub fn register_files(&self, files: impl IntoIterator) { - let mut map = self.file_meta.write().unwrap(); - for meta in files { - map.insert(meta.id, meta); - } - } - - pub async fn fetch_file(&self, file_id: FileId) -> Result { - let meta = { - let files = self.file_meta.read().unwrap(); - files.get(&file_id).cloned() - .ok_or(FetchError::FileNotFound(file_id))? - }; - - let origin = { - let origins = self.origins.read().unwrap(); - origins.get(&meta.real_path.origin_id).cloned() - .ok_or_else(|| FetchError::OriginNotFound(meta.real_path.origin_id.clone()))? - }; - - info!("Fetching file {:?} from origin {}", file_id, origin.id()); - - let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await - .map_err(|e| FetchError::OriginRead(e.to_string()))?; - - let hash = self.store.put(&data).await - .map_err(FetchError::Store)?; - - let manifest = ChunkManifest { - file_id, - total_size: meta.size, - chunks: vec![ChunkRef { - hash, - offset: 0, - size: data.len() as u32, - }], - }; - - debug!("Created manifest for {:?}: {} bytes, 1 chunk", file_id, meta.size); - - Ok(manifest) - } - - pub fn emit_access_event(&self, meta: &FileMeta, offset: u64, size: u32) { - if let Some(bus) = &self.event_bus { - bus.publish(Event::FileAccessed { - path: meta.virtual_path.clone(), - origin_id: meta.real_path.origin_id.clone(), - offset, - size, - }); - } - } - - pub async fn ensure_cached(&self, file_id: FileId) -> Result { - self.fetch_file(file_id).await - } - - pub fn get_file_meta(&self, file_id: FileId) -> Option { - self.file_meta.read().unwrap().get(&file_id).cloned() - } -} - -#[derive(Debug, thiserror::Error)] -pub enum FetchError { - #[error("File not found: {0:?}")] - FileNotFound(FileId), - - #[error("Origin not found: {0}")] - OriginNotFound(OriginId), - - #[error("Origin read error: {0}")] - OriginRead(String), - - #[error("Store error: {0}")] - Store(#[from] crate::CasError), -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::CasConfig; - use musicfs_core::VirtualPath; - use musicfs_origins::LocalOrigin; - use std::path::PathBuf; - use std::time::SystemTime; - use tempfile::TempDir; - - #[tokio::test] - async fn test_fetch_file() { - let cas_dir = TempDir::new().unwrap(); - let origin_dir = TempDir::new().unwrap(); - - std::fs::write(origin_dir.path().join("test.flac"), b"fake audio data").unwrap(); - - let config = CasConfig { - chunks_dir: cas_dir.path().join("chunks"), - ..Default::default() - }; - let store = Arc::new(CasStore::open(config).await.unwrap()); - let fetcher = ContentFetcher::new(store.clone()); - - let origin = Arc::new(LocalOrigin::new("local", origin_dir.path())); - fetcher.register_origin(origin); - - let meta = FileMeta { - id: FileId(1), - virtual_path: VirtualPath::new("/Artist/Album/test.flac"), - real_path: RealPath { - origin_id: OriginId::from("local"), - path: PathBuf::from("/test.flac"), - }, - size: 15, - mtime: SystemTime::now(), - content_hash: None, - audio: None, - }; - fetcher.register_file(meta); - - let manifest = fetcher.fetch_file(FileId(1)).await.unwrap(); - assert_eq!(manifest.total_size, 15); - assert_eq!(manifest.chunks.len(), 1); - - let data = store.get(&manifest.chunks[0].hash).await.unwrap(); - assert_eq!(&data[..], b"fake audio data"); - } - - #[tokio::test] - async fn test_fetch_file_not_found() { - let cas_dir = TempDir::new().unwrap(); - let config = CasConfig { - chunks_dir: cas_dir.path().join("chunks"), - ..Default::default() - }; - let store = Arc::new(CasStore::open(config).await.unwrap()); - let fetcher = ContentFetcher::new(store); - - let result = fetcher.fetch_file(FileId(999)).await; - assert!(matches!(result, Err(FetchError::FileNotFound(_)))); - } - - #[tokio::test] - async fn test_fetch_emits_event() { - let cas_dir = TempDir::new().unwrap(); - let origin_dir = TempDir::new().unwrap(); - std::fs::write(origin_dir.path().join("test.flac"), b"audio").unwrap(); - - let config = CasConfig { - chunks_dir: cas_dir.path().join("chunks"), - ..Default::default() - }; - let store = Arc::new(CasStore::open(config).await.unwrap()); - let event_bus = Arc::new(EventBus::default()); - let mut rx = event_bus.subscribe(); - - let fetcher = ContentFetcher::with_event_bus(store, event_bus); - let origin = Arc::new(LocalOrigin::new("local", origin_dir.path())); - fetcher.register_origin(origin); - - let meta = FileMeta { - id: FileId(1), - virtual_path: VirtualPath::new("/Artist/test.flac"), - real_path: RealPath { - origin_id: OriginId::from("local"), - path: PathBuf::from("/test.flac"), - }, - size: 5, - mtime: SystemTime::now(), - content_hash: None, - audio: None, - }; - fetcher.register_file(meta.clone()); - - fetcher.emit_access_event(&meta, 0, 5); - - let event = rx.try_recv().unwrap(); - assert!(matches!(event, Event::FileAccessed { .. })); - } -} -``` - ---- - -## Task 2: Update FileReader - -### 2.1 Update `musicfs-cas/src/reader.rs` - -Add fetcher integration for cache-miss handling: - -```rust -use crate::fetcher::{ContentFetcher, FetchError}; - -pub struct FileReader { - store: Arc, - fetcher: Option>, - manifests: RwLock>, -} - -impl FileReader { - pub fn new(store: Arc) -> Self { - Self { - store, - fetcher: None, - manifests: RwLock::new(HashMap::new()), - } - } - - pub fn with_fetcher(store: Arc, fetcher: Arc) -> Self { - Self { - store, - fetcher: Some(fetcher), - manifests: RwLock::new(HashMap::new()), - } - } - - pub async fn read( - &self, - file_id: FileId, - offset: u64, - size: u32, - ) -> Result { - let manifest = self.get_or_fetch_manifest(file_id).await?; - - if let Some(fetcher) = &self.fetcher { - if let Some(meta) = fetcher.get_file_meta(file_id) { - fetcher.emit_access_event(&meta, offset, size); - } - } - - // ... rest of read logic unchanged - } - - async fn get_or_fetch_manifest(&self, file_id: FileId) -> Result { - { - let manifests = self.manifests.read().unwrap(); - if let Some(m) = manifests.get(&file_id) { - return Ok(m.clone()); - } - } - - let Some(fetcher) = &self.fetcher else { - return Err(ReaderError::ManifestNotFound(file_id)); - }; - - let manifest = fetcher.ensure_cached(file_id).await - .map_err(ReaderError::Fetch)?; - - self.manifests.write().unwrap().insert(file_id, manifest.clone()); - Ok(manifest) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum ReaderError { - #[error("Manifest not found for file {0:?}")] - ManifestNotFound(FileId), - - #[error("Fetch error: {0}")] - Fetch(#[from] FetchError), - - #[error("CAS error: {0}")] - Cas(#[from] crate::CasError), -} -``` - ---- - -## Task 3: Update lib.rs - -### 3.1 Update `musicfs-cas/src/lib.rs` - -```rust -mod chunks; -mod fetcher; -mod reader; -mod store; - -pub use chunks::{ChunkLocation, ChunkRef}; -pub use fetcher::{ContentFetcher, FetchError}; -pub use reader::{ChunkManifest, FileReader, ReaderError}; -pub use store::{CasConfig, CasError, CasStore, DedupStats}; -``` - ---- - -## Task 4: Update Cargo.toml - -### 4.1 Update `musicfs-cas/Cargo.toml` - -```toml -[dependencies] -musicfs-core = { path = "../musicfs-core" } -musicfs-origins = { path = "../musicfs-origins" } -# ... rest unchanged -``` - ---- - -## Task 5: Update FUSE Integration - -### 5.1 Update `musicfs-fuse/src/filesystem.rs` - -```rust -use musicfs_cas::{ContentFetcher, FileReader}; - -pub struct MusicFs { - tree: Arc>, - reader: Option>, - fetcher: Option>, - uid: u32, - gid: u32, -} - -impl MusicFs { - pub fn with_content_access( - tree: Arc>, - reader: Arc, - fetcher: Arc, - ) -> Self { - Self { - tree, - reader: Some(reader), - fetcher: Some(fetcher), - uid: unsafe { libc::getuid() }, - gid: unsafe { libc::getgid() }, - } - } -} -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_fetch_file` | Unit | Origin → CAS fetch works | -| `test_fetch_file_not_found` | Unit | Missing file error | -| `test_fetch_emits_event` | Unit | FileAccessed event emitted (FR-18.1) | -| `test_reader_with_fetcher` | Unit | Cache-miss triggers fetch | -| `test_e2e_cat_file` | Integration | `cat` returns file content | - ---- - -## Exit Criteria - -- [ ] `ContentFetcher` fetches from Origin and stores in CAS -- [ ] `FileReader` calls fetcher on cache miss -- [ ] File metadata (FileId → FileMeta) is resolvable -- [ ] `cat /mnt/musicfs/Artist/Album/track.flac` returns actual audio data -- [ ] All existing tests still pass - ---- - -## Dependencies - -### `musicfs-cas/Cargo.toml` - -```toml -[dependencies] -musicfs-origins = { path = "../musicfs-origins" } -``` - ---- - -## Implementation Notes - -1. **Week 4 treated whole files as single chunks** - this continues that approach -2. **CDC chunking deferred to Week 5** - fetcher will be updated then -3. **No OriginFederation yet** - single origin lookup for MVP -4. **FileMeta registration** - caller must register files before they can be fetched -5. **EventBus integration** - emits `FileAccessed` event per FR-18.1 (P0) -6. **Full file fetch** - currently fetches entire file on cache miss; byte-range optimization deferred - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.3.5 | Cache miss → fetch from origin | ✅ | -| 4.3.5 | Store chunks by hash | ✅ | -| 4.3.5 | Update chunk manifest | ✅ | -| 4.3.5 | Emit FileAccessed event | ✅ | -| 4.3.3 | OriginFederation (multi-origin) | ⏳ Deferred | -| 4.3.5 | Byte-range fetch | ⏳ Deferred | -| 4.3.5 | CDC chunking | ⏳ Week 5 | - ---- - -## Next Steps - -After this, the MVP is complete: -- Mount filesystem -- Browse virtual tree (Artist/Album/Track) -- Read actual file content through FUSE -- Audio playback works - -Week 5 adds CDC chunking for efficient delta sync. diff --git a/docs/v2/plans/week-05-cdc-delta.md b/docs/v2/plans/week-05-cdc-delta.md deleted file mode 100644 index 52be7f8..0000000 --- a/docs/v2/plans/week-05-cdc-delta.md +++ /dev/null @@ -1,1219 +0,0 @@ -# Week 5: CDC & Delta Detection - -**Phase**: 2 (Delta Sync & Multi-Origin) -**Prerequisites**: Week 4b (Origin-CAS Connector) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement Content-Defined Chunking (CDC) using FastCDC and delta detection for efficient synchronization. This enables the >90% bandwidth reduction requirement (NFR-6.4) by only transferring changed chunks. - -**Critical Fix**: The MVP performance review identified that `Origin::read()` only returns ~2MB per call due to tokio's async read behavior. This must be fixed as part of CDC implementation since CDC requires the full file content. - ---- - -## Oracle Review Fixes (MUST IMPLEMENT) - -| Severity | Issue | Fix | -|----------|-------|-----| -| 🔴 Critical | **u32 overflow** - `file.size as u32` fails for files >4GB | Add `read_full(path) -> Result>` to Origin trait, use u64 for sizes | -| 🔴 Critical | **Memory explosion** - 200MB+ per file (data + chunk copies) | Use `chunk_refs()` and store immediately, drop source buffer after each chunk | -| 🔴 Critical | **`scan_origin()` is stub** - returns empty Vec, delta detection non-functional | Implement recursive walk using `Origin::readdir()` | -| 🟡 Arch | **Duplicate types** - `FileManifest` duplicates existing `ChunkManifest` | Extend existing `ChunkManifest` with `mtime` field instead of new type | -| 🟡 Arch | **Watcher spawns separate runtime** - wasteful | Use `tokio::task::spawn_blocking` instead of `std::thread::spawn` | -| ⚠️ Watch | No event debouncing (rapid saves flood events) | Add 200ms debounce before emitting events | -| ⚠️ Watch | Missing test for >90% bandwidth reduction claim | Add concrete reuse ratio test with metadata-only file edit | - ---- - -## Architecture Reference - -From architecture.md section 4.3.2 (CAS): - -``` -Avg chunk: 64KB -Min: 16KB, Max: 256KB -Stable boundaries for delta sync -``` - -From section 4.3.5 (Read Operation): - -``` -|CAS| -:chunk fetched data (CDC); -:store chunks by hash; -:update chunk manifest; -``` - ---- - -## Requirements Covered - -| ID | Requirement | Priority | -|----|-------------|----------| -| FR-8.2 | Content-defined chunking for cache efficiency | P0 | -| FR-11.1 | Download only changed portions of files | P0 | -| FR-11.2 | Use CDC to identify changed chunks | P0 | -| FR-11.3 | Preserve unchanged chunks in cache | P0 | -| FR-11.4 | Handle file additions and deletions | P0 | -| FR-10.1 | Detect changes to origin files | P0 | -| FR-10.4 | Compare mtime and size for change detection | P0 | -| NFR-6.4 | Delta sync >90% bandwidth reduction | P0 | - ---- - -## Deliverables - -| Task | Crate | Files | Est. | -|------|-------|-------|------| -| Fix async read (read full file) | musicfs-origins | `local.rs` | 0.5d | -| FastCDC integration | musicfs-sync | `cdc.rs` | 1d | -| ChunkManifest persistence | musicfs-sync | `manifest.rs` | 0.5d | -| Delta detector | musicfs-sync | `delta.rs` | 1d | -| Change watcher (inotify) | musicfs-sync | `watcher.rs` | 1d | -| Update ContentFetcher for CDC | musicfs-cas | `fetcher.rs` | 0.5d | -| Integration tests | tests | `delta_sync.rs` | 0.5d | - ---- - -## Task 1: Fix Async Read - -### 1.1 Problem - -Current `LocalOrigin::read()` uses `file.read()` which returns when the kernel buffer is exhausted (~2MB), not when the requested size is read. - -### 1.2 Update Origin trait to add `read_full()` method - -Add to `musicfs-origins/src/traits.rs`: - -```rust -/// Read entire file content (for CDC chunking) -/// NOTE: Use u64 for size to support files >4GB -async fn read_full(&self, path: &Path) -> Result>; -``` - -### 1.3 Update `musicfs-origins/src/local.rs` - -```rust -async fn read(&self, path: &Path, offset: u64, size: u64) -> Result> { - use tokio::io::{AsyncReadExt, AsyncSeekExt}; - - let full_path = self.full_path(path); - debug!( - "LocalOrigin::read({:?}, offset={}, size={})", - full_path, offset, size - ); - - let mut file = fs::File::open(&full_path).await?; - file.seek(std::io::SeekFrom::Start(offset)).await?; - - // FIX: Use loop instead of single read() to get all requested bytes - let mut buffer = Vec::with_capacity(size as usize); - - // Read until we have all requested bytes or EOF - let mut total_read = 0u64; - let mut temp_buf = vec![0u8; 64 * 1024]; // 64KB chunks - - while total_read < size { - let to_read = std::cmp::min(temp_buf.len() as u64, size - total_read) as usize; - let n = file.read(&mut temp_buf[..to_read]).await?; - if n == 0 { - break; // EOF - } - buffer.extend_from_slice(&temp_buf[..n]); - total_read += n as u64; - } - - Ok(buffer) -} - -/// Read entire file (Oracle fix: separate method to avoid u32 overflow) -async fn read_full(&self, path: &Path) -> Result> { - let full_path = self.full_path(path); - debug!("LocalOrigin::read_full({:?})", full_path); - Ok(tokio::fs::read(&full_path).await?) -} -``` - -**NOTE**: Change `size: u32` to `size: u64` throughout the Origin trait to support files >4GB. - ---- - -## Task 2: FastCDC Integration - -### 2.1 Add dependencies to `musicfs-sync/Cargo.toml` - -```toml -[dependencies] -musicfs-core = { path = "../musicfs-core" } -musicfs-cas = { path = "../musicfs-cas" } - -fastcdc = "3" -xxhash-rust = { version = "0.8", features = ["xxh64"] } -tokio = { workspace = true } -tracing = { workspace = true } -thiserror = { workspace = true } -serde = { workspace = true } -``` - -### 2.2 Create `musicfs-sync/src/cdc.rs` - -```rust -use fastcdc::v2020::FastCDC; -use musicfs_core::ChunkHash; -use xxhash_rust::xxh64::xxh64; - -/// CDC chunker configuration per architecture spec -pub struct CdcChunker { - min_size: u32, // 16 KB - avg_size: u32, // 64 KB - max_size: u32, // 256 KB -} - -impl Default for CdcChunker { - fn default() -> Self { - Self { - min_size: 16 * 1024, - avg_size: 64 * 1024, - max_size: 256 * 1024, - } - } -} - -/// A chunk produced by CDC -#[derive(Debug, Clone)] -pub struct Chunk { - pub hash: ChunkHash, - pub offset: u64, - pub length: u32, - pub data: Vec, -} - -impl CdcChunker { - pub fn new(min_size: u32, avg_size: u32, max_size: u32) -> Self { - Self { min_size, avg_size, max_size } - } - - /// Chunk data using FastCDC algorithm - /// Returns chunks with stable boundaries for delta sync - /// - /// WARNING: This copies all chunk data. For large files, use `chunk_refs()` - /// and store immediately to avoid memory explosion. - pub fn chunk(&self, data: &[u8]) -> Vec { - let chunker = FastCDC::new( - data, - self.min_size, - self.avg_size, - self.max_size, - ); - - chunker - .map(|c| { - let chunk_data = &data[c.offset..c.offset + c.length]; - let hash = ChunkHash::from_bytes(chunk_data); - - Chunk { - hash, - offset: c.offset as u64, - length: c.length as u32, - data: chunk_data.to_vec(), - } - }) - .collect() - } - - /// Chunk data without copying (returns references) - PREFERRED for large files - /// - /// Oracle fix: Use this method and store each chunk immediately before - /// processing the next to avoid 200MB+ memory usage per file. - pub fn chunk_refs<'a>(&self, data: &'a [u8]) -> Vec> { - let chunker = FastCDC::new( - data, - self.min_size, - self.avg_size, - self.max_size, - ); - - chunker - .map(|c| { - let chunk_data = &data[c.offset..c.offset + c.length]; - ChunkRef { - hash: ChunkHash::from_bytes(chunk_data), - offset: c.offset as u64, - length: c.length as u32, - data: chunk_data, - } - }) - .collect() - } - - /// Stream-process chunks to minimize memory (Oracle fix: avoid memory explosion) - /// Calls `processor` for each chunk, allowing immediate storage before next chunk - pub fn chunk_streaming(&self, data: &[u8], mut processor: F) -> usize - where - F: FnMut(ChunkRef<'_>), - { - let chunker = FastCDC::new( - data, - self.min_size, - self.avg_size, - self.max_size, - ); - - let mut count = 0; - for c in chunker { - let chunk_data = &data[c.offset..c.offset + c.length]; - processor(ChunkRef { - hash: ChunkHash::from_bytes(chunk_data), - offset: c.offset as u64, - length: c.length as u32, - data: chunk_data, - }); - count += 1; - } - count - } -} - -#[derive(Debug)] -pub struct ChunkRef<'a> { - pub hash: ChunkHash, - pub offset: u64, - pub length: u32, - pub data: &'a [u8], -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_cdc_basic() { - let chunker = CdcChunker::default(); - let data = vec![0u8; 256 * 1024]; // 256KB of zeros - - let chunks = chunker.chunk(&data); - - // Should produce multiple chunks - assert!(!chunks.is_empty()); - - // Total size should match - let total: u64 = chunks.iter().map(|c| c.length as u64).sum(); - assert_eq!(total, data.len() as u64); - - // Chunks should be contiguous - let mut offset = 0u64; - for chunk in &chunks { - assert_eq!(chunk.offset, offset); - offset += chunk.length as u64; - } - } - - #[test] - fn test_cdc_stable_boundaries() { - let chunker = CdcChunker::default(); - - // Original data - let mut data1 = vec![0u8; 128 * 1024]; - for (i, b) in data1.iter_mut().enumerate() { - *b = (i % 256) as u8; - } - - // Data with insertion at start (should only affect first chunk) - let mut data2 = vec![0xFFu8; 1024]; // 1KB insertion - data2.extend_from_slice(&data1); - - let chunks1 = chunker.chunk(&data1); - let chunks2 = chunker.chunk(&data2); - - // Most chunk hashes should be shared (CDC stability) - let hashes1: std::collections::HashSet<_> = chunks1.iter().map(|c| c.hash).collect(); - let hashes2: std::collections::HashSet<_> = chunks2.iter().map(|c| c.hash).collect(); - - let shared = hashes1.intersection(&hashes2).count(); - - // At least 50% of chunks should be reusable - // (In practice, CDC achieves much better than this) - assert!(shared > 0, "CDC should produce stable boundaries"); - } - - #[test] - fn test_cdc_chunk_sizes() { - let chunker = CdcChunker::default(); - - // Random-ish data (to avoid degenerate cases) - let data: Vec = (0..1024 * 1024) - .map(|i| ((i * 17 + 31) % 256) as u8) - .collect(); - - let chunks = chunker.chunk(&data); - - for chunk in &chunks { - // Chunks should respect size bounds (with some tolerance for last chunk) - if chunk.offset + chunk.length as u64 != data.len() as u64 { - assert!(chunk.length >= chunker.min_size / 2, - "Chunk too small: {}", chunk.length); - assert!(chunk.length <= chunker.max_size * 2, - "Chunk too large: {}", chunk.length); - } - } - } -} -``` - ---- - -## Task 3: Manifest Persistence - -### 3.1 Extend existing `ChunkManifest` in `musicfs-cas/src/manifest.rs` - -**Oracle fix**: Don't create duplicate `FileManifest` type. Extend existing `ChunkManifest` with `mtime` field. - -```rust -use musicfs_core::{ChunkHash, FileId}; -use serde::{Deserialize, Serialize}; - -/// Persistent chunk manifest for a file -/// NOTE: Extended from original to include mtime for delta detection -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct ChunkManifest { - pub file_id: FileId, - pub total_size: u64, - pub mtime: i64, // Oracle fix: added for delta detection - pub chunks: Vec, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct ManifestChunk { - pub hash: ChunkHash, - pub offset: u64, - pub size: u32, -} - -impl FileManifest { - pub fn new(file_id: FileId, total_size: u64, mtime: i64) -> Self { - Self { - file_id, - total_size, - mtime, - chunks: Vec::new(), - } - } - - pub fn add_chunk(&mut self, hash: ChunkHash, offset: u64, size: u32) { - self.chunks.push(ManifestChunk { hash, offset, size }); - } - - /// Serialize to msgpack for storage in SQLite - pub fn to_bytes(&self) -> Vec { - rmp_serde::to_vec(self).unwrap_or_default() - } - - /// Deserialize from msgpack - pub fn from_bytes(data: &[u8]) -> Option { - rmp_serde::from_slice(data).ok() - } - - /// Get all unique chunk hashes - pub fn chunk_hashes(&self) -> impl Iterator { - self.chunks.iter().map(|c| &c.hash) - } -} - -/// Result of comparing two manifests -#[derive(Debug)] -pub struct ManifestDiff { - /// Chunks in new manifest that exist in old (reusable) - pub reuse: Vec, - /// Chunks in new manifest that don't exist in old (need fetch) - pub fetch: Vec, - /// Chunks in old manifest that don't exist in new (can evict) - pub orphaned: Vec, -} - -impl FileManifest { - /// Compare this manifest to a new one - pub fn diff(&self, new_chunks: &[ManifestChunk]) -> ManifestDiff { - use std::collections::HashSet; - - let old_hashes: HashSet<_> = self.chunks.iter().map(|c| c.hash).collect(); - let new_hashes: HashSet<_> = new_chunks.iter().map(|c| c.hash).collect(); - - ManifestDiff { - reuse: new_chunks.iter() - .filter(|c| old_hashes.contains(&c.hash)) - .cloned() - .collect(), - fetch: new_chunks.iter() - .filter(|c| !old_hashes.contains(&c.hash)) - .cloned() - .collect(), - orphaned: self.chunks.iter() - .filter(|c| !new_hashes.contains(&c.hash)) - .map(|c| c.hash) - .collect(), - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_manifest_roundtrip() { - let mut manifest = FileManifest::new(FileId(1), 1024, 12345); - manifest.add_chunk(ChunkHash::from_bytes(b"chunk1"), 0, 512); - manifest.add_chunk(ChunkHash::from_bytes(b"chunk2"), 512, 512); - - let bytes = manifest.to_bytes(); - let restored = FileManifest::from_bytes(&bytes).unwrap(); - - assert_eq!(restored.file_id, manifest.file_id); - assert_eq!(restored.chunks.len(), 2); - } - - #[test] - fn test_manifest_diff() { - let mut old = FileManifest::new(FileId(1), 1024, 12345); - old.add_chunk(ChunkHash::from_bytes(b"A"), 0, 256); - old.add_chunk(ChunkHash::from_bytes(b"B"), 256, 256); - old.add_chunk(ChunkHash::from_bytes(b"C"), 512, 256); - old.add_chunk(ChunkHash::from_bytes(b"D"), 768, 256); - - // New manifest: A stays, B removed, C stays, D removed, E added - let new_chunks = vec![ - ManifestChunk { hash: ChunkHash::from_bytes(b"A"), offset: 0, size: 256 }, - ManifestChunk { hash: ChunkHash::from_bytes(b"C"), offset: 256, size: 256 }, - ManifestChunk { hash: ChunkHash::from_bytes(b"E"), offset: 512, size: 256 }, - ]; - - let diff = old.diff(&new_chunks); - - assert_eq!(diff.reuse.len(), 2); // A, C - assert_eq!(diff.fetch.len(), 1); // E - assert_eq!(diff.orphaned.len(), 2); // B, D - } -} -``` - ---- - -## Task 4: Delta Detector - -### 4.1 Create `musicfs-sync/src/delta.rs` - -```rust -use crate::cdc::CdcChunker; -use crate::manifest::{FileManifest, ManifestChunk, ManifestDiff}; -use musicfs_core::{FileId, FileMeta, OriginId}; -use musicfs_origins::Origin; -use std::collections::HashMap; -use std::sync::Arc; -use std::time::SystemTime; -use tracing::{debug, info}; - -/// Detected changes between origin and cache -#[derive(Debug, Default)] -pub struct ChangeSet { - pub added: Vec, - pub removed: Vec, - pub modified: Vec<(FileId, ManifestDiff)>, -} - -impl ChangeSet { - pub fn is_empty(&self) -> bool { - self.added.is_empty() && self.removed.is_empty() && self.modified.is_empty() - } - - pub fn total_changes(&self) -> usize { - self.added.len() + self.removed.len() + self.modified.len() - } -} - -/// Delta detector compares origin state to cached state -pub struct DeltaDetector { - chunker: CdcChunker, -} - -impl DeltaDetector { - pub fn new() -> Self { - Self { - chunker: CdcChunker::default(), - } - } - - pub fn with_chunker(chunker: CdcChunker) -> Self { - Self { chunker } - } - - /// Detect changes between cached files and origin - pub async fn detect_changes( - &self, - origin: &dyn Origin, - cached: &HashMap, - manifests: &HashMap, - ) -> Result { - let mut changes = ChangeSet::default(); - - // Scan origin for current files - let origin_files = self.scan_origin(origin).await?; - - // Build lookup by real path - let cached_by_path: HashMap<_, _> = cached.values() - .map(|m| (m.real_path.path.clone(), m)) - .collect(); - - // Check for added/modified - for origin_file in &origin_files { - if let Some(cached_file) = cached_by_path.get(&origin_file.real_path.path) { - // File exists - check if modified - if self.is_modified(cached_file, origin_file) { - debug!("File modified: {:?}", origin_file.real_path.path); - - if let Some(old_manifest) = manifests.get(&cached_file.id) { - // Compute new chunks and diff - let new_chunks = self.compute_chunks(origin, origin_file).await?; - let diff = old_manifest.diff(&new_chunks); - changes.modified.push((cached_file.id, diff)); - } - } - } else { - // New file - debug!("File added: {:?}", origin_file.real_path.path); - changes.added.push(origin_file.clone()); - } - } - - // Check for removed - let origin_paths: std::collections::HashSet<_> = origin_files.iter() - .map(|f| &f.real_path.path) - .collect(); - - for cached_file in cached.values() { - if !origin_paths.contains(&cached_file.real_path.path) { - debug!("File removed: {:?}", cached_file.real_path.path); - changes.removed.push(cached_file.id); - } - } - - info!( - "Delta detection complete: {} added, {} removed, {} modified", - changes.added.len(), - changes.removed.len(), - changes.modified.len() - ); - - Ok(changes) - } - - /// Check if file was modified based on mtime/size - fn is_modified(&self, cached: &FileMeta, origin: &FileMeta) -> bool { - cached.size != origin.size || cached.mtime != origin.mtime - } - - /// Scan origin for all files (Oracle fix: implement recursive walk) - async fn scan_origin(&self, origin: &dyn Origin) -> Result, DeltaError> { - let mut files = Vec::new(); - let mut dirs_to_scan = vec![PathBuf::from("/")]; - - while let Some(dir) = dirs_to_scan.pop() { - let entries = origin.readdir(&dir) - .await - .map_err(|e| DeltaError::OriginScan(e.to_string()))?; - - for entry in entries { - let entry_path = dir.join(&entry.name); - - if entry.is_dir { - dirs_to_scan.push(entry_path); - } else if Self::is_audio_file(&entry.name) { - // Get full stat for mtime - let stat = origin.stat(&entry_path) - .await - .map_err(|e| DeltaError::OriginScan(e.to_string()))?; - - files.push(FileMeta { - id: FileId(0), // Will be assigned by caller - virtual_path: VirtualPath::new(&format!("{}", entry_path.display())), - real_path: RealPath { - origin_id: origin.id().clone(), - path: entry_path, - }, - size: stat.size, - mtime: stat.mtime, - content_hash: None, - audio: None, - }); - } - } - } - - Ok(files) - } - - /// Check if file is an audio file by extension - fn is_audio_file(name: &str) -> bool { - let lower = name.to_lowercase(); - lower.ends_with(".flac") || lower.ends_with(".mp3") || - lower.ends_with(".ogg") || lower.ends_with(".wav") || - lower.ends_with(".m4a") || lower.ends_with(".aac") || - lower.ends_with(".opus") - } - - /// Compute CDC chunks for a file - async fn compute_chunks( - &self, - origin: &dyn Origin, - file: &FileMeta, - ) -> Result, DeltaError> { - let data = origin - .read(&file.real_path.path, 0, file.size as u32) - .await - .map_err(|e| DeltaError::OriginRead(e.to_string()))?; - - let chunks = self.chunker.chunk(&data); - - Ok(chunks - .into_iter() - .map(|c| ManifestChunk { - hash: c.hash, - offset: c.offset, - size: c.length, - }) - .collect()) - } -} - -impl Default for DeltaDetector { - fn default() -> Self { - Self::new() - } -} - -#[derive(Debug, thiserror::Error)] -pub enum DeltaError { - #[error("Origin read error: {0}")] - OriginRead(String), - - #[error("Origin scan error: {0}")] - OriginScan(String), -} - -#[cfg(test)] -mod tests { - use super::*; - use musicfs_core::{RealPath, VirtualPath}; - use std::path::PathBuf; - - fn make_file_meta(id: i64, path: &str, size: u64) -> FileMeta { - FileMeta { - id: FileId(id), - virtual_path: VirtualPath::new(&format!("/test/{}", path)), - real_path: RealPath { - origin_id: OriginId::from("test"), - path: PathBuf::from(path), - }, - size, - mtime: SystemTime::UNIX_EPOCH, - content_hash: None, - audio: None, - } - } - - #[test] - fn test_is_modified_size_change() { - let detector = DeltaDetector::new(); - - let cached = make_file_meta(1, "test.flac", 1000); - let mut origin = cached.clone(); - origin.size = 2000; - - assert!(detector.is_modified(&cached, &origin)); - } - - #[test] - fn test_is_modified_same() { - let detector = DeltaDetector::new(); - - let cached = make_file_meta(1, "test.flac", 1000); - let origin = cached.clone(); - - assert!(!detector.is_modified(&cached, &origin)); - } -} -``` - ---- - -## Task 5: File Watcher - -### 5.1 Create `musicfs-sync/src/watcher.rs` - -```rust -use musicfs_core::{Event, EventBus, OriginId}; -use notify::{Config, RecommendedWatcher, RecursiveMode, Watcher}; -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use tokio::sync::mpsc; -use tracing::{debug, error, info, warn}; - -/// Watches origin filesystem for changes (inotify on Linux) -pub struct OriginWatcher { - origin_id: OriginId, - root: PathBuf, - event_bus: Arc, -} - -impl OriginWatcher { - pub fn new(origin_id: OriginId, root: PathBuf, event_bus: Arc) -> Self { - Self { - origin_id, - root, - event_bus, - } - } - - /// Start watching for changes - /// Returns a handle that stops watching when dropped - /// - /// Oracle fix: Use spawn_blocking instead of spawning separate runtime - pub fn start(self) -> WatchHandle { - let (stop_tx, mut stop_rx) = mpsc::channel::<()>(1); - - let origin_id = self.origin_id.clone(); - let root = self.root.clone(); - let event_bus = self.event_bus.clone(); - - // Oracle fix: Use tokio::task::spawn_blocking instead of std::thread::spawn - // This integrates with existing runtime rather than creating a new one - tokio::task::spawn_blocking(move || { - let rt = tokio::runtime::Builder::new_current_thread() - .enable_all() - .build() - .unwrap(); - - rt.block_on(async { - if let Err(e) = Self::watch_loop(&origin_id, &root, &event_bus, &mut stop_rx).await { - error!("Watcher error: {}", e); - } - }); - }); - - WatchHandle { stop_tx } - } - - async fn watch_loop( - origin_id: &OriginId, - root: &Path, - event_bus: &EventBus, - stop_rx: &mut mpsc::Receiver<()>, - ) -> Result<(), WatchError> { - let (tx, mut rx) = mpsc::channel(100); - - let mut watcher = RecommendedWatcher::new( - move |res: Result| { - if let Ok(event) = res { - let _ = tx.blocking_send(event); - } - }, - Config::default(), - ) - .map_err(|e| WatchError::Init(e.to_string()))?; - - watcher - .watch(root, RecursiveMode::Recursive) - .map_err(|e| WatchError::Watch(e.to_string()))?; - - info!("Watching origin {} at {:?}", origin_id, root); - - loop { - tokio::select! { - Some(event) = rx.recv() => { - Self::handle_notify_event(origin_id, root, event_bus, event); - } - _ = stop_rx.recv() => { - info!("Stopping watcher for {}", origin_id); - break; - } - } - } - - Ok(()) - } - - /// Oracle fix: Add debouncing to handle rapid saves - /// Debounce window before emitting events - const DEBOUNCE_MS: u64 = 200; - - fn handle_notify_event( - origin_id: &OriginId, - root: &Path, - event_bus: &EventBus, - event: notify::Event, - debouncer: &mut HashMap, - ) { - use notify::EventKind; - - let now = Instant::now(); - - for path in event.paths { - let relative = match path.strip_prefix(root) { - Ok(p) => p.to_path_buf(), - Err(_) => continue, - }; - - // Only care about audio files - if !Self::is_audio_file(&path) { - continue; - } - - // Oracle fix: Debounce - skip if we saw this path recently - if let Some(last_seen) = debouncer.get(&relative) { - if now.duration_since(*last_seen).as_millis() < Self::DEBOUNCE_MS as u128 { - debug!("Debouncing event for {:?}", relative); - continue; - } - } - debouncer.insert(relative.clone(), now); - - let vpath = musicfs_core::VirtualPath::new(&format!("/{}", relative.display())); - - match event.kind { - EventKind::Create(_) => { - debug!("File created: {:?}", relative); - event_bus.publish(Event::FileAdded { - path: vpath, - origin_id: origin_id.clone(), - }); - } - EventKind::Remove(_) => { - debug!("File removed: {:?}", relative); - event_bus.publish(Event::FileRemoved { path: vpath }); - } - EventKind::Modify(_) => { - debug!("File modified: {:?}", relative); - event_bus.publish(Event::FileModified { path: vpath }); - } - _ => {} - } - } - } - - fn is_audio_file(path: &Path) -> bool { - matches!( - path.extension().and_then(|e| e.to_str()).map(|e| e.to_lowercase()).as_deref(), - Some("flac" | "mp3" | "ogg" | "wav" | "m4a" | "aac" | "opus") - ) - } -} - -pub struct WatchHandle { - stop_tx: mpsc::Sender<()>, -} - -impl WatchHandle { - pub async fn stop(self) { - let _ = self.stop_tx.send(()).await; - } -} - -impl Drop for WatchHandle { - fn drop(&mut self) { - // Best effort stop on drop - let _ = self.stop_tx.try_send(()); - } -} - -#[derive(Debug, thiserror::Error)] -pub enum WatchError { - #[error("Failed to initialize watcher: {0}")] - Init(String), - - #[error("Failed to watch path: {0}")] - Watch(String), -} - -#[cfg(test)] -mod tests { - use super::*; - use std::time::Duration; - use tempfile::TempDir; - - #[tokio::test] - async fn test_watcher_detects_create() { - let dir = TempDir::new().unwrap(); - let event_bus = Arc::new(EventBus::default()); - let mut rx = event_bus.subscribe(); - - let watcher = OriginWatcher::new( - OriginId::from("test"), - dir.path().to_path_buf(), - event_bus, - ); - let handle = watcher.start(); - - // Give watcher time to start - tokio::time::sleep(Duration::from_millis(100)).await; - - // Create a file - std::fs::write(dir.path().join("test.flac"), b"audio").unwrap(); - - // Wait for event - tokio::time::sleep(Duration::from_millis(200)).await; - - // Should receive FileAdded event - let event = rx.try_recv(); - assert!(matches!(event, Ok(Event::FileAdded { .. }))); - - handle.stop().await; - } -} -``` - ---- - -## Task 6: Update ContentFetcher for CDC - -### 6.1 Update `musicfs-cas/src/fetcher.rs` - -```rust -use crate::{CasStore, ChunkManifest, ChunkRef}; -use musicfs_core::{ChunkHash, Event, EventBus, FileId, FileMeta, OriginId}; -use musicfs_origins::Origin; -use musicfs_sync::cdc::CdcChunker; -use std::collections::HashMap; -use std::sync::{Arc, RwLock}; -use tracing::{debug, info}; - -pub struct ContentFetcher { - store: Arc, - origins: RwLock>>, - file_meta: RwLock>, - event_bus: Option>, - chunker: CdcChunker, -} - -impl ContentFetcher { - pub fn new(store: Arc) -> Self { - Self { - store, - origins: RwLock::new(HashMap::new()), - file_meta: RwLock::new(HashMap::new()), - event_bus: None, - chunker: CdcChunker::default(), - } - } - - // ... existing methods ... - - /// Fetch file with CDC chunking - pub async fn fetch_file(&self, file_id: FileId) -> Result { - let meta = { - let files = self.file_meta.read().unwrap(); - files.get(&file_id).cloned() - .ok_or(FetchError::FileNotFound(file_id))? - }; - - let origin = { - let origins = self.origins.read().unwrap(); - origins.get(&meta.real_path.origin_id).cloned() - .ok_or_else(|| FetchError::OriginNotFound(meta.real_path.origin_id.clone()))? - }; - - info!("Fetching file {:?} from origin {}", file_id, origin.id()); - - // Read full file content - let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await - .map_err(|e| FetchError::OriginRead(e.to_string()))?; - - // CDC chunk the data - let chunks = self.chunker.chunk(&data); - info!("Chunked {:?} into {} chunks", file_id, chunks.len()); - - // Store each chunk in CAS - let mut chunk_refs = Vec::with_capacity(chunks.len()); - for chunk in chunks { - // Dedup: only store if not already present - if !self.store.exists(&chunk.hash) { - self.store.put(&chunk.data).await - .map_err(FetchError::Store)?; - } - - chunk_refs.push(ChunkRef { - hash: chunk.hash, - offset: chunk.offset, - size: chunk.length, - }); - } - - let manifest = ChunkManifest { - file_id, - total_size: meta.size, - chunks: chunk_refs, - }; - - debug!( - "Created manifest for {:?}: {} bytes, {} chunks", - file_id, meta.size, manifest.chunks.len() - ); - - Ok(manifest) - } -} -``` - ---- - -## Task 7: Update lib.rs - -### 7.1 Create `musicfs-sync/src/lib.rs` - -```rust -pub mod cdc; -pub mod delta; -pub mod manifest; -pub mod watcher; - -pub use cdc::{CdcChunker, Chunk}; -pub use delta::{ChangeSet, DeltaDetector, DeltaError}; -pub use manifest::{FileManifest, ManifestChunk, ManifestDiff}; -pub use watcher::{OriginWatcher, WatchHandle, WatchError}; -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_read_full_file` | Unit | Fix: full file read works | -| `test_read_full_large_file` | Unit | Oracle fix: files >4GB don't overflow | -| `test_cdc_basic` | Unit | CDC produces chunks | -| `test_cdc_stable_boundaries` | Unit | Insertions don't shift all chunks | -| `test_cdc_chunk_sizes` | Unit | Chunks respect min/avg/max | -| `test_cdc_streaming_memory` | Unit | Oracle fix: streaming doesn't explode memory | -| `test_manifest_roundtrip` | Unit | Manifest serialization | -| `test_manifest_diff` | Unit | Diff identifies reuse/fetch/orphan | -| `test_delta_detect_modified` | Unit | Modified files detected | -| `test_scan_origin_recursive` | Unit | Oracle fix: scan_origin finds all files | -| `test_watcher_detects_create` | Integration | inotify works | -| `test_watcher_debounce` | Unit | Oracle fix: rapid events debounced | -| `test_bandwidth_reduction_90pct` | Integration | Oracle fix: >90% reduction on metadata edit | - -### Oracle fix: Add concrete bandwidth reduction test - -```rust -#[tokio::test] -async fn test_bandwidth_reduction_90pct() { - // Create a 10MB FLAC file - let original = create_test_flac(10 * 1024 * 1024); - - // Chunk it - let chunker = CdcChunker::default(); - let chunks1 = chunker.chunk(&original); - let hashes1: HashSet<_> = chunks1.iter().map(|c| c.hash).collect(); - - // Modify only metadata (first 1KB - FLAC header area) - let mut modified = original.clone(); - for i in 100..200 { - modified[i] = 0xFF; - } - - // Chunk modified version - let chunks2 = chunker.chunk(&modified); - let hashes2: HashSet<_> = chunks2.iter().map(|c| c.hash).collect(); - - // Calculate reuse ratio - let reused = hashes1.intersection(&hashes2).count(); - let reuse_ratio = reused as f64 / chunks2.len() as f64; - - // Must achieve >90% reuse for metadata-only edit - assert!( - reuse_ratio > 0.90, - "Bandwidth reduction {:.1}% < 90% target. Reused {}/{} chunks", - reuse_ratio * 100.0, reused, chunks2.len() - ); -} -``` - ---- - -## Benchmark - -```rust -// benches/cdc.rs -fn bench_cdc_64mb(c: &mut Criterion) { - let chunker = CdcChunker::default(); - let data = vec![0u8; 64 * 1024 * 1024]; - - c.bench_function("cdc_64mb", |b| { - b.iter(|| chunker.chunk(&data)) - }); -} - -fn bench_bandwidth_reduction(c: &mut Criterion) { - // Simulate metadata-only edit (tag change) - // Measure chunk reuse ratio -} -``` - ---- - -## Exit Criteria - -- [ ] Full file content is read (not just first 2MB) -- [ ] CDC produces 16KB-256KB chunks with 64KB average -- [ ] Chunk boundaries are stable on insertions -- [ ] Manifest diff correctly identifies reuse/fetch/orphan -- [ ] inotify watcher detects file changes -- [ ] Delta sync achieves >90% bandwidth reduction on metadata edit -- [ ] All existing tests pass - ---- - -## Dependencies - -### `musicfs-sync/Cargo.toml` - -```toml -[package] -name = "musicfs-sync" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -musicfs-cas = { path = "../musicfs-cas" } -musicfs-origins = { path = "../musicfs-origins" } - -fastcdc = "3" -xxhash-rust = { version = "0.8", features = ["xxh64"] } -notify = "6" -rmp-serde = "1" - -tokio = { workspace = true } -tracing = { workspace = true } -thiserror = { workspace = true } -serde = { workspace = true } - -[dev-dependencies] -tempfile = { workspace = true } -``` - ---- - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.3.2 | CDC chunking (64KB avg) | ✅ | -| 4.3.2 | Min 16KB, Max 256KB | ✅ | -| 4.3.2 | Stable boundaries for delta sync | ✅ | -| 4.3.5 | Chunk fetched data (CDC) | ✅ | -| 4.3.5 | Store chunks by hash | ✅ | -| FR-10.2 | inotify for local origins | ✅ | -| FR-11.2 | Use CDC to identify changed chunks | ✅ | -| NFR-6.4 | >90% bandwidth reduction | ✅ | diff --git a/docs/v2/plans/week-06-origin-federation.md b/docs/v2/plans/week-06-origin-federation.md deleted file mode 100644 index 024d6e0..0000000 --- a/docs/v2/plans/week-06-origin-federation.md +++ /dev/null @@ -1,1113 +0,0 @@ -# Week 6: Origin Federation - -**Phase**: 2 (Delta Sync & Multi-Origin) -**Prerequisites**: Week 5 (CDC & Delta Detection) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement multi-origin support with priority-based routing, health monitoring, and automatic failover. This enables serving files from multiple storage backends with graceful degradation. - ---- - -## Oracle Review Fixes (MUST IMPLEMENT) - -| Severity | Issue | Fix | -|----------|-------|-----| -| 🔴 Critical | **All origins unhealthy** - no defined behavior | Emit event, serve from cache, select "least-bad" origin with fewest failures | -| 🔴 Critical | **Watch handle cleanup** - not specified on `unregister()` | Track active watches per-origin in registry, drop handles on removal | -| 🟡 Medium | **Routing formula ambiguous** - text says multiplication, code shows tuple | Clarify: use tuple `(priority, latency)` for priority-dominant ordering | -| 🟡 Medium | **Health threshold hardcoded** - 3 failures for all origin types | Make configurable per `OriginType` (Local=1, Remote=3) | -| ⚠️ Watch | **Retry backoff mismatch** - Plan: 100ms×2.0, Spec NFR-7.3: 100ms, 500ms, 2s | Align with spec: use 100ms, 500ms, 2000ms sequence | - ---- - -## Architecture Reference - -From architecture.md section 4.3.3 (Origin Federation): - -```plantuml -VPR -> OF : read(real_path, offset, size) -OF -> OF : select_origin(priority, health) - -alt Origin[Local] healthy (pri=1) - OF -> O1 : read() - O1 --> OF : data -else Origin[Local] unhealthy, try NFS (pri=2) - OF -> O2 : read() - ... -end -``` - -From section 4.3.3: -> "Background health checks every 30s per origin" - ---- - -## Requirements Covered - -| ID | Requirement | Priority | -|----|-------------|----------| -| FR-13.1 | Support multiple simultaneous origins | P0 | -| FR-13.2 | Present unified virtual tree across origins | P0 | -| FR-13.3 | Support origin priority/preference ordering | P0 | -| FR-13.4 | Handle duplicate files across origins | P0 | -| FR-13.5 | Support per-origin configuration | P0 | -| NFR-7.1 | Serve cached data when origin unavailable | P0 | -| NFR-7.2 | Gracefully degrade with network failures | P0 | -| NFR-7.3 | Retry failed operations with exponential backoff | P0 | - ---- - -## Deliverables - -| Task | Crate | Files | Est. | -|------|-------|-------|------| -| Origin registry | musicfs-origins | `registry.rs` | 0.5d | -| Priority router | musicfs-origins | `router.rs` | 1d | -| Health monitor | musicfs-origins | `health.rs` | 1d | -| Failover logic | musicfs-origins | `failover.rs` | 1d | -| Origin configuration | musicfs-core | `config.rs` | 0.5d | -| Integration with FUSE | musicfs-fuse | updates | 0.5d | -| Integration tests | tests | `federation.rs` | 0.5d | - ---- - -## Task 1: Origin Configuration - -### 1.1 Create `musicfs-core/src/config.rs` - -```rust -use crate::OriginId; -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; -use std::path::PathBuf; - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct Config { - pub mount_point: PathBuf, - pub cache_dir: PathBuf, - pub origins: Vec, - - #[serde(default)] - pub cache: CacheConfig, - - #[serde(default)] - pub health: HealthConfig, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct OriginConfig { - pub id: String, - pub origin_type: OriginType, - pub priority: u8, - - #[serde(default)] - pub enabled: bool, - - #[serde(flatten)] - pub settings: HashMap, -} - -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] -#[serde(rename_all = "lowercase")] -pub enum OriginType { - Local, - Nfs, - Smb, - S3, - Sftp, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct CacheConfig { - #[serde(default = "default_metadata_cache_mb")] - pub metadata_cache_mb: u64, - - #[serde(default = "default_content_cache_gb")] - pub content_cache_gb: u64, -} - -impl Default for CacheConfig { - fn default() -> Self { - Self { - metadata_cache_mb: 100, - content_cache_gb: 10, - } - } -} - -fn default_metadata_cache_mb() -> u64 { 100 } -fn default_content_cache_gb() -> u64 { 10 } - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct HealthConfig { - #[serde(default = "default_check_interval_secs")] - pub check_interval_secs: u64, - - #[serde(default = "default_timeout_ms")] - pub timeout_ms: u64, - - #[serde(default = "default_unhealthy_threshold")] - pub unhealthy_threshold: u32, - - /// Oracle fix: Per-origin-type thresholds (Local=1, Remote=3) - #[serde(default)] - pub per_origin_thresholds: HashMap, -} - -impl Default for HealthConfig { - fn default() -> Self { - Self { - check_interval_secs: 30, - timeout_ms: 5000, - unhealthy_threshold: 3, - } - } -} - -fn default_check_interval_secs() -> u64 { 30 } -fn default_timeout_ms() -> u64 { 5000 } -fn default_unhealthy_threshold() -> u32 { 3 } - -impl Config { - pub fn from_file(path: &std::path::Path) -> Result { - let content = std::fs::read_to_string(path) - .map_err(|e| ConfigError::Read(e.to_string()))?; - toml::from_str(&content) - .map_err(|e| ConfigError::Parse(e.to_string())) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum ConfigError { - #[error("Failed to read config: {0}")] - Read(String), - - #[error("Failed to parse config: {0}")] - Parse(String), -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_parse_config() { - let toml = r#" -mount_point = "/mnt/music" -cache_dir = "/home/user/.cache/musicfs" - -[[origins]] -id = "local" -origin_type = "local" -priority = 1 -path = "/mnt/nas/music" - -[[origins]] -id = "backup" -origin_type = "s3" -priority = 2 -bucket = "music-backup" -region = "us-east-1" -"#; - - let config: Config = toml::from_str(toml).unwrap(); - assert_eq!(config.origins.len(), 2); - assert_eq!(config.origins[0].priority, 1); - assert_eq!(config.origins[1].origin_type, OriginType::S3); - } -} -``` - ---- - -## Task 2: Origin Registry - -### 2.1 Create `musicfs-origins/src/registry.rs` - -```rust -use crate::traits::{Origin, OriginType}; -use crate::health::{HealthMonitor, HealthSnapshot}; -use crate::router::Router; -use musicfs_core::{OriginId, RealPath}; -use std::collections::HashMap; -use std::sync::{Arc, RwLock}; -use tracing::{debug, info, warn}; - -/// Central registry for all origins -pub struct OriginRegistry { - origins: RwLock>>, - router: Router, - health_monitor: Arc, - /// Oracle fix: Track active watch handles per origin for cleanup - watch_handles: RwLock>>, -} - -impl OriginRegistry { - pub fn new(health_monitor: Arc) -> Self { - Self { - origins: RwLock::new(HashMap::new()), - router: Router::new(), - health_monitor, - } - } - - /// Register a new origin - pub fn register(&self, origin: Arc, priority: u8) { - let id = origin.id().clone(); - info!("Registering origin {} with priority {}", id, priority); - - self.router.set_priority(id.clone(), priority); - self.health_monitor.add_origin(origin.clone()); - self.origins.write().unwrap().insert(id, origin); - } - - /// Unregister an origin - /// Oracle fix: Clean up watch handles when origin is removed - pub fn unregister(&self, id: &OriginId) { - info!("Unregistering origin {}", id); - - // Oracle fix: Drop all watch handles for this origin - if let Some(handles) = self.watch_handles.write().unwrap().remove(id) { - info!("Dropping {} watch handles for origin {}", handles.len(), id); - // Handles are dropped here, which triggers their stop signal - } - - self.origins.write().unwrap().remove(id); - self.router.remove_priority(id); - self.health_monitor.remove_origin(id); - } - - /// Register a watch handle for an origin (for cleanup on unregister) - pub fn register_watch(&self, origin_id: &OriginId, handle: WatchHandle) { - self.watch_handles - .write() - .unwrap() - .entry(origin_id.clone()) - .or_default() - .push(handle); - } - - /// Get origin by ID - pub fn get(&self, id: &OriginId) -> Option> { - self.origins.read().unwrap().get(id).cloned() - } - - /// Get all registered origins - pub fn list(&self) -> Vec> { - self.origins.read().unwrap().values().cloned().collect() - } - - /// Route request to best available origin for a path - pub fn route(&self, path: &RealPath) -> Option> { - let origins = self.origins.read().unwrap(); - let health = self.health_monitor.snapshot(); - - // Get all origins that could serve this path - let candidates: Vec<_> = origins - .iter() - .filter(|(id, _)| self.can_serve(id, path)) - .map(|(id, origin)| (id.clone(), origin.clone())) - .collect(); - - if candidates.is_empty() { - warn!("No origin can serve path: {:?}", path); - return None; - } - - // Select best based on priority and health - let candidate_ids: Vec<_> = candidates.iter().map(|(id, _)| id.clone()).collect(); - let selected = self.router.select(&candidate_ids, &health)?; - - candidates - .into_iter() - .find(|(id, _)| id == &selected) - .map(|(_, origin)| origin) - } - - /// Route to all available origins (for redundancy) - pub fn route_all(&self, path: &RealPath) -> Vec> { - let origins = self.origins.read().unwrap(); - let health = self.health_monitor.snapshot(); - - let mut result: Vec<_> = origins - .iter() - .filter(|(id, _)| self.can_serve(id, path) && health.is_healthy(id)) - .map(|(_, origin)| origin.clone()) - .collect(); - - // Sort by priority - result.sort_by_key(|o| self.router.get_priority(o.id())); - result - } - - /// Check if origin can serve a given path - fn can_serve(&self, _origin_id: &OriginId, path: &RealPath) -> bool { - // For now, origin_id in path must match - // Future: support path mappings - path.origin_id == *_origin_id - } - - /// Get current health snapshot - pub fn health(&self) -> HealthSnapshot { - self.health_monitor.snapshot() - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::LocalOrigin; - use tempfile::TempDir; - use std::path::PathBuf; - - #[test] - fn test_register_and_get() { - let monitor = Arc::new(HealthMonitor::new(std::time::Duration::from_secs(30))); - let registry = OriginRegistry::new(monitor); - - let dir = TempDir::new().unwrap(); - let origin = Arc::new(LocalOrigin::new("test", dir.path())); - - registry.register(origin.clone(), 1); - - let retrieved = registry.get(&OriginId::from("test")); - assert!(retrieved.is_some()); - } - - #[test] - fn test_route_by_priority() { - let monitor = Arc::new(HealthMonitor::new(std::time::Duration::from_secs(30))); - let registry = OriginRegistry::new(monitor); - - let dir1 = TempDir::new().unwrap(); - let dir2 = TempDir::new().unwrap(); - - let origin1 = Arc::new(LocalOrigin::new("primary", dir1.path())); - let origin2 = Arc::new(LocalOrigin::new("backup", dir2.path())); - - registry.register(origin1, 1); // Higher priority - registry.register(origin2, 2); // Lower priority - - let path = RealPath { - origin_id: OriginId::from("primary"), - path: PathBuf::from("/test.flac"), - }; - - let routed = registry.route(&path); - assert!(routed.is_some()); - assert_eq!(routed.unwrap().id(), &OriginId::from("primary")); - } -} -``` - ---- - -## Task 3: Priority Router - -### 3.1 Create `musicfs-origins/src/router.rs` - -```rust -use crate::health::HealthSnapshot; -use dashmap::DashMap; -use musicfs_core::OriginId; -use std::time::Instant; -use tracing::debug; - -/// Routes requests to origins based on priority and health -pub struct Router { - /// Origin priority (lower = higher priority) - priorities: DashMap, - - /// Latency statistics per origin - latency_stats: DashMap, -} - -#[derive(Debug, Clone, Default)] -pub struct LatencyStats { - pub samples: Vec, // Recent latency samples in ms - pub p50_ms: u64, - pub p99_ms: u64, - pub last_update: Option, -} - -impl LatencyStats { - pub fn record(&mut self, latency_ms: u64) { - self.samples.push(latency_ms); - - // Keep last 100 samples - if self.samples.len() > 100 { - self.samples.remove(0); - } - - // Recalculate percentiles - if !self.samples.is_empty() { - let mut sorted = self.samples.clone(); - sorted.sort_unstable(); - - let p50_idx = sorted.len() / 2; - let p99_idx = (sorted.len() * 99) / 100; - - self.p50_ms = sorted[p50_idx]; - self.p99_ms = sorted.get(p99_idx).copied().unwrap_or(self.p50_ms); - } - - self.last_update = Some(Instant::now()); - } -} - -impl Router { - pub fn new() -> Self { - Self { - priorities: DashMap::new(), - latency_stats: DashMap::new(), - } - } - - /// Set priority for an origin - pub fn set_priority(&self, id: OriginId, priority: u8) { - self.priorities.insert(id, priority); - } - - /// Remove priority for an origin - pub fn remove_priority(&self, id: &OriginId) { - self.priorities.remove(id); - self.latency_stats.remove(id); - } - - /// Get priority for an origin - pub fn get_priority(&self, id: &OriginId) -> u8 { - self.priorities.get(id).map(|p| *p).unwrap_or(100) - } - - /// Record latency sample for an origin - pub fn record_latency(&self, id: &OriginId, latency_ms: u64) { - self.latency_stats - .entry(id.clone()) - .or_default() - .record(latency_ms); - } - - /// Select best origin from candidates - /// - /// Oracle fix: Clarified routing - uses tuple ordering (priority, latency) - /// Priority is dominant: priority 1 always beats priority 2 regardless of latency - /// Latency is tiebreaker: among same priority, lower latency wins - pub fn select(&self, candidates: &[OriginId], health: &HealthSnapshot) -> Option { - candidates - .iter() - .filter(|id| health.is_healthy(id)) - .min_by_key(|id| { - let priority = self.get_priority(id); - let latency = self.latency_stats - .get(*id) - .map(|s| s.p50_ms) - .unwrap_or(0); - - // Oracle fix: Use tuple for clear priority-dominant ordering - // (1, 1000ms) < (2, 10ms) - priority 1 always wins - (priority, latency) - }) - .cloned() - } - - /// Select with fallback to unhealthy if no healthy available - /// - /// Oracle fix: Define behavior when all origins unhealthy: - /// 1. Try healthy origins first - /// 2. Fall back to degraded origins - /// 3. If all unhealthy, select "least-bad" (fewest consecutive failures) - /// 4. Emit AllOriginsUnhealthy event for monitoring - pub fn select_with_fallback( - &self, - candidates: &[OriginId], - health: &HealthSnapshot, - ) -> Option { - // Try healthy first - if let Some(id) = self.select(candidates, health) { - return Some(id); - } - - // Fall back to degraded - debug!("No healthy origins, trying degraded"); - if let Some(id) = candidates - .iter() - .filter(|id| health.is_degraded(id)) - .min_by_key(|id| self.get_priority(id)) - .cloned() - { - return Some(id); - } - - // Oracle fix: All origins unhealthy - select least-bad - warn!("All origins unhealthy, selecting least-bad by failure count"); - candidates - .iter() - .min_by_key(|id| { - let failures = health.failure_count(id).unwrap_or(u32::MAX); - let priority = self.get_priority(id); - (failures, priority) - }) - .cloned() - } -} - -impl Default for Router { - fn default() -> Self { - Self::new() - } -} - -#[cfg(test)] -mod tests { - use super::*; - - fn mock_health(healthy: &[&str], degraded: &[&str]) -> HealthSnapshot { - HealthSnapshot { - healthy: healthy.iter().map(|s| OriginId::from(*s)).collect(), - degraded: degraded.iter().map(|s| OriginId::from(*s)).collect(), - unhealthy: Vec::new(), - } - } - - #[test] - fn test_select_by_priority() { - let router = Router::new(); - router.set_priority(OriginId::from("high"), 1); - router.set_priority(OriginId::from("low"), 2); - - let candidates = vec![ - OriginId::from("low"), - OriginId::from("high"), - ]; - let health = mock_health(&["high", "low"], &[]); - - let selected = router.select(&candidates, &health); - assert_eq!(selected, Some(OriginId::from("high"))); - } - - #[test] - fn test_select_skips_unhealthy() { - let router = Router::new(); - router.set_priority(OriginId::from("high"), 1); - router.set_priority(OriginId::from("low"), 2); - - let candidates = vec![ - OriginId::from("high"), - OriginId::from("low"), - ]; - // "high" is unhealthy - let health = mock_health(&["low"], &[]); - - let selected = router.select(&candidates, &health); - assert_eq!(selected, Some(OriginId::from("low"))); - } - - #[test] - fn test_latency_affects_tiebreak() { - let router = Router::new(); - router.set_priority(OriginId::from("a"), 1); - router.set_priority(OriginId::from("b"), 1); // Same priority - - router.record_latency(&OriginId::from("a"), 100); - router.record_latency(&OriginId::from("b"), 10); // Lower latency - - let candidates = vec![ - OriginId::from("a"), - OriginId::from("b"), - ]; - let health = mock_health(&["a", "b"], &[]); - - let selected = router.select(&candidates, &health); - assert_eq!(selected, Some(OriginId::from("b"))); // Lower latency wins - } -} -``` - ---- - -## Task 4: Health Monitor - -### 4.1 Create `musicfs-origins/src/health.rs` - -```rust -use crate::traits::Origin; -use dashmap::DashMap; -use musicfs_core::{HealthStatus, OriginId}; -use std::sync::Arc; -use std::time::{Duration, Instant}; -use tokio::sync::mpsc; -use tracing::{debug, info, warn}; - -/// Monitors health of all origins -pub struct HealthMonitor { - origins: DashMap>, - state: DashMap, - check_interval: Duration, - stop_tx: Option>, -} - -#[derive(Debug, Clone)] -pub struct OriginHealthState { - pub status: HealthStatus, - pub last_check: Instant, - pub consecutive_failures: u32, - pub last_latency_ms: Option, -} - -impl Default for OriginHealthState { - fn default() -> Self { - Self { - status: HealthStatus::Unknown, - last_check: Instant::now(), - consecutive_failures: 0, - last_latency_ms: None, - } - } -} - -/// Snapshot of health state for routing decisions -#[derive(Debug, Clone)] -pub struct HealthSnapshot { - pub healthy: Vec, - pub degraded: Vec, - pub unhealthy: Vec, - /// Oracle fix: Track failure counts for least-bad selection - pub failure_counts: HashMap, -} - -impl HealthSnapshot { - pub fn is_healthy(&self, id: &OriginId) -> bool { - self.healthy.contains(id) - } - - pub fn is_degraded(&self, id: &OriginId) -> bool { - self.degraded.contains(id) - } - - pub fn is_unhealthy(&self, id: &OriginId) -> bool { - self.unhealthy.contains(id) - } - - /// Oracle fix: Get failure count for least-bad selection - pub fn failure_count(&self, id: &OriginId) -> Option { - self.failure_counts.get(id).copied() - } - - /// Oracle fix: Check if all origins are unhealthy - pub fn all_unhealthy(&self) -> bool { - self.healthy.is_empty() && self.degraded.is_empty() - } -} - -impl HealthMonitor { - pub fn new(check_interval: Duration) -> Self { - Self { - origins: DashMap::new(), - state: DashMap::new(), - check_interval, - stop_tx: None, - } - } - - /// Add origin to monitoring - pub fn add_origin(&self, origin: Arc) { - let id = origin.id().clone(); - self.origins.insert(id.clone(), origin); - self.state.insert(id, OriginHealthState::default()); - } - - /// Remove origin from monitoring - pub fn remove_origin(&self, id: &OriginId) { - self.origins.remove(id); - self.state.remove(id); - } - - /// Get current health snapshot - pub fn snapshot(&self) -> HealthSnapshot { - let mut healthy = Vec::new(); - let mut degraded = Vec::new(); - let mut unhealthy = Vec::new(); - - for entry in self.state.iter() { - let id = entry.key().clone(); - match entry.value().status { - HealthStatus::Healthy => healthy.push(id), - HealthStatus::Degraded => degraded.push(id), - HealthStatus::Unhealthy => unhealthy.push(id), - HealthStatus::Unknown => degraded.push(id), // Treat unknown as degraded - } - } - - HealthSnapshot { healthy, degraded, unhealthy } - } - - /// Start background health check loop - pub fn start(self: Arc) -> HealthCheckHandle { - let (stop_tx, mut stop_rx) = mpsc::channel::<()>(1); - let monitor = self.clone(); - - tokio::spawn(async move { - let mut interval = tokio::time::interval(monitor.check_interval); - - loop { - tokio::select! { - _ = interval.tick() => { - monitor.check_all().await; - } - _ = stop_rx.recv() => { - info!("Health monitor stopping"); - break; - } - } - } - }); - - HealthCheckHandle { stop_tx } - } - - /// Check health of all origins - async fn check_all(&self) { - let origins: Vec<_> = self.origins.iter() - .map(|e| (e.key().clone(), e.value().clone())) - .collect(); - - for (id, origin) in origins { - self.check_one(&id, &origin).await; - } - } - - /// Check health of one origin - async fn check_one(&self, id: &OriginId, origin: &Arc) { - let start = Instant::now(); - let status = origin.health().await; - let latency_ms = start.elapsed().as_millis() as u64; - - let mut state = self.state.entry(id.clone()).or_default(); - - match status { - HealthStatus::Healthy => { - if state.status != HealthStatus::Healthy { - info!("Origin {} is now healthy", id); - } - state.status = HealthStatus::Healthy; - state.consecutive_failures = 0; - } - HealthStatus::Degraded => { - if state.status != HealthStatus::Degraded { - warn!("Origin {} is degraded", id); - } - state.status = HealthStatus::Degraded; - } - HealthStatus::Unhealthy => { - state.consecutive_failures += 1; - if state.consecutive_failures >= 3 { - if state.status != HealthStatus::Unhealthy { - warn!("Origin {} is now unhealthy ({} failures)", id, state.consecutive_failures); - } - state.status = HealthStatus::Unhealthy; - } else { - debug!("Origin {} check failed ({}/3)", id, state.consecutive_failures); - state.status = HealthStatus::Degraded; - } - } - HealthStatus::Unknown => { - state.status = HealthStatus::Unknown; - } - } - - state.last_check = Instant::now(); - state.last_latency_ms = Some(latency_ms); - } - - /// Force immediate health check - pub async fn check_now(&self, id: &OriginId) { - if let Some(origin) = self.origins.get(id) { - self.check_one(id, &origin.clone()).await; - } - } -} - -pub struct HealthCheckHandle { - stop_tx: mpsc::Sender<()>, -} - -impl HealthCheckHandle { - pub async fn stop(self) { - let _ = self.stop_tx.send(()).await; - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::LocalOrigin; - use tempfile::TempDir; - - #[tokio::test] - async fn test_health_monitor_basic() { - let monitor = HealthMonitor::new(Duration::from_secs(30)); - - let dir = TempDir::new().unwrap(); - let origin = Arc::new(LocalOrigin::new("test", dir.path())); - - monitor.add_origin(origin); - - let snapshot = monitor.snapshot(); - // Initially unknown (treated as degraded) - assert!(!snapshot.is_healthy(&OriginId::from("test"))); - } - - #[tokio::test] - async fn test_health_check() { - let monitor = Arc::new(HealthMonitor::new(Duration::from_secs(30))); - - let dir = TempDir::new().unwrap(); - let origin = Arc::new(LocalOrigin::new("test", dir.path())); - - monitor.add_origin(origin); - monitor.check_now(&OriginId::from("test")).await; - - let snapshot = monitor.snapshot(); - assert!(snapshot.is_healthy(&OriginId::from("test"))); - } -} -``` - ---- - -## Task 5: Failover Logic - -### 5.1 Create `musicfs-origins/src/failover.rs` - -```rust -use crate::registry::OriginRegistry; -use musicfs_core::{OriginId, RealPath, Result}; -use std::sync::Arc; -use std::time::Duration; -use tracing::{debug, warn}; - -/// Retry configuration -#[derive(Debug, Clone)] -pub struct RetryConfig { - pub max_attempts: u32, - pub initial_delay: Duration, - pub max_delay: Duration, - pub backoff_factor: f64, -} - -impl Default for RetryConfig { - fn default() -> Self { - // Oracle fix: Align with NFR-7.3 spec: 100ms, 500ms, 2000ms - // Use fixed delays instead of exponential to match spec exactly - Self { - max_attempts: 3, - initial_delay: Duration::from_millis(100), - max_delay: Duration::from_secs(2), - backoff_factor: 5.0, // 100ms * 5 = 500ms, 500ms * 4 = 2000ms - } - } -} - -impl RetryConfig { - /// Oracle fix: Create config that matches NFR-7.3 exactly - pub fn spec_compliant() -> Self { - Self { - max_attempts: 3, - initial_delay: Duration::from_millis(100), - max_delay: Duration::from_secs(2), - backoff_factor: 5.0, // Produces 100ms, 500ms, 2000ms sequence - } - } -} - -/// Execute operation with failover across origins -pub struct FailoverExecutor { - registry: Arc, - retry_config: RetryConfig, -} - -impl FailoverExecutor { - pub fn new(registry: Arc, retry_config: RetryConfig) -> Self { - Self { registry, retry_config } - } - - /// Execute read with automatic failover - pub async fn read_with_failover( - &self, - path: &RealPath, - offset: u64, - size: u32, - ) -> Result> { - let origins = self.registry.route_all(path); - - if origins.is_empty() { - return Err(musicfs_core::Error::NoOriginAvailable); - } - - let mut last_error = None; - - for origin in origins { - match self.read_with_retry(&origin, &path.path, offset, size).await { - Ok(data) => return Ok(data), - Err(e) => { - warn!("Origin {} failed: {}, trying next", origin.id(), e); - last_error = Some(e); - } - } - } - - Err(last_error.unwrap_or(musicfs_core::Error::NoOriginAvailable)) - } - - /// Read with exponential backoff retry - async fn read_with_retry( - &self, - origin: &Arc, - path: &std::path::Path, - offset: u64, - size: u32, - ) -> Result> { - let mut delay = self.retry_config.initial_delay; - - for attempt in 0..self.retry_config.max_attempts { - match origin.read(path, offset, size).await { - Ok(data) => return Ok(data), - Err(e) if attempt + 1 < self.retry_config.max_attempts => { - debug!( - "Retry {}/{} for {} after {:?}: {}", - attempt + 1, - self.retry_config.max_attempts, - origin.id(), - delay, - e - ); - tokio::time::sleep(delay).await; - - // Exponential backoff - delay = std::cmp::min( - Duration::from_secs_f64(delay.as_secs_f64() * self.retry_config.backoff_factor), - self.retry_config.max_delay, - ); - } - Err(e) => return Err(e), - } - } - - Err(musicfs_core::Error::MaxRetriesExceeded) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_retry_config_default() { - let config = RetryConfig::default(); - assert_eq!(config.max_attempts, 3); - assert_eq!(config.initial_delay, Duration::from_millis(100)); - } - - #[test] - fn test_backoff_calculation() { - let config = RetryConfig::default(); - let mut delay = config.initial_delay; - - // First retry: 100ms - assert_eq!(delay, Duration::from_millis(100)); - - // Second retry: 200ms - delay = Duration::from_secs_f64(delay.as_secs_f64() * config.backoff_factor); - assert_eq!(delay, Duration::from_millis(200)); - - // Third retry: 400ms - delay = Duration::from_secs_f64(delay.as_secs_f64() * config.backoff_factor); - assert_eq!(delay, Duration::from_millis(400)); - } -} -``` - ---- - -## Task 6: Update lib.rs - -### 6.1 Update `musicfs-origins/src/lib.rs` - -```rust -mod failover; -mod health; -mod local; -mod registry; -mod router; -mod traits; - -pub use failover::{FailoverExecutor, RetryConfig}; -pub use health::{HealthCheckHandle, HealthMonitor, HealthSnapshot, OriginHealthState}; -pub use local::LocalOrigin; -pub use registry::OriginRegistry; -pub use router::{LatencyStats, Router}; -pub use traits::{Origin, OriginType, WatchCallback, WatchHandle}; -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_register_and_get` | Unit | Origin registration (FR-13.1) | -| `test_route_by_priority` | Unit | Priority routing (FR-13.3) | -| `test_select_skips_unhealthy` | Unit | Health-aware routing | -| `test_latency_affects_tiebreak` | Unit | Latency-based selection | -| `test_health_check` | Integration | Health monitoring | -| `test_failover_to_backup` | Integration | Automatic failover | -| `test_retry_with_backoff` | Unit | Exponential backoff (NFR-7.3) | -| `test_serve_cached_offline` | Integration | Offline mode (NFR-7.1) | -| `test_all_origins_unhealthy` | Unit | Oracle fix: least-bad selection | -| `test_watch_cleanup_on_unregister` | Unit | Oracle fix: handles dropped | -| `test_per_origin_health_threshold` | Unit | Oracle fix: Local=1, Remote=3 | -| `test_retry_delays_match_spec` | Unit | Oracle fix: 100ms, 500ms, 2000ms | - ---- - -## Exit Criteria - -- [ ] Multiple origins can be registered simultaneously -- [ ] Requests route to highest priority healthy origin -- [ ] Automatic failover when primary origin fails -- [ ] Health checks run every 30s per origin -- [ ] Retries use spec-compliant backoff (100ms, 500ms, 2000ms) - Oracle fix -- [ ] Cached data served when all origins offline -- [ ] All origins unhealthy: select least-bad, emit event - Oracle fix -- [ ] Watch handles cleaned up on origin unregister - Oracle fix -- [ ] Per-origin-type health thresholds (Local=1, Remote=3) - Oracle fix -- [ ] All existing tests pass - ---- - -## Dependencies - -### `musicfs-origins/Cargo.toml` additions - -```toml -[dependencies] -dashmap = "5" -# ... existing deps -``` - ---- - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.3.3 | Priority-based routing | ✅ | -| 4.3.3 | Health tracking | ✅ | -| 4.3.3 | Background health checks every 30s | ✅ | -| 4.3.3 | Automatic failover | ✅ | -| NFR-7.1 | Serve cached when offline | ✅ | -| NFR-7.3 | Retry with exponential backoff | ✅ | diff --git a/docs/v2/plans/week-07-remote-origins.md b/docs/v2/plans/week-07-remote-origins.md deleted file mode 100644 index 8c91457..0000000 --- a/docs/v2/plans/week-07-remote-origins.md +++ /dev/null @@ -1,1261 +0,0 @@ -# Week 7: Remote Origins - -**Phase**: 2 (Delta Sync & Multi-Origin) -**Prerequisites**: Week 6 (Origin Federation) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement remote origin plugins for NFS, SMB, S3, and SFTP, enabling federated music libraries across local and cloud storage. - ---- - -## Oracle Review Fixes (MUST IMPLEMENT) - -| Severity | Issue | Fix | -|----------|-------|-----| -| 🔴 Critical | **SFTP single mutex** - `Arc>` kills concurrency | Use connection pool (`deadpool` or `bb8`) with configurable pool size | -| 🔴 Critical | **SFTP `open_read` OOM** - reads entire file (`u32::MAX` bytes) | Implement chunked streaming or cap at file size | -| 🔴 Critical | **SSH host verification disabled** - MITM vulnerability | Verify against `~/.ssh/known_hosts` file | -| 🔴 Critical | **No timeout handling** - hung connections block forever | Wrap all remote calls with `tokio::time::timeout(30s)` | -| 🔴 Critical | **Credential Debug leaks** - `#[derive(Debug)]` exposes passwords | Custom `Debug` impl that redacts secrets | -| 🔴 Critical | **S3 range EOF** - 416 error if range exceeds file size | Clamp range to `min(requested_end, file_size)` | -| 🔴 Critical | **NFS retry closure** - `FnMut` across async boundary | Change to `Fn` or ensure stateless operation | -| 🟡 Medium | **S3 health too heavy** - `list_objects_v2` | Use `head_bucket` instead | -| 🟡 Medium | **SMB stale mounts** - no handling for ENOTCONN | Add SMB-specific reconnection error handling | -| ⚠️ Watch | **inotify unreliable over NFS/SMB** | Document limitation, default to polling for remote mounts | - ---- - -## Architecture Reference - -From architecture.md section 4.3.4 (Plugin System): - -```plantuml -interface "OriginPlugin" { - +list_dir(path): Vec - +read(path, offset, size): Vec - +stat(path): FileStat - +watch(path, callback): WatchHandle -} - -class "LocalFSPlugin" implements OriginPlugin -class "S3Plugin" implements OriginPlugin -``` - ---- - -## Requirements Covered - -| ID | Requirement | Priority | -|----|-------------|----------| -| FR-12.2 | Support NFS mounted filesystems | P1 | -| FR-12.3 | Support SMB/CIFS shares | P1 | -| FR-12.4 | Support S3-compatible object storage | P1 | -| FR-12.5 | Support SFTP servers | P1 | -| FR-12.6 | Provide pluggable origin interface | P0 | -| NFR-6.2 | Connection pooling for remote origins | P1 | -| NFR-13.3 | Credential storage for remote origins | P1 | - ---- - -## Deliverables - -| Task | Crate | Files | Est. | -|------|-------|-------|------| -| NFS origin | musicfs-origins | `nfs.rs` | 0.5d | -| SMB origin | musicfs-origins | `smb.rs` | 1d | -| S3 origin | musicfs-origins | `s3.rs` | 1.5d | -| SFTP origin | musicfs-origins | `sftp.rs` | 1d | -| Credential handling | musicfs-core | `credentials.rs` | 0.5d | -| Integration tests | tests | `remote_origins.rs` | 0.5d | - ---- - -## Task 1: Credential Handling - -### 1.1 Create `musicfs-core/src/credentials.rs` - -```rust -use serde::{Deserialize, Serialize}; -use std::collections::HashMap; -use std::path::PathBuf; -use thiserror::Error; - -/// Credential store for remote origins -/// -/// Security: Credentials are loaded from environment, keyring, or file. -/// They are NEVER logged or exposed in process list. -/// -/// Oracle fix: Custom Debug to redact secrets -#[derive(Clone)] -pub struct CredentialStore { - cache: HashMap, -} - -impl std::fmt::Debug for CredentialStore { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("CredentialStore") - .field("cache_keys", &self.cache.keys().collect::>()) - .finish() - } -} - -/// Oracle fix: Custom Debug that redacts sensitive fields -#[derive(Clone, Serialize, Deserialize)] -#[serde(tag = "type")] -pub enum Credential { - /// Username/password authentication - Basic { - username: String, - #[serde(skip_serializing)] // Never serialize password - password: String, - }, - - /// AWS-style access key - AwsKey { - access_key_id: String, - #[serde(skip_serializing)] - secret_access_key: String, - session_token: Option, - region: String, - }, - - /// SSH key authentication - SshKey { - username: String, - private_key_path: PathBuf, - passphrase: Option, - }, - - /// Environment variable reference - EnvVar { - var_name: String, - }, -} - -/// Oracle fix: Custom Debug implementation that redacts secrets -impl std::fmt::Debug for Credential { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - match self { - Self::Basic { username, .. } => { - f.debug_struct("Basic") - .field("username", username) - .field("password", &"[REDACTED]") - .finish() - } - Self::AwsKey { access_key_id, region, .. } => { - f.debug_struct("AwsKey") - .field("access_key_id", &format!("{}...", &access_key_id[..4.min(access_key_id.len())])) - .field("secret_access_key", &"[REDACTED]") - .field("region", region) - .finish() - } - Self::SshKey { username, private_key_path, .. } => { - f.debug_struct("SshKey") - .field("username", username) - .field("private_key_path", private_key_path) - .field("passphrase", &"[REDACTED]") - .finish() - } - Self::EnvVar { var_name } => { - f.debug_struct("EnvVar") - .field("var_name", var_name) - .finish() - } - } - } -} - -impl CredentialStore { - pub fn new() -> Self { - Self { - cache: HashMap::new(), - } - } - - /// Load credential for an origin - pub fn load(&mut self, origin_id: &str, config: &CredentialConfig) -> Result { - // Check cache first - if let Some(cred) = self.cache.get(origin_id) { - return Ok(cred.clone()); - } - - let cred = match config { - CredentialConfig::Environment { prefix } => { - self.load_from_env(prefix)? - } - CredentialConfig::File { path } => { - self.load_from_file(path)? - } - CredentialConfig::Keyring { service } => { - self.load_from_keyring(service)? - } - CredentialConfig::Inline(cred) => { - cred.clone() - } - }; - - self.cache.insert(origin_id.to_string(), cred.clone()); - Ok(cred) - } - - fn load_from_env(&self, prefix: &str) -> Result { - // Try AWS-style first - if let (Ok(key), Ok(secret)) = ( - std::env::var(format!("{}_ACCESS_KEY_ID", prefix)), - std::env::var(format!("{}_SECRET_ACCESS_KEY", prefix)), - ) { - return Ok(Credential::AwsKey { - access_key_id: key, - secret_access_key: secret, - session_token: std::env::var(format!("{}_SESSION_TOKEN", prefix)).ok(), - region: std::env::var(format!("{}_REGION", prefix)) - .unwrap_or_else(|_| "us-east-1".to_string()), - }); - } - - // Try basic auth - if let (Ok(user), Ok(pass)) = ( - std::env::var(format!("{}_USERNAME", prefix)), - std::env::var(format!("{}_PASSWORD", prefix)), - ) { - return Ok(Credential::Basic { - username: user, - password: pass, - }); - } - - Err(CredentialError::NotFound(format!("No credentials found with prefix {}", prefix))) - } - - fn load_from_file(&self, path: &PathBuf) -> Result { - let content = std::fs::read_to_string(path) - .map_err(|e| CredentialError::FileRead(e.to_string()))?; - - // Support JSON or TOML - if path.extension().map(|e| e == "json").unwrap_or(false) { - serde_json::from_str(&content) - .map_err(|e| CredentialError::Parse(e.to_string())) - } else { - toml::from_str(&content) - .map_err(|e| CredentialError::Parse(e.to_string())) - } - } - - fn load_from_keyring(&self, service: &str) -> Result { - // Use secret-service on Linux, Keychain on macOS - #[cfg(any(target_os = "linux", target_os = "macos"))] - { - let entry = keyring::Entry::new(service, "musicfs") - .map_err(|e| CredentialError::Keyring(e.to_string()))?; - - let secret = entry.get_password() - .map_err(|e| CredentialError::Keyring(e.to_string()))?; - - // Assume JSON-encoded credential - serde_json::from_str(&secret) - .map_err(|e| CredentialError::Parse(e.to_string())) - } - - #[cfg(not(any(target_os = "linux", target_os = "macos")))] - { - Err(CredentialError::NotSupported("Keyring not supported on this platform".into())) - } - } -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "source")] -pub enum CredentialConfig { - Environment { prefix: String }, - File { path: PathBuf }, - Keyring { service: String }, - Inline(Credential), -} - -#[derive(Debug, Error)] -pub enum CredentialError { - #[error("Credential not found: {0}")] - NotFound(String), - - #[error("Failed to read credential file: {0}")] - FileRead(String), - - #[error("Failed to parse credential: {0}")] - Parse(String), - - #[error("Keyring error: {0}")] - Keyring(String), - - #[error("Not supported: {0}")] - NotSupported(String), -} - -impl Default for CredentialStore { - fn default() -> Self { - Self::new() - } -} -``` - ---- - -## Task 2: NFS Origin - -### 2.1 Create `musicfs-origins/src/nfs.rs` - -NFS mounts are treated as local filesystems. The key difference is handling NFS-specific errors like stale file handles. - -```rust -use crate::local::LocalOrigin; -use crate::traits::{Origin, OriginType, WatchCallback, WatchHandle}; -use async_trait::async_trait; -use musicfs_core::{DirEntry, FileStat, HealthStatus, OriginId, Result}; -use std::path::{Path, PathBuf}; -use std::time::Duration; -use tokio::time::sleep; -use tracing::{debug, warn}; - -/// NFS origin - wraps local filesystem with NFS-specific error handling -pub struct NfsOrigin { - inner: LocalOrigin, - max_retries: u32, -} - -impl NfsOrigin { - pub fn new(id: impl Into, mount_point: impl Into) -> Self { - let mount_point = mount_point.into(); - let display = format!("NFS: {}", mount_point.display()); - - Self { - inner: LocalOrigin::new(id, mount_point), - max_retries: 3, - } - } - - /// Retry operation on ESTALE (stale NFS handle) - /// - /// Oracle fix: Changed from FnMut to Fn to avoid issues across async boundary - async fn retry_on_stale(&self, op: F) -> Result - where - F: Fn() -> Fut, - Fut: std::future::Future>, - { - let mut delay = Duration::from_millis(100); - - for attempt in 0..self.max_retries { - match op().await { - Ok(result) => return Ok(result), - Err(e) => { - // Check for ESTALE - if let Some(io_err) = e.downcast_io() { - if io_err.raw_os_error() == Some(libc::ESTALE) { - warn!( - "NFS stale handle (attempt {}/{}), retrying after {:?}", - attempt + 1, self.max_retries, delay - ); - sleep(delay).await; - delay *= 2; // Exponential backoff - continue; - } - } - return Err(e); - } - } - } - - Err(musicfs_core::Error::NfsStaleHandle) - } -} - -#[async_trait] -impl Origin for NfsOrigin { - fn id(&self) -> &OriginId { - self.inner.id() - } - - fn origin_type(&self) -> OriginType { - OriginType::Nfs - } - - fn display_name(&self) -> &str { - self.inner.display_name() - } - - async fn readdir(&self, path: &Path) -> Result> { - self.retry_on_stale(|| self.inner.readdir(path)).await - } - - async fn stat(&self, path: &Path) -> Result { - self.retry_on_stale(|| self.inner.stat(path)).await - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result> { - self.retry_on_stale(|| self.inner.read(path, offset, size)).await - } - - async fn exists(&self, path: &Path) -> Result { - self.retry_on_stale(|| self.inner.exists(path)).await - } - - async fn health(&self) -> HealthStatus { - // For NFS, check if mount is responsive - match self.inner.stat(Path::new("/")).await { - Ok(_) => HealthStatus::Healthy, - Err(_) => HealthStatus::Unhealthy, - } - } - - async fn open_read(&self, path: &Path) -> Result> { - self.inner.open_read(path).await - } - - async fn watch(&self, path: &Path, callback: WatchCallback) -> Result { - // inotify works over NFS (with limitations) - self.inner.watch(path, callback).await - } -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - #[tokio::test] - async fn test_nfs_origin_basic() { - let dir = TempDir::new().unwrap(); - std::fs::write(dir.path().join("test.flac"), b"audio").unwrap(); - - let origin = NfsOrigin::new("nfs-test", dir.path()); - - let entries = origin.readdir(Path::new("/")).await.unwrap(); - assert_eq!(entries.len(), 1); - - let data = origin.read(Path::new("/test.flac"), 0, 5).await.unwrap(); - assert_eq!(&data, b"audio"); - } -} -``` - ---- - -## Task 3: S3 Origin - -### 3.1 Create `musicfs-origins/src/s3.rs` - -```rust -use crate::traits::{Origin, OriginType, WatchCallback, WatchHandle}; -use async_trait::async_trait; -use aws_sdk_s3::Client; -use aws_sdk_s3::primitives::ByteStream; -use musicfs_core::{DirEntry, FileStat, HealthStatus, OriginId, Result}; -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::SystemTime; -use tracing::{debug, info}; - -/// S3-compatible object storage origin -pub struct S3Origin { - id: OriginId, - client: Client, - bucket: String, - prefix: String, - display_name: String, -} - -impl S3Origin { - pub async fn new( - id: impl Into, - bucket: impl Into, - prefix: impl Into, - config: aws_config::SdkConfig, - ) -> Self { - let id = id.into(); - let bucket = bucket.into(); - let prefix = prefix.into(); - - Self { - display_name: format!("S3: s3://{}/{}", bucket, prefix), - client: Client::new(&config), - bucket, - prefix, - id, - } - } - - /// Build S3 key from path - fn key(&self, path: &Path) -> String { - let path_str = path.to_string_lossy(); - let path_str = path_str.trim_start_matches('/'); - - if self.prefix.is_empty() { - path_str.to_string() - } else { - format!("{}/{}", self.prefix.trim_end_matches('/'), path_str) - } - } - - /// Parse S3 key to extract filename - fn key_to_name(&self, key: &str) -> String { - key.rsplit('/').next().unwrap_or(key).to_string() - } -} - -#[async_trait] -impl Origin for S3Origin { - fn id(&self) -> &OriginId { - &self.id - } - - fn origin_type(&self) -> OriginType { - OriginType::S3 - } - - fn display_name(&self) -> &str { - &self.display_name - } - - async fn readdir(&self, path: &Path) -> Result> { - let prefix = self.key(path); - let prefix = if prefix.is_empty() || prefix.ends_with('/') { - prefix - } else { - format!("{}/", prefix) - }; - - debug!("S3 list objects: bucket={}, prefix={}", self.bucket, prefix); - - let mut entries = Vec::new(); - let mut continuation_token = None; - - loop { - let mut req = self.client - .list_objects_v2() - .bucket(&self.bucket) - .prefix(&prefix) - .delimiter("/"); - - if let Some(token) = continuation_token.take() { - req = req.continuation_token(token); - } - - let resp = req.send().await - .map_err(|e| musicfs_core::Error::S3(e.to_string()))?; - - // Add "directories" (common prefixes) - if let Some(prefixes) = resp.common_prefixes() { - for cp in prefixes { - if let Some(p) = cp.prefix() { - let name = p.trim_end_matches('/') - .rsplit('/') - .next() - .unwrap_or(p); - - entries.push(DirEntry { - name: name.to_string(), - is_dir: true, - size: 0, - mtime: SystemTime::UNIX_EPOCH, - }); - } - } - } - - // Add files - if let Some(contents) = resp.contents() { - for obj in contents { - if let Some(key) = obj.key() { - // Skip the directory marker itself - if key == prefix { - continue; - } - - let name = self.key_to_name(key); - let size = obj.size().unwrap_or(0) as u64; - let mtime = obj.last_modified() - .and_then(|dt| SystemTime::try_from(*dt).ok()) - .unwrap_or(SystemTime::UNIX_EPOCH); - - entries.push(DirEntry { - name, - is_dir: false, - size, - mtime, - }); - } - } - } - - // Check for more results - if resp.is_truncated() == Some(true) { - continuation_token = resp.next_continuation_token().map(|s| s.to_string()); - } else { - break; - } - } - - Ok(entries) - } - - async fn stat(&self, path: &Path) -> Result { - let key = self.key(path); - - debug!("S3 head object: bucket={}, key={}", self.bucket, key); - - let resp = self.client - .head_object() - .bucket(&self.bucket) - .key(&key) - .send() - .await - .map_err(|e| musicfs_core::Error::S3(e.to_string()))?; - - let size = resp.content_length().unwrap_or(0) as u64; - let mtime = resp.last_modified() - .and_then(|dt| SystemTime::try_from(*dt).ok()) - .unwrap_or(SystemTime::UNIX_EPOCH); - - Ok(FileStat { - size, - mtime, - is_dir: false, - }) - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result> { - let key = self.key(path); - - // Oracle fix: Clamp range to file size to avoid 416 error - let file_size = self.stat(path).await?.size; - let end = std::cmp::min(offset + size as u64, file_size).saturating_sub(1); - - if offset >= file_size { - return Ok(Vec::new()); // EOF - } - - let range = format!("bytes={}-{}", offset, end); - - debug!("S3 get object: bucket={}, key={}, range={}", self.bucket, key, range); - - // Oracle fix: Add timeout to prevent hung connections - let resp = tokio::time::timeout( - Duration::from_secs(30), - self.client - .get_object() - .bucket(&self.bucket) - .key(&key) - .range(range) - .send() - ) - .await - .map_err(|_| musicfs_core::Error::Timeout("S3 read timed out".into()))? - .map_err(|e| musicfs_core::Error::S3(e.to_string()))?; - - let body = resp.body.collect().await - .map_err(|e| musicfs_core::Error::S3(e.to_string()))?; - - Ok(body.into_bytes().to_vec()) - } - - async fn exists(&self, path: &Path) -> Result { - match self.stat(path).await { - Ok(_) => Ok(true), - Err(e) if e.is_not_found() => Ok(false), - Err(e) => Err(e), - } - } - - async fn health(&self) -> HealthStatus { - // Oracle fix: Use head_bucket instead of list_objects_v2 (lighter) - match self.client - .head_bucket() - .bucket(&self.bucket) - .send() - .await - { - Ok(_) => HealthStatus::Healthy, - Err(_) => HealthStatus::Unhealthy, - } - } - - async fn open_read(&self, path: &Path) -> Result> { - // For streaming, return a ByteStream wrapper - let key = self.key(path); - - let resp = self.client - .get_object() - .bucket(&self.bucket) - .key(&key) - .send() - .await - .map_err(|e| musicfs_core::Error::S3(e.to_string()))?; - - Ok(Box::new(resp.body.into_async_read())) - } - - async fn watch(&self, _path: &Path, _callback: WatchCallback) -> Result { - // S3 doesn't support real-time watching - // Return a no-op handle; use polling instead - debug!("S3 watch not supported, use polling"); - let (tx, _rx) = tokio::sync::oneshot::channel(); - Ok(WatchHandle::new(tx)) - } -} - -#[cfg(test)] -mod tests { - // S3 tests require real credentials or localstack - // See tests/integration/s3_origin.rs -} -``` - ---- - -## Task 4: SFTP Origin - -### 4.1 Create `musicfs-origins/src/sftp.rs` - -```rust -use crate::traits::{Origin, OriginType, WatchCallback, WatchHandle}; -use async_trait::async_trait; -use musicfs_core::{DirEntry, FileStat, HealthStatus, OriginId, Result}; -use russh_sftp::client::SftpSession; -use std::path::{Path, PathBuf}; -use std::sync::Arc; -use std::time::SystemTime; -use tokio::sync::Mutex; -use tracing::{debug, info}; - -/// SFTP origin for remote file access -/// -/// Oracle fix: Use connection pool instead of single mutex -pub struct SftpOrigin { - id: OriginId, - display_name: String, - /// Oracle fix: Connection pool instead of Arc> - pool: deadpool::managed::Pool, - base_path: PathBuf, - /// Oracle fix: Timeout for all operations - timeout: Duration, -} - -/// Connection pool manager for SFTP sessions -struct SftpManager { - host: String, - port: u16, - username: String, - auth: SftpAuth, -} - -impl deadpool::managed::Manager for SftpManager { - type Type = SftpSession; - type Error = musicfs_core::Error; - - async fn create(&self) -> Result { - // Connect and authenticate (see connect() implementation) - todo!("Implement pooled connection creation") - } - - async fn recycle(&self, _conn: &mut Self::Type, _metrics: &deadpool::managed::Metrics) -> deadpool::managed::RecycleResult { - // Check if connection is still alive - Ok(()) - } -} - -impl SftpOrigin { - pub async fn connect( - id: impl Into, - host: &str, - port: u16, - username: &str, - auth: SftpAuth, - base_path: impl Into, - ) -> Result { - let id = id.into(); - let base_path = base_path.into(); - - info!("Connecting to SFTP {}@{}:{}", username, host, port); - - // Connect using russh - let config = Arc::new(russh::client::Config::default()); - let mut session = russh::client::connect(config, (host, port), SftpHandler) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - // Authenticate - match auth { - SftpAuth::Password(password) => { - session.authenticate_password(username, &password) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - } - SftpAuth::Key { path, passphrase } => { - let key = russh_keys::load_secret_key(&path, passphrase.as_deref()) - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - session.authenticate_publickey(username, Arc::new(key)) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - } - } - - // Start SFTP subsystem - let channel = session.channel_open_session() - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - channel.request_subsystem(true, "sftp") - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - let sftp = SftpSession::new(channel.into_stream()) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - Ok(Self { - display_name: format!("SFTP: {}@{}:{}{}", username, host, port, base_path.display()), - session: Arc::new(Mutex::new(sftp)), - base_path, - id, - }) - } - - fn full_path(&self, path: &Path) -> PathBuf { - if path.as_os_str().is_empty() || path == Path::new("/") { - self.base_path.clone() - } else { - self.base_path.join(path.strip_prefix("/").unwrap_or(path)) - } - } -} - -pub enum SftpAuth { - Password(String), - Key { path: PathBuf, passphrase: Option }, -} - -// SSH client handler with host verification -struct SftpHandler { - /// Oracle fix: Path to known_hosts file for verification - known_hosts_path: PathBuf, -} - -impl SftpHandler { - fn new() -> Self { - Self { - known_hosts_path: dirs::home_dir() - .unwrap_or_default() - .join(".ssh") - .join("known_hosts"), - } - } -} - -#[async_trait] -impl russh::client::Handler for SftpHandler { - type Error = russh::Error; - - /// Oracle fix: Verify server key against known_hosts - async fn check_server_key( - &mut self, - server_public_key: &russh_keys::key::PublicKey, - ) -> std::result::Result { - // Load and check known_hosts - if !self.known_hosts_path.exists() { - tracing::warn!("known_hosts not found at {:?}, accepting key (INSECURE)", self.known_hosts_path); - return Ok(true); - } - - // Parse known_hosts and verify key - // In production, use russh_keys::known_hosts module - match russh_keys::check_known_hosts_path( - &self.known_hosts_path, - "", // hostname filled by caller - 0, // port filled by caller - server_public_key, - ) { - Ok(true) => Ok(true), - Ok(false) => { - tracing::error!("SSH host key verification FAILED - potential MITM attack"); - Ok(false) - } - Err(e) => { - tracing::warn!("Could not verify known_hosts: {}", e); - Ok(false) // Fail closed on error - } - } - } -} - -#[async_trait] -impl Origin for SftpOrigin { - fn id(&self) -> &OriginId { - &self.id - } - - fn origin_type(&self) -> OriginType { - OriginType::Sftp - } - - fn display_name(&self) -> &str { - &self.display_name - } - - async fn readdir(&self, path: &Path) -> Result> { - let full_path = self.full_path(path); - let path_str = full_path.to_string_lossy(); - - debug!("SFTP readdir: {}", path_str); - - let sftp = self.session.lock().await; - let entries = sftp.read_dir(&path_str) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - Ok(entries - .into_iter() - .filter(|e| e.filename() != "." && e.filename() != "..") - .map(|e| { - let attrs = e.metadata(); - DirEntry { - name: e.filename().to_string(), - is_dir: attrs.is_dir(), - size: attrs.size.unwrap_or(0), - mtime: attrs.mtime - .map(|t| SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(t as u64)) - .unwrap_or(SystemTime::UNIX_EPOCH), - } - }) - .collect()) - } - - async fn stat(&self, path: &Path) -> Result { - let full_path = self.full_path(path); - let path_str = full_path.to_string_lossy(); - - debug!("SFTP stat: {}", path_str); - - let sftp = self.session.lock().await; - let attrs = sftp.metadata(&path_str) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - Ok(FileStat { - size: attrs.size.unwrap_or(0), - mtime: attrs.mtime - .map(|t| SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(t as u64)) - .unwrap_or(SystemTime::UNIX_EPOCH), - is_dir: attrs.is_dir(), - }) - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result> { - let full_path = self.full_path(path); - let path_str = full_path.to_string_lossy(); - - debug!("SFTP read: {}, offset={}, size={}", path_str, offset, size); - - let sftp = self.session.lock().await; - let mut file = sftp.open(&path_str) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - // Seek to offset - file.seek(std::io::SeekFrom::Start(offset)) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - - // Read data - let mut buffer = vec![0u8; size as usize]; - let mut total_read = 0; - - while total_read < size as usize { - let n = file.read(&mut buffer[total_read..]) - .await - .map_err(|e| musicfs_core::Error::Sftp(e.to_string()))?; - if n == 0 { - break; - } - total_read += n; - } - - buffer.truncate(total_read); - Ok(buffer) - } - - async fn exists(&self, path: &Path) -> Result { - match self.stat(path).await { - Ok(_) => Ok(true), - Err(e) if e.is_not_found() => Ok(false), - Err(e) => Err(e), - } - } - - async fn health(&self) -> HealthStatus { - match self.stat(Path::new("/")).await { - Ok(_) => HealthStatus::Healthy, - Err(_) => HealthStatus::Unhealthy, - } - } - - async fn open_read(&self, path: &Path) -> Result> { - // Oracle fix: Don't read u32::MAX bytes - get actual file size first - let stat = self.stat(path).await?; - let size = stat.size; - - // Oracle fix: For large files, stream in chunks instead of loading all into memory - if size > 100 * 1024 * 1024 { - // >100MB: warn about memory usage - tracing::warn!("SFTP open_read on large file ({} MB) - consider chunked access", size / (1024 * 1024)); - } - - let data = self.read(path, 0, size as u32).await?; - Ok(Box::new(std::io::Cursor::new(data))) - } - - async fn watch(&self, _path: &Path, _callback: WatchCallback) -> Result { - // SFTP doesn't support watching - debug!("SFTP watch not supported, use polling"); - let (tx, _rx) = tokio::sync::oneshot::channel(); - Ok(WatchHandle::new(tx)) - } -} -``` - ---- - -## Task 5: SMB Origin - -### 5.1 Create `musicfs-origins/src/smb.rs` - -```rust -use crate::local::LocalOrigin; -use crate::traits::{Origin, OriginType, WatchCallback, WatchHandle}; -use async_trait::async_trait; -use musicfs_core::{DirEntry, FileStat, HealthStatus, OriginId, Result}; -use std::path::{Path, PathBuf}; -use std::process::Command; -use tracing::{debug, info, warn}; - -/// SMB/CIFS origin -/// -/// Strategy: Assume share is mounted via system mount or gvfs. -/// We wrap LocalOrigin and add SMB-specific error handling. -pub struct SmbOrigin { - inner: LocalOrigin, - share_path: String, -} - -impl SmbOrigin { - /// Create SMB origin from already-mounted share - pub fn from_mount( - id: impl Into, - mount_point: impl Into, - share_path: impl Into, - ) -> Self { - let mount_point = mount_point.into(); - let share_path = share_path.into(); - - Self { - inner: LocalOrigin::new(id, &mount_point), - share_path, - } - } - - /// Check if SMB mount is accessible - pub async fn is_mounted(&self) -> bool { - self.inner.exists(Path::new("/")).await.unwrap_or(false) - } -} - -#[async_trait] -impl Origin for SmbOrigin { - fn id(&self) -> &OriginId { - self.inner.id() - } - - fn origin_type(&self) -> OriginType { - OriginType::Smb - } - - fn display_name(&self) -> &str { - &self.share_path - } - - async fn readdir(&self, path: &Path) -> Result> { - self.inner.readdir(path).await - } - - async fn stat(&self, path: &Path) -> Result { - self.inner.stat(path).await - } - - async fn read(&self, path: &Path, offset: u64, size: u32) -> Result> { - self.inner.read(path, offset, size).await - } - - async fn exists(&self, path: &Path) -> Result { - self.inner.exists(path).await - } - - async fn health(&self) -> HealthStatus { - if self.is_mounted().await { - HealthStatus::Healthy - } else { - HealthStatus::Unhealthy - } - } - - async fn open_read(&self, path: &Path) -> Result> { - self.inner.open_read(path).await - } - - async fn watch(&self, path: &Path, callback: WatchCallback) -> Result { - // Oracle fix: Document inotify unreliability over SMB - // inotify may or may not work over SMB depending on mount options - // For reliable change detection, use polling instead - tracing::warn!( - "SMB watch using inotify - may be unreliable. Consider polling for remote mounts." - ); - self.inner.watch(path, callback).await - } -} -``` - ---- - -## Task 6: Update lib.rs - -### 6.1 Update `musicfs-origins/src/lib.rs` - -```rust -mod failover; -mod health; -mod local; -mod nfs; -mod registry; -mod router; -mod s3; -mod sftp; -mod smb; -mod traits; - -pub use failover::{FailoverExecutor, RetryConfig}; -pub use health::{HealthCheckHandle, HealthMonitor, HealthSnapshot, OriginHealthState}; -pub use local::LocalOrigin; -pub use nfs::NfsOrigin; -pub use registry::OriginRegistry; -pub use router::{LatencyStats, Router}; -pub use s3::S3Origin; -pub use sftp::{SftpAuth, SftpOrigin}; -pub use smb::SmbOrigin; -pub use traits::{Origin, OriginType, WatchCallback, WatchHandle}; -``` - ---- - -## Dependencies - -### `musicfs-origins/Cargo.toml` additions - -```toml -[dependencies] -# Existing -musicfs-core = { path = "../musicfs-core" } -tokio = { workspace = true } -async-trait = { workspace = true } -tracing = { workspace = true } -dashmap = "5" - -# S3 -aws-sdk-s3 = "1" -aws-config = "1" - -# SFTP -russh = "0.43" -russh-sftp = "2" -russh-keys = "0.43" - -# Oracle fix: Connection pooling for SFTP -deadpool = "0.10" - -# Oracle fix: Home directory for known_hosts path -dirs = "5" - -# Optional keyring support -keyring = { version = "2", optional = true } - -[features] -default = [] -keyring = ["dep:keyring"] -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_nfs_origin_basic` | Unit | NFS wrapper works | -| `test_nfs_stale_retry` | Unit | ESTALE handling | -| `test_nfs_retry_uses_fn` | Unit | Oracle fix: Fn not FnMut | -| `test_s3_list_objects` | Integration* | S3 readdir | -| `test_s3_get_object` | Integration* | S3 read | -| `test_s3_range_clamp` | Unit | Oracle fix: no 416 on EOF | -| `test_s3_health_uses_head` | Unit | Oracle fix: head_bucket not list | -| `test_s3_timeout` | Unit | Oracle fix: 30s timeout | -| `test_sftp_connect` | Integration* | SFTP connection | -| `test_sftp_readdir` | Integration* | SFTP listing | -| `test_sftp_pool_concurrency` | Integration* | Oracle fix: pool allows parallel | -| `test_sftp_host_verification` | Unit | Oracle fix: known_hosts checked | -| `test_smb_mounted` | Integration | SMB via mount | -| `test_smb_stale_handling` | Unit | Oracle fix: ENOTCONN handling | -| `test_mixed_origins` | Integration | Local + S3 together | -| `test_credential_debug_redacted` | Unit | Oracle fix: secrets not in Debug | - -*Requires credentials or localstack/test server - ---- - -## Exit Criteria - -- [ ] NFS origin handles ESTALE with retry (using `Fn` not `FnMut`) - Oracle fix -- [ ] S3 origin lists and reads objects -- [ ] S3 range requests clamped to file size (no 416 errors) - Oracle fix -- [ ] S3 health check uses `head_bucket` not `list_objects_v2` - Oracle fix -- [ ] All remote operations have 30s timeout - Oracle fix -- [ ] SFTP uses connection pool (not single mutex) - Oracle fix -- [ ] SFTP verifies SSH host keys against known_hosts - Oracle fix -- [ ] SMB origin works with mounted shares -- [ ] All origins implement health checks -- [ ] Mixed local + remote origins work together -- [ ] Credentials loaded securely (no logging) -- [ ] Credential Debug impl redacts secrets - Oracle fix - ---- - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.3.4 | OriginPlugin interface | ✅ | -| FR-12.2 | NFS support | ✅ | -| FR-12.3 | SMB support | ✅ | -| FR-12.4 | S3 support | ✅ | -| FR-12.5 | SFTP support | ✅ | -| NFR-13.3 | Secure credential storage | ✅ | -| NFR-13.4 | No credential exposure in logs | ✅ | - ---- - -## Security Considerations - -1. **Credentials never logged** - `#[serde(skip_serializing)]` on sensitive fields -2. **Custom Debug impl** - Oracle fix: All secrets redacted in Debug output -3. **Environment variables** - Preferred for CI/CD -4. **Keyring integration** - Uses system secret service -5. **SSH host verification** - Oracle fix: MUST verify against `~/.ssh/known_hosts` -6. **S3 IAM** - Recommend IAM roles over access keys where possible -7. **Connection timeouts** - Oracle fix: 30s timeout on all remote operations prevents DoS diff --git a/docs/v2/plans/week-08-search-index.md b/docs/v2/plans/week-08-search-index.md deleted file mode 100644 index 315d54b..0000000 --- a/docs/v2/plans/week-08-search-index.md +++ /dev/null @@ -1,1266 +0,0 @@ -# Week 8: Search Index - -**Phase**: 3 (Search & Smart Features) -**Prerequisites**: Week 7 (Remote Origins) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement full-text search using tantivy with a virtual `/.search/` directory interface. Users can browse search results as symlinks to matching files, enabling integration with any file manager or media player. - ---- - -## Architecture Reference - -From architecture.md section 4.2: -> Search Engine | Full-text metadata search | tantivy - -From architecture.md section 3.2.1: -> Search query (1M files) | <500ms | 1000ms | FR-14 - -From architecture.md section 8.3: -> tantivy | 0.21+ | Full-text search - ---- - -## Requirements Covered - -| ID | Requirement | Priority | -|----|-------------|----------| -| FR-14.1 | Index metadata for full-text search | P1 | -| FR-14.2 | Expose search via virtual directory (`/.search/query/`) | P1 | -| FR-14.3 | Support fuzzy matching | P1 | -| FR-14.4 | Support search by audio fingerprint | P1 (DEFER) | -| G7 | Sub-second search across 1M+ tracks | Goal | - -**Note**: FR-14.4 (audio fingerprint) requires chromaprint dependency - deferred to Phase 5. - ---- - -## Deliverables - -| Task | Crate | Files | Est. | -|------|-------|-------|------| -| tantivy schema & index | musicfs-search | `index.rs` | 1d | -| Query parser (fuzzy) | musicfs-search | `query.rs` | 0.5d | -| Incremental indexer | musicfs-search | `indexer.rs` | 1d | -| Search virtual directory | musicfs-fuse | `ops/search.rs` | 1d | -| FUSE integration | musicfs-fuse | `filesystem.rs` | 0.5d | -| **gRPC Search API** | musicfs-grpc | `search_service.rs` | 0.5d | -| **API Documentation** | docs | `api/search.md` | 0.5d | -| Integration tests | tests | `search_test.rs` | 0.5d | -| Benchmark (1M tracks) | benches | `search_bench.rs` | 0.5d | - ---- - -## Task 1: tantivy Schema & Index - -### 1.1 Add dependencies to `musicfs-search/Cargo.toml` - -```toml -[package] -name = "musicfs-search" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } - -tantivy = "0.22" -tokio = { workspace = true } -tracing = { workspace = true } -thiserror = { workspace = true } -moka = { version = "0.12", features = ["sync"] } # TTL-based LRU for result cache - -[dev-dependencies] -tempfile = { workspace = true } -``` - -### 1.2 Create `musicfs-search/src/index.rs` - -```rust -use musicfs_core::{FileId, FileMeta, VirtualPath}; -use std::path::Path; -use std::sync::Arc; -use tantivy::collector::TopDocs; -use tantivy::query::QueryParser; -use tantivy::schema::{Field, Schema, STORED, TEXT, INDEXED}; -use tantivy::{Document, Index, IndexReader, IndexWriter, ReloadPolicy}; -use tokio::sync::mpsc; -use tracing::{debug, info, error}; - -/// Commands sent to the single-writer task -pub enum IndexCommand { - Add(FileMeta), - Remove(FileId), - Commit, - Shutdown, -} - -pub struct SearchIndex { - index: Index, - reader: IndexReader, - /// Single-writer channel - IndexWriter is NOT thread-safe - cmd_tx: mpsc::UnboundedSender, - schema: SearchSchema, - /// Schema version for migration detection - pub schema_version: u32, -} - -const SCHEMA_VERSION: u32 = 1; - -struct SearchSchema { - schema: Schema, - file_id: Field, - virtual_path: Field, - artist: Field, - album: Field, - album_artist: Field, // FR-6.4 requires album_artist - title: Field, - genre: Field, - composer: Field, - year: Field, - duration_ms: Field, // Additional fields from architecture SQL schema - bitrate: Field, - sample_rate: Field, -} - -impl SearchSchema { - fn new() -> Self { - let mut builder = Schema::builder(); - - Self { - file_id: builder.add_u64_field("file_id", STORED), - virtual_path: builder.add_text_field("virtual_path", STORED), - artist: builder.add_text_field("artist", TEXT | STORED), - album: builder.add_text_field("album", TEXT | STORED), - album_artist: builder.add_text_field("album_artist", TEXT | STORED), - title: builder.add_text_field("title", TEXT | STORED), - genre: builder.add_text_field("genre", TEXT | STORED), // Now searchable - composer: builder.add_text_field("composer", TEXT | STORED), - year: builder.add_u64_field("year", INDEXED | STORED), // Indexed for range queries - duration_ms: builder.add_u64_field("duration_ms", STORED), - bitrate: builder.add_u64_field("bitrate", STORED), - sample_rate: builder.add_u64_field("sample_rate", STORED), - schema: builder.build(), - } - } -} - -#[derive(Debug, Clone)] -pub struct SearchHit { - pub file_id: FileId, - pub virtual_path: VirtualPath, - pub artist: Option, - pub album: Option, - pub title: Option, - pub score: f32, -} - -impl SearchIndex { - /// Opens the search index and spawns a single-writer background task. - /// IndexWriter is NOT thread-safe - all writes go through the channel. - pub fn open(index_path: &Path) -> Result { - let schema = SearchSchema::new(); - - let index = if index_path.exists() { - Index::open_in_dir(index_path)? - } else { - std::fs::create_dir_all(index_path)?; - Index::create_in_dir(index_path, schema.schema.clone())? - }; - - let reader = index - .reader_builder() - .reload_policy(ReloadPolicy::OnCommit) - .try_into()?; - - // Single-writer pattern: IndexWriter lives in dedicated task - let (cmd_tx, cmd_rx) = mpsc::unbounded_channel(); - let writer = index.writer(50_000_000)?; // 50MB heap - - // Spawn writer task - owns IndexWriter exclusively - Self::spawn_writer_task(writer, cmd_rx, schema.file_id); - - info!("Search index opened at {:?}", index_path); - - Ok(Self { - index, - reader, - cmd_tx, - schema, - schema_version: SCHEMA_VERSION, - }) - } - - /// Spawns background task that owns IndexWriter exclusively. - /// All index mutations go through the channel. - fn spawn_writer_task( - mut writer: IndexWriter, - mut cmd_rx: mpsc::UnboundedReceiver, - file_id_field: Field, - ) { - tokio::spawn(async move { - while let Some(cmd) = cmd_rx.recv().await { - match cmd { - IndexCommand::Add(file) => { - if let Err(e) = Self::add_document(&mut writer, &file, file_id_field) { - error!("Index add failed: {}", e); - } - } - IndexCommand::Remove(id) => { - let term = tantivy::Term::from_field_u64(file_id_field, id.0 as u64); - writer.delete_term(term); - } - IndexCommand::Commit => { - if let Err(e) = writer.commit() { - error!("Index commit failed: {}", e); - } else { - info!("Search index committed"); - } - } - IndexCommand::Shutdown => { - let _ = writer.commit(); - info!("Index writer shutdown"); - break; - } - } - } - }); - } - - fn add_document(writer: &mut IndexWriter, file: &FileMeta, _file_id_field: Field) -> Result<(), SearchError> { - // Document creation happens in writer task - // (schema fields would be passed or stored in writer context) - let _ = writer.add_document(tantivy::doc!())?; - debug!("Indexed file {:?}", file.id); - Ok(()) - } - - /// Queue a file for indexing (non-blocking) - pub fn index_file(&self, file: &FileMeta) -> Result<(), SearchError> { - self.cmd_tx.send(IndexCommand::Add(file.clone())) - .map_err(|_| SearchError::WriterShutdown)?; - Ok(()) - } - - /// Queue file removal (non-blocking) - pub fn remove_file(&self, file_id: FileId) -> Result<(), SearchError> { - self.cmd_tx.send(IndexCommand::Remove(file_id)) - .map_err(|_| SearchError::WriterShutdown)?; - Ok(()) - } - - /// Request commit (non-blocking) - pub fn commit(&self) -> Result<(), SearchError> { - self.cmd_tx.send(IndexCommand::Commit) - .map_err(|_| SearchError::WriterShutdown)?; - Ok(()) - } - - /// Shutdown the writer task gracefully - pub fn shutdown(&self) -> Result<(), SearchError> { - self.cmd_tx.send(IndexCommand::Shutdown) - .map_err(|_| SearchError::WriterShutdown)?; - Ok(()) - } - - pub fn search(&self, query: &str, limit: usize) -> Result, SearchError> { - let searcher = self.reader.searcher(); - - // Include genre in searchable fields (Oracle fix) - let query_parser = QueryParser::for_index( - &self.index, - vec![ - self.schema.artist, - self.schema.album, - self.schema.album_artist, - self.schema.title, - self.schema.genre, // Now searchable - self.schema.composer, - ], - ); - - let query = query_parser.parse_query(query)?; - let top_docs = searcher.search(&query, &TopDocs::with_limit(limit))?; - - let mut results = Vec::with_capacity(top_docs.len()); - for (score, doc_address) in top_docs { - let doc = searcher.doc(doc_address)?; - - let file_id = doc - .get_first(self.schema.file_id) - .and_then(|v| v.as_u64()) - .map(|id| FileId(id as i64)) - .ok_or(SearchError::CorruptedIndex)?; - - let virtual_path = doc - .get_first(self.schema.virtual_path) - .and_then(|v| v.as_text()) - .map(|s| VirtualPath::new(s)) - .ok_or(SearchError::CorruptedIndex)?; - - results.push(SearchHit { - file_id, - virtual_path, - artist: doc.get_first(self.schema.artist).and_then(|v| v.as_text()).map(String::from), - album: doc.get_first(self.schema.album).and_then(|v| v.as_text()).map(String::from), - title: doc.get_first(self.schema.title).and_then(|v| v.as_text()).map(String::from), - score, - }); - } - - debug!("Search '{}' returned {} results", query, results.len()); - Ok(results) - } - - pub fn count(&self) -> u64 { - self.reader.searcher().num_docs() - } -} - -#[derive(Debug, thiserror::Error)] -pub enum SearchError { - #[error("tantivy error: {0}")] - Tantivy(#[from] tantivy::TantivyError), - - #[error("query parse error: {0}")] - QueryParse(#[from] tantivy::query::QueryParserError), - - #[error("IO error: {0}")] - Io(#[from] std::io::Error), - - #[error("corrupted search index")] - CorruptedIndex, - - #[error("index writer shutdown")] - WriterShutdown, -} - -#[cfg(test)] -mod tests { - use super::*; - use musicfs_core::{AudioMeta, RealPath, OriginId}; - use std::path::PathBuf; - use tempfile::TempDir; - - fn make_file(id: i64, artist: &str, album: &str, title: &str) -> FileMeta { - FileMeta { - id: FileId(id), - virtual_path: VirtualPath::new(&format!("/{}/{}/{}.flac", artist, album, title)), - real_path: RealPath { - origin_id: OriginId::from("test"), - path: PathBuf::from("test.flac"), - }, - size: 1000, - mtime: std::time::SystemTime::UNIX_EPOCH, - content_hash: None, - audio: Some(AudioMeta { - artist: Some(artist.to_string()), - album: Some(album.to_string()), - title: Some(title.to_string()), - track_number: Some(1), - duration_ms: Some(180000), - format: musicfs_core::AudioFormat::Flac, - }), - } - } - - #[test] - fn test_search_basic() { - let dir = TempDir::new().unwrap(); - let index = SearchIndex::open(dir.path()).unwrap(); - - index.index_file(&make_file(1, "Metallica", "Black Album", "Enter Sandman")).unwrap(); - index.index_file(&make_file(2, "Metallica", "Master of Puppets", "Battery")).unwrap(); - index.index_file(&make_file(3, "Iron Maiden", "Powerslave", "Aces High")).unwrap(); - index.commit().unwrap(); - - let results = index.search("metallica", 10).unwrap(); - assert_eq!(results.len(), 2); - - let results = index.search("sandman", 10).unwrap(); - assert_eq!(results.len(), 1); - assert_eq!(results[0].title.as_deref(), Some("Enter Sandman")); - } - - #[test] - fn test_search_fuzzy() { - let dir = TempDir::new().unwrap(); - let index = SearchIndex::open(dir.path()).unwrap(); - - index.index_file(&make_file(1, "Metallica", "Black Album", "Enter Sandman")).unwrap(); - index.commit().unwrap(); - - // Fuzzy match with typo - let results = index.search("metalica~1", 10).unwrap(); - assert_eq!(results.len(), 1); - } -} -``` - ---- - -## Task 2: Query Parser - -### 2.1 Create `musicfs-search/src/query.rs` - -```rust -use tantivy::query::{BooleanQuery, FuzzyTermQuery, Occur, Query, TermQuery}; -use tantivy::schema::{Field, IndexRecordOption}; -use tantivy::Term; - -pub struct SearchQueryBuilder { - fields: Vec, - default_fuzziness: u8, -} - -impl SearchQueryBuilder { - pub fn new(fields: Vec) -> Self { - Self { - fields, - default_fuzziness: 1, - } - } - - pub fn with_fuzziness(mut self, fuzziness: u8) -> Self { - self.default_fuzziness = fuzziness; - self - } - - pub fn build_fuzzy(&self, query_text: &str) -> Box { - let terms: Vec<_> = query_text - .split_whitespace() - .filter(|t| !t.is_empty()) - .collect(); - - if terms.is_empty() { - return Box::new(tantivy::query::AllQuery); - } - - let mut clauses: Vec<(Occur, Box)> = Vec::new(); - - for term in terms { - let mut field_queries: Vec<(Occur, Box)> = Vec::new(); - - for field in &self.fields { - let fuzzy = FuzzyTermQuery::new( - Term::from_field_text(*field, &term.to_lowercase()), - self.default_fuzziness, - true, - ); - field_queries.push((Occur::Should, Box::new(fuzzy))); - } - - let field_union = BooleanQuery::new(field_queries); - clauses.push((Occur::Must, Box::new(field_union))); - } - - Box::new(BooleanQuery::new(clauses)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use tantivy::schema::{Schema, TEXT}; - - #[test] - fn test_query_builder() { - let mut schema_builder = Schema::builder(); - let artist = schema_builder.add_text_field("artist", TEXT); - let title = schema_builder.add_text_field("title", TEXT); - - let builder = SearchQueryBuilder::new(vec![artist, title]); - let _query = builder.build_fuzzy("metallica sandman"); - } -} -``` - ---- - -## Task 3: Incremental Indexer - -### 3.1 Create `musicfs-search/src/indexer.rs` - -```rust -use crate::index::{SearchError, SearchIndex}; -use musicfs_cache::MetadataCache; -use musicfs_core::{Event, EventBus, FileId, FileMeta, VirtualPath}; -use std::sync::Arc; -use tokio::sync::mpsc; -use tracing::{debug, error, info, warn}; - -pub struct Indexer { - index: Arc, - event_bus: Arc, - /// MetadataCache for fetching FileMeta on events (Oracle fix - not placeholder) - metadata_cache: Arc, -} - -impl Indexer { - pub fn new( - index: Arc, - event_bus: Arc, - metadata_cache: Arc, - ) -> Self { - Self { index, event_bus, metadata_cache } - } - - pub fn start(self) -> IndexerHandle { - let (stop_tx, mut stop_rx) = mpsc::channel::<()>(1); - let mut event_rx = self.event_bus.subscribe(); - - tokio::spawn(async move { - let mut pending_commit = false; - let mut commit_timer = tokio::time::interval(std::time::Duration::from_secs(5)); - - loop { - tokio::select! { - Ok(event) = event_rx.recv() => { - if let Err(e) = self.handle_event(&event).await { - error!("Indexer error: {}", e); - } - pending_commit = true; - } - _ = commit_timer.tick() => { - if pending_commit { - if let Err(e) = self.index.commit() { - error!("Index commit error: {}", e); - } - pending_commit = false; - } - } - _ = stop_rx.recv() => { - info!("Indexer stopping"); - if pending_commit { - let _ = self.index.commit(); - } - break; - } - } - } - }); - - IndexerHandle { stop_tx } - } - - async fn handle_event(&self, event: &Event) -> Result<(), SearchError> { - match event { - Event::FileAdded { path, file_id } => { - debug!("Indexing added file: {:?}", path); - // Fetch FileMeta from MetadataCache (Oracle fix - real integration) - if let Some(meta) = self.metadata_cache.get_by_path(path).await { - self.index.index_file(&meta)?; - } else { - warn!("No metadata found for added file: {:?}", path); - } - } - Event::FileRemoved { path, file_id } => { - debug!("Removing from index: {:?}", path); - // Lookup FileId and remove from index - if let Some(id) = file_id { - self.index.remove_file(*id)?; - } else if let Some(meta) = self.metadata_cache.get_by_path(path).await { - self.index.remove_file(meta.id)?; - } - } - Event::FileModified { path, file_id } => { - debug!("Re-indexing modified file: {:?}", path); - // Re-index with updated metadata - if let Some(meta) = self.metadata_cache.get_by_path(path).await { - self.index.remove_file(meta.id)?; - self.index.index_file(&meta)?; - } - } - _ => {} - } - Ok(()) - } - - pub fn index_batch(&self, files: &[FileMeta]) -> Result { - let mut count = 0; - for file in files { - self.index.index_file(file)?; - count += 1; - } - self.index.commit()?; - info!("Indexed {} files", count); - Ok(count) - } -} - -pub struct IndexerHandle { - stop_tx: mpsc::Sender<()>, -} - -impl IndexerHandle { - pub async fn stop(self) { - let _ = self.stop_tx.send(()).await; - } -} -``` - ---- - -## Task 4: Search Virtual Directory - -### 4.1 Create `musicfs-fuse/src/ops/search.rs` - -```rust -use fuser::{FileType, ReplyDirectory, ReplyEntry, ReplyData}; -use moka::sync::Cache; -use musicfs_search::{SearchHit, SearchIndex}; -use std::collections::HashMap; -use std::ffi::OsStr; -use std::sync::Arc; -use std::time::{Duration, SystemTime}; -use tracing::debug; - -const SEARCH_DIR_INODE: u64 = 0xFFFF_FFFF_0000_0001; -const SEARCH_RESULT_BASE: u64 = 0xFFFF_FFFF_1000_0000; - -/// Result cache config - prevents unbounded memory growth (Oracle fix) -const RESULT_CACHE_MAX_ENTRIES: u64 = 1000; -const RESULT_CACHE_TTL_SECS: u64 = 300; // 5 minutes - -pub struct SearchOps { - index: Arc, - /// TTL-based LRU cache for search results (moka) - prevents OOM - result_cache: Cache>, - inode_to_result: parking_lot::RwLock>, - /// Mount point for absolute symlink targets - mount_point: String, -} - -impl SearchOps { - pub fn new(index: Arc, mount_point: &str) -> Self { - // moka cache with TTL and max entries (Oracle fix for unbounded growth) - let result_cache = Cache::builder() - .max_capacity(RESULT_CACHE_MAX_ENTRIES) - .time_to_live(Duration::from_secs(RESULT_CACHE_TTL_SECS)) - .build(); - - Self { - index, - result_cache, - inode_to_result: parking_lot::RwLock::new(HashMap::new()), - mount_point: mount_point.to_string(), - } - } - - pub fn is_search_path(path: &str) -> bool { - path.starts_with("/.search/") - } - - pub fn is_search_inode(inode: u64) -> bool { - inode == SEARCH_DIR_INODE || inode >= SEARCH_RESULT_BASE - } - - pub fn lookup_search_dir(&self, reply: ReplyEntry) { - let attr = Self::dir_attr(SEARCH_DIR_INODE); - reply.entry(&Duration::from_secs(60), &attr, 0); - } - - pub fn lookup_query_dir(&self, query: &str, reply: ReplyEntry) { - let results = self.execute_query(query); - if results.is_empty() { - reply.error(libc::ENOENT); - return; - } - - let attr = Self::dir_attr(SEARCH_DIR_INODE + 1); - reply.entry(&Duration::from_secs(1), &attr, 0); - } - - pub fn readdir_search_root(&self, reply: &mut ReplyDirectory) { - reply.add(SEARCH_DIR_INODE, 1, FileType::Directory, "."); - reply.add(1, 2, FileType::Directory, ".."); - } - - pub fn readdir_query(&self, query: &str, offset: i64, reply: &mut ReplyDirectory) { - let results = self.execute_query(query); - - for (i, hit) in results.iter().enumerate().skip(offset as usize) { - let inode = SEARCH_RESULT_BASE + i as u64; - let name = self.result_filename(hit, i); - - { - let mut inode_map = self.inode_to_result.write(); - inode_map.insert(inode, (query.to_string(), i)); - } - - if reply.add(inode, (i + 3) as i64, FileType::Symlink, &name) { - break; - } - } - } - - pub fn readlink(&self, inode: u64, reply: ReplyData) { - let (query, index) = { - let inode_map = self.inode_to_result.read(); - match inode_map.get(&inode) { - Some((q, i)) => (q.clone(), *i), - None => { - reply.error(libc::ENOENT); - return; - } - } - }; - - let results = self.execute_query(&query); - if let Some(hit) = results.get(index) { - // Use ABSOLUTE path for reliable symlink resolution (Oracle fix) - let target = format!("{}{}", self.mount_point, hit.virtual_path.as_str()); - reply.data(target.as_bytes()); - } else { - reply.error(libc::ENOENT); - } - } - - fn execute_query(&self, query: &str) -> Vec { - // moka cache handles TTL/LRU automatically - if let Some(results) = self.result_cache.get(query) { - return results; - } - - let results = self.index.search(query, 1000).unwrap_or_default(); - self.result_cache.insert(query.to_string(), results.clone()); - results - } - - fn result_filename(&self, hit: &SearchHit, index: usize) -> String { - let artist = hit.artist.as_deref().unwrap_or("Unknown"); - let title = hit.title.as_deref().unwrap_or("Unknown"); - format!("{:03}. {} - {}.flac", index + 1, artist, title) - } - - fn dir_attr(inode: u64) -> fuser::FileAttr { - fuser::FileAttr { - ino: inode, - size: 0, - blocks: 0, - atime: SystemTime::UNIX_EPOCH, - mtime: SystemTime::UNIX_EPOCH, - ctime: SystemTime::UNIX_EPOCH, - crtime: SystemTime::UNIX_EPOCH, - kind: FileType::Directory, - perm: 0o555, - nlink: 2, - uid: 1000, - gid: 1000, - rdev: 0, - blksize: 512, - flags: 0, - } - } -} -``` - ---- - -## Task 5: FUSE Integration - -### 5.1 Update `musicfs-fuse/src/filesystem.rs` - -Add search handling to FUSE operations: - -```rust -// In lookup() -if name == ".search" && parent == 1 { - self.search_ops.lookup_search_dir(reply); - return; -} - -if let Some(path) = self.inode_to_path(parent) { - if path.starts_with("/.search/") { - let query = &path[9..]; // Strip "/.search/" - self.search_ops.lookup_query_dir(query, reply); - return; - } -} - -// In readdir() -if ino == SEARCH_DIR_INODE { - self.search_ops.readdir_search_root(&mut reply); - reply.ok(); - return; -} - -// In readlink() -if SearchOps::is_search_inode(ino) { - self.search_ops.readlink(ino, reply); - return; -} -``` - ---- - -## Task 6: gRPC Search API - -**Oracle fix**: Architecture 4.3.7 defines `Search` and `SearchStream` RPCs that must be implemented. - -### 6.1 Create `musicfs-grpc/src/search_service.rs` - -```rust -use musicfs_proto::musicfs::v1::{ - SearchRequest, SearchResponse, SearchResult, - music_fs_server::MusicFs, -}; -use musicfs_search::{SearchIndex, SearchHit}; -use std::sync::Arc; -use std::time::Instant; -use tonic::{Request, Response, Status}; -use tracing::{debug, info}; - -pub struct SearchService { - index: Arc, -} - -impl SearchService { - pub fn new(index: Arc) -> Self { - Self { index } - } -} - -#[tonic::async_trait] -impl MusicFs for SearchService { - async fn search( - &self, - request: Request, - ) -> Result, Status> { - let start = Instant::now(); - let req = request.into_inner(); - - let limit = req.limit.unwrap_or(100) as usize; - let offset = req.offset.unwrap_or(0) as usize; - - // Execute search - let results = self.index - .search(&req.query, limit + offset) - .map_err(|e| Status::internal(format!("Search failed: {}", e)))?; - - // Apply offset and convert to proto - let hits: Vec = results - .into_iter() - .skip(offset) - .take(limit) - .map(|hit| SearchResult { - file_id: hit.file_id.0, - virtual_path: hit.virtual_path.to_string(), - artist: hit.artist, - album: hit.album, - title: hit.title, - score: hit.score, - highlights: Default::default(), // TODO: implement highlighting - }) - .collect(); - - let total_matches = self.index.count() as u64; // Approximate - let query_time_ms = start.elapsed().as_millis() as u32; - - debug!("Search '{}' returned {} results in {}ms", req.query, hits.len(), query_time_ms); - - Ok(Response::new(SearchResponse { - results: hits, - total_matches, - query_time_ms, - })) - } - - type SearchStreamStream = tokio_stream::wrappers::ReceiverStream>; - - async fn search_stream( - &self, - request: Request, - ) -> Result, Status> { - let req = request.into_inner(); - let limit = req.limit.unwrap_or(1000) as usize; - - let results = self.index - .search(&req.query, limit) - .map_err(|e| Status::internal(format!("Search failed: {}", e)))?; - - let (tx, rx) = tokio::sync::mpsc::channel(100); - - tokio::spawn(async move { - for hit in results { - let result = SearchResult { - file_id: hit.file_id.0, - virtual_path: hit.virtual_path.to_string(), - artist: hit.artist, - album: hit.album, - title: hit.title, - score: hit.score, - highlights: Default::default(), - }; - if tx.send(Ok(result)).await.is_err() { - break; // Client disconnected - } - } - }); - - Ok(Response::new(tokio_stream::wrappers::ReceiverStream::new(rx))) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - #[tokio::test] - async fn test_grpc_search() { - let dir = TempDir::new().unwrap(); - let index = Arc::new(SearchIndex::open(dir.path()).unwrap()); - let service = SearchService::new(index); - - let request = Request::new(SearchRequest { - query: "test".to_string(), - limit: Some(10), - offset: None, - origin_id: None, - }); - - let response = service.search(request).await.unwrap(); - assert!(response.get_ref().query_time_ms > 0); - } -} -``` - ---- - -## Task 7: API Documentation - -**All APIs must be fully documented with happy and non-happy paths.** - -### 7.1 Create `docs/api/search.md` - -```markdown -# Search API Documentation - -## Overview - -MusicFS provides two search interfaces: -1. **FUSE Virtual Directory** - `/.search/query/` for file manager integration -2. **gRPC API** - `Search` and `SearchStream` RPCs for programmatic access - ---- - -## FUSE Search Interface - -### Endpoint: `/.search/{query}/` - -Browse search results as symlinks in a virtual directory. - -### Happy Path - -1. User navigates to `/.search/metallica/` -2. FUSE returns directory listing of symlinks -3. Each symlink points to absolute path: `/mnt/music/Metallica/Album/Track.flac` -4. User can open symlink directly in media player - -**Example:** -```bash -$ ls -la /mnt/musicfs/.search/metallica/ -001. Metallica - Enter Sandman.flac -> /mnt/musicfs/Metallica/Black Album/Enter Sandman.flac -002. Metallica - Battery.flac -> /mnt/musicfs/Metallica/Master of Puppets/Battery.flac -``` - -### Error Cases - -| Scenario | Behavior | FUSE Error | -|----------|----------|------------| -| Empty query | Empty directory | (none) | -| No results | Empty directory | (none) | -| Query too long (>256 chars) | Truncated | (none) | -| Invalid UTF-8 in query | EINVAL | `libc::EINVAL` | -| Index corrupted | ENOENT | `libc::ENOENT` | -| Index writer shutdown | EIO | `libc::EIO` | - -### Cache Behavior - -- Results cached for 5 minutes (TTL) -- Maximum 1000 cached queries (LRU eviction) -- Cache miss triggers tantivy query - ---- - -## gRPC Search API - -### `Search(SearchRequest) -> SearchResponse` - -Single request/response search. - -#### Request Schema - -```protobuf -message SearchRequest { - string query = 1; // Required: tantivy query string - optional uint32 limit = 2; // Default: 100, max: 10000 - optional uint32 offset = 3; // Default: 0, for pagination - optional string origin_id = 4; // Filter by origin (optional) -} -``` - -#### Response Schema - -```protobuf -message SearchResponse { - repeated SearchResult results = 1; - uint64 total_matches = 2; // Approximate total - uint32 query_time_ms = 3; // Query execution time -} - -message SearchResult { - int64 file_id = 1; - string virtual_path = 2; - optional string artist = 3; - optional string album = 4; - optional string title = 5; - float score = 6; // Relevance score - map highlights = 7; // Matched fragments -} -``` - -### Happy Path - -``` -Client Server - | | - |-- SearchRequest ------------->| - | query: "metallica" | - | limit: 10 | - | |-- Query tantivy index - | |-- Collect top 10 results - |<-- SearchResponse ------------| - | results: [...] | - | total_matches: 42 | - | query_time_ms: 12 | -``` - -### Error Cases - -| Scenario | gRPC Status | Details | -|----------|-------------|---------| -| Empty query | `INVALID_ARGUMENT` | "Query cannot be empty" | -| Malformed query syntax | `INVALID_ARGUMENT` | tantivy parse error message | -| limit > 10000 | `INVALID_ARGUMENT` | "Limit exceeds maximum (10000)" | -| Index unavailable | `UNAVAILABLE` | "Search index not ready" | -| Index corrupted | `INTERNAL` | "Search index corrupted" | -| Writer shutdown | `INTERNAL` | "Index writer shutdown" | -| Timeout (>5s) | `DEADLINE_EXCEEDED` | Client-specified deadline | - -### Retry Strategy - -| Error | Retryable | Backoff | -|-------|-----------|---------| -| `UNAVAILABLE` | Yes | Exponential (100ms, 200ms, 400ms) | -| `DEADLINE_EXCEEDED` | Yes | None (immediate) | -| `INTERNAL` | No | - | -| `INVALID_ARGUMENT` | No | - | - ---- - -### `SearchStream(SearchRequest) -> stream SearchResult` - -Streaming search for large result sets. - -### Happy Path - -``` -Client Server - | | - |-- SearchRequest ------------->| - | query: "rock" | - | limit: 10000 | - | |-- Query tantivy index - |<-- SearchResult (stream) -----| - |<-- SearchResult --------------| - |<-- SearchResult --------------| - | ... (continues) | - |<-- (stream ends) -------------| -``` - -### Error Cases - -Same as `Search`, plus: - -| Scenario | Behavior | -|----------|----------| -| Client disconnects mid-stream | Server stops sending, cleans up | -| Backpressure (slow client) | Server buffers up to 100 results | -| Buffer overflow | Server drops connection | - ---- - -## Query Syntax - -MusicFS uses tantivy query syntax. - -### Supported Operators - -| Operator | Example | Description | -|----------|---------|-------------| -| Term | `metallica` | Match in any field | -| Field | `artist:metallica` | Match specific field | -| Phrase | `"enter sandman"` | Exact phrase match | -| Fuzzy | `metalica~1` | 1-character edit distance | -| Boolean | `metallica AND 1991` | Combine conditions | -| Range | `year:[1980 TO 1989]` | Numeric range | - -### Searchable Fields - -| Field | Type | Notes | -|-------|------|-------| -| `artist` | TEXT | Full-text searchable | -| `album` | TEXT | Full-text searchable | -| `album_artist` | TEXT | Full-text searchable | -| `title` | TEXT | Full-text searchable | -| `genre` | TEXT | Full-text searchable | -| `composer` | TEXT | Full-text searchable | -| `year` | u64 | Range queries only | - ---- - -## Performance - -| Metric | Target | Measured | -|--------|--------|----------| -| Query latency (1M tracks) | <500ms | TBD | -| Index throughput | >1000 files/sec | TBD | -| Memory per 1M tracks | <500MB | TBD | - ---- - -## Integration Examples - -### CLI Search - -```bash -# Using grpcurl -grpcurl -plaintext -d '{"query": "metallica", "limit": 5}' \ - localhost:50051 musicfs.v1.MusicFS/Search - -# Using musicfs-cli -musicfs search "artist:metallica AND year:[1980 TO 1990]" -``` - -### Programmatic (Rust) - -```rust -use musicfs_client::MusicFsClient; - -let mut client = MusicFsClient::connect("http://localhost:50051").await?; - -let response = client.search(SearchRequest { - query: "metallica".to_string(), - limit: Some(10), - ..Default::default() -}).await?; - -for result in response.results { - println!("{} - {}", result.artist.unwrap_or_default(), result.title.unwrap_or_default()); -} -``` -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_search_basic` | Unit | Basic search returns results | -| `test_search_fuzzy` | Unit | Typo tolerance (FR-14.3) | -| `test_search_multi_field` | Unit | Searches artist+album+title | -| `test_search_empty` | Unit | Empty query returns nothing | -| `test_index_persistence` | Integration | Index survives restart | -| `test_incremental_index` | Integration | New files indexed via events | -| `test_search_virtual_dir` | E2E | `ls /.search/metallica/` works | -| `test_search_symlinks` | E2E | Results are valid symlinks | -| `test_search_1m_tracks` | Benchmark | <500ms for 1M tracks (G7) | - ---- - -## Benchmark - -```rust -// benches/search_bench.rs -use criterion::{criterion_group, criterion_main, Criterion}; - -fn bench_search_1m(c: &mut Criterion) { - // Pre-populate index with 1M synthetic tracks - let index = create_index_with_n_tracks(1_000_000); - - c.bench_function("search_1m_tracks", |b| { - b.iter(|| { - index.search("metallica master puppets", 100).unwrap() - }) - }); -} - -fn bench_index_throughput(c: &mut Criterion) { - c.bench_function("index_1000_tracks", |b| { - let dir = tempfile::TempDir::new().unwrap(); - let index = SearchIndex::open(dir.path()).unwrap(); - let files = generate_test_files(1000); - - b.iter(|| { - for file in &files { - index.index_file(file).unwrap(); - } - index.commit().unwrap(); - }) - }); -} - -criterion_group!(benches, bench_search_1m, bench_index_throughput); -criterion_main!(benches); -``` - ---- - -## Exit Criteria - -- [ ] tantivy index opens/creates successfully -- [ ] Files are indexed with artist/album/album_artist/title/genre/composer -- [ ] Search returns relevant results in <500ms for 1M tracks -- [ ] Fuzzy matching handles typos (e.g., "metalica" finds "Metallica") -- [ ] `/.search/query/` directory shows symlinks to results -- [ ] Symlinks resolve to actual files (absolute paths) -- [ ] Index persists across daemon restarts -- [ ] New files are indexed via event bus + MetadataCache integration -- [ ] gRPC `Search` and `SearchStream` RPCs functional -- [ ] Result cache uses TTL-based LRU (moka), max 1000 entries -- [ ] IndexWriter uses single-writer channel pattern (thread-safe) -- [ ] API documentation covers happy/error paths for FUSE and gRPC - ---- - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.2 | Search Engine: tantivy | ✅ | -| 4.3.7 | Search RPC (Search, SearchStream) | ✅ Task 6 | -| 3.2.1 | Search <500ms for 1M files | ✅ Benchmark | -| FR-14.1 | Index metadata for full-text search | ✅ | -| FR-14.2 | Expose via virtual directory | ✅ | -| FR-14.3 | Support fuzzy matching | ✅ | -| FR-6.4 | album_artist field indexed | ✅ Schema | -| G7 | Sub-second search 1M+ tracks | ✅ Benchmark | - -## Oracle Fixes Applied - -| Issue | Fix | Location | -|-------|-----|----------| -| IndexWriter thread-safety | Single-writer channel pattern | `index.rs` | -| Unbounded result cache | moka TTL-based LRU (1000 max, 5min TTL) | `search.rs` | -| gRPC Search API missing | Task 6 added | `search_service.rs` | -| Event handler incomplete | MetadataCache integration | `indexer.rs` | -| Genre not searchable | Added to QueryParser fields | `index.rs` | -| Missing fields | album_artist, composer, duration_ms, bitrate, sample_rate | `index.rs` | -| Relative symlinks | Absolute paths with mount_point | `search.rs` | diff --git a/docs/v2/plans/week-09-smart-features.md b/docs/v2/plans/week-09-smart-features.md deleted file mode 100644 index bb68d68..0000000 --- a/docs/v2/plans/week-09-smart-features.md +++ /dev/null @@ -1,1686 +0,0 @@ -# Week 9: Smart Features - -**Phase**: 3 (Search & Smart Features) -**Prerequisites**: Week 8 (Search Index) -**Estimated effort**: 5 days - ---- - -## Objective - -Implement smart collections (query-based virtual folders), cover art extraction with thumbnails, and intelligent prefetching based on access patterns. These features transform MusicFS from a basic filesystem into an intelligent music library. - ---- - -## Architecture Reference - -From architecture.md section 4.3.6 (Data Schema): -```sql -CREATE TABLE artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER REFERENCES files(id), - art_type TEXT, -- 'front', 'back' - chunk_hash TEXT, -- reference to CAS - width INTEGER, - height INTEGER, - UNIQUE(file_id, art_type) -); - -CREATE TABLE collections ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE, - query_json TEXT, -- smart collection query - created_at INTEGER -); -``` - -From architecture.md section 3.2.5: -> Cache hit rate (warm) | >95% | Derived -> Deduplication ratio | >10% typical | FR-20 - ---- - -## Requirements Covered - -| ID | Requirement | Priority | -|----|-------------|----------| -| FR-15.1 | Support query-based virtual folders | P1 | -| FR-15.2 | Support saved searches as directories | P1 | -| FR-15.3 | Support dynamic playlists (recently played, most played) | P1 | -| FR-15.4 | Support user-defined metadata fields | P1 (DEFER) | -| FR-16.1 | Extract embedded album art | P1 | -| FR-16.2 | Expose art as virtual files (`cover.jpg`) | P1 | -| FR-16.3 | Cache artwork separately from audio | P1 | -| FR-16.4 | Support multiple art sizes (thumbnail, medium, full) | P1 | -| FR-19.1 | Learn access patterns | P1 | -| FR-19.2 | Support playlist-aware prefetching | P1 | -| FR-19.3 | Support time-based prefetching | P1 | -| FR-19.4 | Support manual prefetch hints (`/.prefetch/`) | P1 | - -**Note**: FR-15.4 (user-defined metadata) deferred to plugin system (Phase 4). - ---- - -## Deliverables - -| Task | Crate | Files | Est. | -|------|-------|-------|------| -| Smart collections | musicfs-search | `collections.rs` | 1d | -| Collection virtual dirs | musicfs-fuse | `ops/collections.rs` | 0.5d | -| Artwork extractor | musicfs-metadata | `artwork.rs` | 1d | -| Artwork cache (CAS) | musicfs-cache | `artwork.rs` | 0.5d | -| Prefetch engine | musicfs-cache | `prefetch.rs` | 1d | -| Access pattern tracker | musicfs-cache | `patterns.rs` | 0.5d | -| **Prefetch virtual dir** | musicfs-fuse | `ops/prefetch.rs` | 0.5d | -| **API Documentation** | docs | `api/smart-features.md` | 0.5d | -| Integration tests | tests | `smart_features.rs` | 0.5d | - ---- - -## Task 1: Smart Collections - -### 1.1 Create `musicfs-search/src/collections.rs` - -```rust -use musicfs_core::FileId; -use serde::{Deserialize, Serialize}; -use std::time::{Duration, SystemTime}; - -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct SmartCollection { - pub id: i64, - pub name: String, - pub query: CollectionQuery, - pub created_at: SystemTime, -} - -#[derive(Debug, Clone, Serialize, Deserialize)] -#[serde(tag = "type")] -pub enum CollectionQuery { - /// Match field against pattern - Match { - field: String, - pattern: String, - }, - - /// Date range (e.g., year between 1980-1989) - DateRange { - field: String, - start: i32, - end: i32, - }, - - /// Recently added files - RecentlyAdded { - days: u32, - }, - - /// Recently played files - RecentlyPlayed { - days: u32, - }, - - /// Most played files - MostPlayed { - limit: u32, - }, - - /// Genre-based collection - Genre { - genre: String, - }, - - /// Compound query (AND/OR) - Compound { - op: BoolOp, - children: Vec, - }, -} - -#[derive(Debug, Clone, Copy, Serialize, Deserialize)] -pub enum BoolOp { - And, - Or, -} - -impl CollectionQuery { - pub fn to_tantivy_query(&self) -> String { - match self { - CollectionQuery::Match { field, pattern } => { - format!("{}:{}", field, pattern) - } - CollectionQuery::DateRange { field, start, end } => { - format!("{}:[{} TO {}]", field, start, end) - } - CollectionQuery::Genre { genre } => { - format!("genre:{}", genre) - } - CollectionQuery::Compound { op, children } => { - let sep = match op { - BoolOp::And => " AND ", - BoolOp::Or => " OR ", - }; - let parts: Vec<_> = children.iter() - .map(|c| format!("({})", c.to_tantivy_query())) - .collect(); - parts.join(sep) - } - // Dynamic queries handled separately - _ => String::new(), - } - } - - pub fn is_dynamic(&self) -> bool { - matches!( - self, - CollectionQuery::RecentlyAdded { .. } - | CollectionQuery::RecentlyPlayed { .. } - | CollectionQuery::MostPlayed { .. } - ) - } -} - -pub struct CollectionStore { - db: rusqlite::Connection, -} - -impl CollectionStore { - pub fn new(db_path: &std::path::Path) -> Result { - let db = rusqlite::Connection::open(db_path)?; - - db.execute( - "CREATE TABLE IF NOT EXISTS collections ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE NOT NULL, - query_json TEXT NOT NULL, - created_at INTEGER NOT NULL - )", - [], - )?; - - Ok(Self { db }) - } - - pub fn create(&mut self, name: &str, query: CollectionQuery) -> Result { - let query_json = serde_json::to_string(&query)?; - let now = SystemTime::now() - .duration_since(SystemTime::UNIX_EPOCH) - .unwrap() - .as_secs() as i64; - - self.db.execute( - "INSERT INTO collections (name, query_json, created_at) VALUES (?1, ?2, ?3)", - rusqlite::params![name, query_json, now], - )?; - - let id = self.db.last_insert_rowid(); - - Ok(SmartCollection { - id, - name: name.to_string(), - query, - created_at: SystemTime::UNIX_EPOCH + Duration::from_secs(now as u64), - }) - } - - pub fn list(&self) -> Result, CollectionError> { - let mut stmt = self.db.prepare( - "SELECT id, name, query_json, created_at FROM collections" - )?; - - let collections = stmt.query_map([], |row| { - let query_json: String = row.get(2)?; - let created_secs: i64 = row.get(3)?; - - Ok(SmartCollection { - id: row.get(0)?, - name: row.get(1)?, - query: serde_json::from_str(&query_json).unwrap_or(CollectionQuery::Match { - field: "title".to_string(), - pattern: "*".to_string(), - }), - created_at: SystemTime::UNIX_EPOCH + Duration::from_secs(created_secs as u64), - }) - })?; - - collections.collect::, _>>().map_err(CollectionError::from) - } - - pub fn delete(&mut self, name: &str) -> Result<(), CollectionError> { - self.db.execute("DELETE FROM collections WHERE name = ?1", [name])?; - Ok(()) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum CollectionError { - #[error("database error: {0}")] - Database(#[from] rusqlite::Error), - - #[error("serialization error: {0}")] - Serialization(#[from] serde_json::Error), -} - -/// Built-in collections -pub fn builtin_collections() -> Vec { - vec![ - SmartCollection { - id: -1, - name: "Recently Added".to_string(), - query: CollectionQuery::RecentlyAdded { days: 30 }, - created_at: SystemTime::UNIX_EPOCH, - }, - SmartCollection { - id: -2, - name: "80s Music".to_string(), - query: CollectionQuery::DateRange { - field: "year".to_string(), - start: 1980, - end: 1989, - }, - created_at: SystemTime::UNIX_EPOCH, - }, - SmartCollection { - id: -3, - name: "90s Music".to_string(), - query: CollectionQuery::DateRange { - field: "year".to_string(), - start: 1990, - end: 1999, - }, - created_at: SystemTime::UNIX_EPOCH, - }, - ] -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - #[test] - fn test_collection_crud() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("collections.db"); - let mut store = CollectionStore::new(&db_path).unwrap(); - - let collection = store.create( - "Jazz", - CollectionQuery::Genre { genre: "Jazz".to_string() }, - ).unwrap(); - - assert_eq!(collection.name, "Jazz"); - - let collections = store.list().unwrap(); - assert_eq!(collections.len(), 1); - - store.delete("Jazz").unwrap(); - let collections = store.list().unwrap(); - assert_eq!(collections.len(), 0); - } - - #[test] - fn test_compound_query() { - let query = CollectionQuery::Compound { - op: BoolOp::And, - children: vec![ - CollectionQuery::Genre { genre: "Metal".to_string() }, - CollectionQuery::DateRange { - field: "year".to_string(), - start: 1980, - end: 1989, - }, - ], - }; - - let tantivy_query = query.to_tantivy_query(); - assert!(tantivy_query.contains("genre:Metal")); - assert!(tantivy_query.contains("year:[1980 TO 1989]")); - assert!(tantivy_query.contains(" AND ")); - } -} -``` - ---- - -## Task 2: Artwork Extraction - -### 2.1 Add dependencies to `musicfs-metadata/Cargo.toml` - -```toml -[dependencies] -image = { version = "0.24", default-features = false, features = ["jpeg", "png"] } -``` - -### 2.2 Create `musicfs-metadata/src/artwork.rs` - -```rust -use image::{DynamicImage, ImageFormat}; -use std::io::Cursor; -use symphonia::core::meta::Visual; -use tracing::debug; - -#[derive(Debug, Clone)] -pub struct Artwork { - pub art_type: ArtType, - pub mime_type: String, - pub width: u32, - pub height: u32, - pub data: Vec, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ArtType { - Front, - Back, - Other, -} - -#[derive(Debug, Clone, Copy)] -pub enum ArtSize { - Thumbnail, // 150x150 - Medium, // 300x300 - Full, // Original -} - -impl ArtSize { - pub fn max_dimension(&self) -> Option { - match self { - ArtSize::Thumbnail => Some(150), - ArtSize::Medium => Some(300), - ArtSize::Full => None, - } - } -} - -pub struct ArtworkExtractor; - -impl ArtworkExtractor { - pub fn extract_from_visual(visual: &Visual) -> Option { - let data = visual.data.to_vec(); - - let img = image::load_from_memory(&data).ok()?; - - let art_type = match visual.usage { - Some(symphonia::core::meta::StandardVisualKey::FrontCover) => ArtType::Front, - Some(symphonia::core::meta::StandardVisualKey::BackCover) => ArtType::Back, - _ => ArtType::Other, - }; - - let mime_type = visual.media_type.clone() - .unwrap_or_else(|| "image/jpeg".to_string()); - - Some(Artwork { - art_type, - mime_type, - width: img.width(), - height: img.height(), - data, - }) - } - - pub fn resize(artwork: &Artwork, size: ArtSize) -> Option { - let max_dim = size.max_dimension()?; - - if artwork.width <= max_dim && artwork.height <= max_dim { - return Some(artwork.clone()); - } - - let img = image::load_from_memory(&artwork.data).ok()?; - let resized = img.thumbnail(max_dim, max_dim); - - let mut output = Vec::new(); - let mut cursor = Cursor::new(&mut output); - resized.write_to(&mut cursor, ImageFormat::Jpeg).ok()?; - - debug!( - "Resized artwork from {}x{} to {}x{}", - artwork.width, artwork.height, - resized.width(), resized.height() - ); - - Some(Artwork { - art_type: artwork.art_type, - mime_type: "image/jpeg".to_string(), - width: resized.width(), - height: resized.height(), - data: output, - }) - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_art_size_dimensions() { - assert_eq!(ArtSize::Thumbnail.max_dimension(), Some(150)); - assert_eq!(ArtSize::Medium.max_dimension(), Some(300)); - assert_eq!(ArtSize::Full.max_dimension(), None); - } -} -``` - -### 2.3 Create `musicfs-cache/src/artwork.rs` - -```rust -use musicfs_core::ChunkHash; -use musicfs_metadata::artwork::{ArtSize, Artwork}; -use crate::CasStore; -use std::sync::Arc; -use tracing::debug; - -pub struct ArtworkCache { - store: Arc, - db: rusqlite::Connection, -} - -#[derive(Debug)] -pub struct CachedArtwork { - pub file_id: i64, - pub art_type: String, - pub chunk_hash: ChunkHash, - pub width: u32, - pub height: u32, -} - -/// Oracle fix: Max input size to prevent memory spikes (3000x3000 = ~36MB) -const MAX_ARTWORK_INPUT_SIZE: usize = 10 * 1024 * 1024; // 10MB - -impl ArtworkCache { - pub fn new(store: Arc, db_path: &std::path::Path) -> Result { - let db = rusqlite::Connection::open(db_path)?; - - // Oracle fix: Schema matches architecture.md 4.3.6 exactly - // Only store full-size artwork, generate thumbnail/medium on-demand - db.execute( - "CREATE TABLE IF NOT EXISTS artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL REFERENCES files(id), - art_type TEXT NOT NULL, - chunk_hash TEXT NOT NULL, - width INTEGER NOT NULL, - height INTEGER NOT NULL, - UNIQUE(file_id, art_type) - )", - [], - )?; - - Ok(Self { store, db }) - } - - /// Store full-size artwork only (Oracle fix: no size column) - /// Thumbnail/medium generated on-demand with in-memory LRU - pub async fn store(&self, file_id: i64, artwork: &Artwork) -> Result { - // Oracle fix: Reject oversized images to prevent memory spikes - if artwork.data.len() > MAX_ARTWORK_INPUT_SIZE { - return Err(ArtworkError::ImageTooLarge(artwork.data.len())); - } - - let hash = self.store.put(&artwork.data).await?; - - let art_type_str = match artwork.art_type { - musicfs_metadata::artwork::ArtType::Front => "front", - musicfs_metadata::artwork::ArtType::Back => "back", - musicfs_metadata::artwork::ArtType::Other => "other", - }; - - // Oracle fix: Use spawn_blocking for rusqlite in async context - let db_path = self.db.path().map(|p| p.to_path_buf()); - let file_id_clone = file_id; - let art_type_clone = art_type_str.to_string(); - let hash_hex = hash.to_hex(); - let width = artwork.width; - let height = artwork.height; - - tokio::task::spawn_blocking(move || { - let db = rusqlite::Connection::open(db_path.unwrap())?; - db.execute( - "INSERT OR REPLACE INTO artwork - (file_id, art_type, chunk_hash, width, height) - VALUES (?1, ?2, ?3, ?4, ?5)", - rusqlite::params![file_id_clone, art_type_clone, hash_hex, width, height], - )?; - Ok::<_, ArtworkError>(()) - }).await.map_err(|e| ArtworkError::SpawnBlocking(e.to_string()))??; - - debug!("Cached artwork for file {}", file_id); - Ok(hash) - } - - /// Get full-size artwork, optionally resize on-demand - pub async fn get(&self, file_id: i64, art_type: &str, size: ArtSize) -> Result>, ArtworkError> { - // Oracle fix: Use spawn_blocking for rusqlite - let db_path = self.db.path().map(|p| p.to_path_buf()); - let file_id_clone = file_id; - let art_type_clone = art_type.to_string(); - - let hash_hex: Option = tokio::task::spawn_blocking(move || { - let db = rusqlite::Connection::open(db_path.unwrap())?; - db.query_row( - "SELECT chunk_hash FROM artwork WHERE file_id = ?1 AND art_type = ?2", - rusqlite::params![file_id_clone, art_type_clone], - |row| row.get(0), - ).ok().ok_or(ArtworkError::NotFound) - }).await.map_err(|e| ArtworkError::SpawnBlocking(e.to_string()))?.ok(); - - match hash_hex { - Some(hex) => { - let hash = ChunkHash::from_hex(&hex).ok_or(ArtworkError::InvalidHash)?; - let data = self.store.get(&hash).await?; - - // On-demand resize if not full size - match size { - ArtSize::Full => Ok(Some(data.to_vec())), - ArtSize::Thumbnail | ArtSize::Medium => { - // Resize on-demand (could add LRU cache here) - let resized = self.resize_on_demand(&data, size)?; - Ok(Some(resized)) - } - } - } - None => Ok(None), - } - } - - fn resize_on_demand(&self, data: &[u8], size: ArtSize) -> Result, ArtworkError> { - use image::ImageFormat; - use std::io::Cursor; - - let max_dim = size.max_dimension().unwrap_or(300); - let img = image::load_from_memory(data).map_err(|_| ArtworkError::InvalidImage)?; - - if img.width() <= max_dim && img.height() <= max_dim { - return Ok(data.to_vec()); - } - - let resized = img.thumbnail(max_dim, max_dim); - let mut output = Vec::new(); - let mut cursor = Cursor::new(&mut output); - resized.write_to(&mut cursor, ImageFormat::Jpeg).map_err(|_| ArtworkError::ResizeFailed)?; - - Ok(output) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum ArtworkError { - #[error("database error: {0}")] - Database(#[from] rusqlite::Error), - - #[error("CAS error: {0}")] - Cas(#[from] crate::store::CasError), - - #[error("invalid hash")] - InvalidHash, - - #[error("artwork not found")] - NotFound, - - #[error("image too large: {0} bytes (max 10MB)")] - ImageTooLarge(usize), - - #[error("invalid image data")] - InvalidImage, - - #[error("resize failed")] - ResizeFailed, - - #[error("spawn_blocking error: {0}")] - SpawnBlocking(String), -} -``` - ---- - -## Task 3: Prefetch Engine - -### 3.1 Create `musicfs-cache/src/patterns.rs` - -```rust -use musicfs_core::FileId; -use std::collections::HashMap; -use std::path::Path; -use std::time::{Duration, SystemTime, UNIX_EPOCH}; - -/// Oracle fix: Use SystemTime for persistence, not Instant -pub struct AccessPattern { - file_id: FileId, - timestamp: SystemTime, - context: AccessContext, - hour_of_day: u8, // For time-based prefetch (FR-19.3) -} - -#[derive(Debug, Clone)] -pub struct AccessContext { - pub album_id: Option, - pub track_number: Option, - pub artist: Option, -} - -/// Oracle fix: Persistent pattern store with SQLite -pub struct PatternStore { - db: rusqlite::Connection, - /// In-memory cache for hot path - sequence_counts: parking_lot::RwLock>, - /// Time-based patterns for FR-19.3 - time_patterns: parking_lot::RwLock>>, // hour -> files - max_history: usize, -} - -impl PatternStore { - pub fn new(db_path: &Path, max_history: usize) -> Result { - let db = rusqlite::Connection::open(db_path)?; - - // Oracle fix: Persist access log for RecentlyPlayed/MostPlayed queries - db.execute( - "CREATE TABLE IF NOT EXISTS access_log ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL, - access_time INTEGER NOT NULL, - hour_of_day INTEGER NOT NULL - )", - [], - )?; - - db.execute( - "CREATE INDEX IF NOT EXISTS idx_access_log_file ON access_log(file_id)", - [], - )?; - - db.execute( - "CREATE INDEX IF NOT EXISTS idx_access_log_time ON access_log(access_time)", - [], - )?; - - // Sequence transitions table - db.execute( - "CREATE TABLE IF NOT EXISTS sequence_counts ( - from_file_id INTEGER NOT NULL, - to_file_id INTEGER NOT NULL, - count INTEGER NOT NULL DEFAULT 1, - PRIMARY KEY (from_file_id, to_file_id) - )", - [], - )?; - - // Load sequence counts into memory - let mut sequence_counts = HashMap::new(); - let mut stmt = db.prepare("SELECT from_file_id, to_file_id, count FROM sequence_counts")?; - let rows = stmt.query_map([], |row| { - Ok(((FileId(row.get::<_, i64>(0)?), FileId(row.get::<_, i64>(1)?)), row.get::<_, u32>(2)?)) - })?; - for row in rows { - let (key, count) = row?; - sequence_counts.insert(key, count); - } - - Ok(Self { - db, - sequence_counts: parking_lot::RwLock::new(sequence_counts), - time_patterns: parking_lot::RwLock::new(HashMap::new()), - max_history, - }) - } - - pub fn record(&self, file_id: FileId, context: AccessContext) -> Result<(), PatternError> { - let now = SystemTime::now(); - let timestamp = now.duration_since(UNIX_EPOCH).unwrap().as_secs() as i64; - let hour = (timestamp / 3600 % 24) as u8; - - // Persist to SQLite - self.db.execute( - "INSERT INTO access_log (file_id, access_time, hour_of_day) VALUES (?1, ?2, ?3)", - rusqlite::params![file_id.0, timestamp, hour], - )?; - - // Update time patterns (FR-19.3) - { - let mut time_patterns = self.time_patterns.write(); - time_patterns.entry(hour).or_default().push(file_id); - } - - // Get previous access for sequence tracking - let prev_file_id: Option = self.db.query_row( - "SELECT file_id FROM access_log WHERE id = (SELECT MAX(id) - 1 FROM access_log)", - [], - |row| row.get(0), - ).ok(); - - if let Some(prev_id) = prev_file_id { - let prev = FileId(prev_id); - - // Update in-memory - { - let mut sequences = self.sequence_counts.write(); - *sequences.entry((prev, file_id)).or_insert(0) += 1; - } - - // Persist sequence - self.db.execute( - "INSERT INTO sequence_counts (from_file_id, to_file_id, count) - VALUES (?1, ?2, 1) - ON CONFLICT(from_file_id, to_file_id) DO UPDATE SET count = count + 1", - rusqlite::params![prev_id, file_id.0], - )?; - } - - // Cleanup old entries - let cutoff = timestamp - (self.max_history as i64 * 86400); // max_history in days - self.db.execute("DELETE FROM access_log WHERE access_time < ?1", [cutoff])?; - - Ok(()) - } - - pub fn predict_next(&self, current: FileId, limit: usize) -> Vec { - let sequences = self.sequence_counts.read(); - - let mut predictions: Vec<_> = sequences - .iter() - .filter(|((from, _), count)| *from == current && **count >= 2) // Oracle fix: min threshold - .map(|((_, to), count)| (*to, *count)) - .collect(); - - predictions.sort_by(|a, b| b.1.cmp(&a.1)); - predictions.into_iter().take(limit).map(|(id, _)| id).collect() - } - - /// FR-19.3: Time-based prefetch - files commonly accessed at this hour - pub fn predict_for_time(&self, hour: u8, limit: usize) -> Vec { - let time_patterns = self.time_patterns.read(); - - time_patterns - .get(&hour) - .map(|files| files.iter().rev().take(limit).copied().collect()) - .unwrap_or_default() - } - - /// For RecentlyPlayed collection query - pub fn recently_played(&self, days: u32) -> Result, PatternError> { - let cutoff = SystemTime::now() - .duration_since(UNIX_EPOCH) - .unwrap() - .as_secs() as i64 - (days as i64 * 86400); - - let mut stmt = self.db.prepare( - "SELECT DISTINCT file_id FROM access_log WHERE access_time >= ?1 ORDER BY access_time DESC" - )?; - - let files: Vec = stmt - .query_map([cutoff], |row| Ok(FileId(row.get(0)?)))? - .filter_map(|r| r.ok()) - .collect(); - - Ok(files) - } - - /// For MostPlayed collection query - pub fn most_played(&self, limit: u32) -> Result, PatternError> { - let mut stmt = self.db.prepare( - "SELECT file_id, COUNT(*) as play_count FROM access_log - GROUP BY file_id ORDER BY play_count DESC LIMIT ?1" - )?; - - let files: Vec = stmt - .query_map([limit], |row| Ok(FileId(row.get(0)?)))? - .filter_map(|r| r.ok()) - .collect(); - - Ok(files) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum PatternError { - #[error("database error: {0}")] - Database(#[from] rusqlite::Error), -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - #[test] - fn test_pattern_prediction() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("patterns.db"); - let store = PatternStore::new(&db_path, 30).unwrap(); - let ctx = AccessContext { album_id: None, track_number: None, artist: None }; - - // Simulate: A -> B -> C pattern multiple times - for _ in 0..5 { - store.record(FileId(1), ctx.clone()).unwrap(); - store.record(FileId(2), ctx.clone()).unwrap(); - store.record(FileId(3), ctx.clone()).unwrap(); - } - - // After playing A, should predict B (needs >= 2 count) - let predictions = store.predict_next(FileId(1), 3); - assert!(!predictions.is_empty()); - assert_eq!(predictions[0], FileId(2)); - } - - #[test] - fn test_pattern_persistence() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("patterns.db"); - let ctx = AccessContext { album_id: None, track_number: None, artist: None }; - - // Record patterns - { - let store = PatternStore::new(&db_path, 30).unwrap(); - for _ in 0..3 { - store.record(FileId(1), ctx.clone()).unwrap(); - store.record(FileId(2), ctx.clone()).unwrap(); - } - } - - // Reopen and verify persistence - { - let store = PatternStore::new(&db_path, 30).unwrap(); - let predictions = store.predict_next(FileId(1), 3); - assert!(!predictions.is_empty()); - assert_eq!(predictions[0], FileId(2)); - } - } - - #[test] - fn test_recently_played() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("patterns.db"); - let store = PatternStore::new(&db_path, 30).unwrap(); - let ctx = AccessContext { album_id: None, track_number: None, artist: None }; - - store.record(FileId(100), ctx.clone()).unwrap(); - store.record(FileId(200), ctx.clone()).unwrap(); - - let recent = store.recently_played(7).unwrap(); - assert!(recent.contains(&FileId(100))); - assert!(recent.contains(&FileId(200))); - } - - #[test] - fn test_most_played() { - let dir = TempDir::new().unwrap(); - let db_path = dir.path().join("patterns.db"); - let store = PatternStore::new(&db_path, 30).unwrap(); - let ctx = AccessContext { album_id: None, track_number: None, artist: None }; - - // Play file 1 more times than file 2 - for _ in 0..5 { - store.record(FileId(1), ctx.clone()).unwrap(); - } - for _ in 0..2 { - store.record(FileId(2), ctx.clone()).unwrap(); - } - - let most = store.most_played(10).unwrap(); - assert_eq!(most[0], FileId(1)); // Most played first - } -} -``` - -### 3.2 Create `musicfs-cache/src/prefetch.rs` - -```rust -use crate::patterns::{AccessContext, PatternStore}; -use crate::CacheManager; -use musicfs_core::{Event, EventBus, FileId}; -use std::collections::HashSet; -use std::sync::Arc; -use tokio::sync::mpsc; -use tracing::{debug, info, warn}; - -pub struct PrefetchEngine { - patterns: Arc, - cache: Arc, - /// Oracle fix: Channel-based queue instead of polling - task_tx: mpsc::Sender, - task_rx: parking_lot::Mutex>>, - /// Oracle fix: Deduplication set to prevent duplicate prefetches - pending: parking_lot::RwLock>, - config: PrefetchConfig, -} - -#[derive(Debug, Clone)] -pub struct PrefetchConfig { - pub enabled: bool, - pub max_queue_size: usize, - pub lookahead: usize, - pub album_aware: bool, -} - -impl Default for PrefetchConfig { - fn default() -> Self { - Self { - enabled: true, - max_queue_size: 100, - lookahead: 3, - album_aware: true, - } - } -} - -#[derive(Debug)] -struct PrefetchTask { - file_id: FileId, - priority: u8, -} - -impl PrefetchEngine { - pub fn new(patterns: Arc, cache: Arc, config: PrefetchConfig) -> Self { - // Oracle fix: Use bounded channel instead of polling VecDeque - let (task_tx, task_rx) = mpsc::channel(config.max_queue_size); - - Self { - patterns, - cache, - task_tx, - task_rx: parking_lot::Mutex::new(Some(task_rx)), - pending: parking_lot::RwLock::new(HashSet::new()), - config, - } - } - - pub fn on_access(&self, file_id: FileId, context: AccessContext) { - if !self.config.enabled { - return; - } - - // Record pattern (now returns Result) - if let Err(e) = self.patterns.record(file_id, context.clone()) { - warn!("Failed to record pattern: {}", e); - } - - // Predict next files based on sequence patterns - let predictions = self.patterns.predict_next(file_id, self.config.lookahead); - - // FR-19.3: Time-based predictions - let hour = chrono::Local::now().hour() as u8; - let time_predictions = self.patterns.predict_for_time(hour, 2); - - // Album-aware: if we know track number, prefetch next tracks - let album_prefetch = if self.config.album_aware { - self.predict_album_next(&context) - } else { - vec![] - }; - - // Oracle fix: Deduplicate before queueing - let pending = self.pending.read(); - - for (i, pred) in predictions.into_iter().enumerate() { - if pending.contains(&pred) { - continue; // Already pending - } - let _ = self.task_tx.try_send(PrefetchTask { - file_id: pred, - priority: (10 - i as u8).min(10), - }); - } - - for pred in time_predictions { - if pending.contains(&pred) { - continue; - } - let _ = self.task_tx.try_send(PrefetchTask { - file_id: pred, - priority: 5, // Medium priority for time-based - }); - } - - for (i, pred) in album_prefetch.into_iter().enumerate() { - if pending.contains(&pred) { - continue; - } - let _ = self.task_tx.try_send(PrefetchTask { - file_id: pred, - priority: (8 - i as u8).min(8), - }); - } - - debug!("Prefetch pending count: {}", pending.len()); - } - - /// FR-19.4: Manual prefetch hint via /.prefetch/path - pub fn prefetch_hint(&self, file_id: FileId, priority: u8) { - let pending = self.pending.read(); - if pending.contains(&file_id) { - return; - } - drop(pending); - - let _ = self.task_tx.try_send(PrefetchTask { file_id, priority }); - } - - fn predict_album_next(&self, context: &AccessContext) -> Vec { - // In real implementation, would query cache for tracks in same album - // with track_number > current - vec![] - } - - /// Oracle fix: Event-driven loop instead of busy-wait polling - pub async fn run(&self) { - info!("Prefetch engine started"); - - // Take ownership of receiver - let mut task_rx = self.task_rx.lock().take() - .expect("run() called twice"); - - while let Some(task) = task_rx.recv().await { - // Mark as pending - { - let mut pending = self.pending.write(); - pending.insert(task.file_id); - } - - debug!("Prefetching {:?} (priority {})", task.file_id, task.priority); - - if let Err(e) = self.cache.prefetch(&task.file_id).await { - warn!("Prefetch failed for {:?}: {}", task.file_id, e); - } - - // Remove from pending - { - let mut pending = self.pending.write(); - pending.remove(&task.file_id); - } - } - - info!("Prefetch engine stopped"); - } - - pub fn start(self: Arc) -> PrefetchHandle { - let (stop_tx, mut stop_rx) = mpsc::channel::<()>(1); - let engine = self.clone(); - - tokio::spawn(async move { - tokio::select! { - _ = engine.run() => {} - _ = stop_rx.recv() => { - info!("Prefetch engine stopped"); - } - } - }); - - PrefetchHandle { stop_tx } - } - - pub fn pending_count(&self) -> usize { - self.pending.read().len() - } -} - -pub struct PrefetchHandle { - stop_tx: mpsc::Sender<()>, -} - -impl PrefetchHandle { - pub async fn stop(self) { - let _ = self.stop_tx.send(()).await; - } -} - -#[cfg(test)] -mod tests { - use super::*; - use tempfile::TempDir; - - #[test] - fn test_prefetch_config_default() { - let config = PrefetchConfig::default(); - assert!(config.enabled); - assert_eq!(config.lookahead, 3); - assert!(config.album_aware); - } - - #[tokio::test] - async fn test_prefetch_deduplication() { - let dir = TempDir::new().unwrap(); - let patterns = Arc::new(PatternStore::new(&dir.path().join("p.db"), 30).unwrap()); - let cache = Arc::new(MockCacheManager::new()); - let config = PrefetchConfig::default(); - - let engine = PrefetchEngine::new(patterns, cache, config); - - // Queue same file twice - engine.prefetch_hint(FileId(1), 10); - engine.prefetch_hint(FileId(1), 10); // Should be deduplicated - - // Only one should be pending - assert_eq!(engine.pending_count(), 0); // Not yet processed - } - - #[test] - fn test_prefetch_channel_based() { - // Verify no busy-wait polling - channel is used - let config = PrefetchConfig { max_queue_size: 50, ..Default::default() }; - // Channel capacity should match config - assert_eq!(config.max_queue_size, 50); - } -} -``` - ---- - ---- - -## Task 4: Prefetch Virtual Directory (FR-19.4) - -### 4.1 Create `musicfs-fuse/src/ops/prefetch.rs` - -```rust -use fuser::{FileType, ReplyDirectory, ReplyEntry, ReplyAttr}; -use musicfs_cache::prefetch::PrefetchEngine; -use musicfs_core::{FileId, VirtualPath}; -use std::sync::Arc; -use std::time::{Duration, SystemTime}; -use tracing::debug; - -const PREFETCH_DIR_INODE: u64 = 0xFFFF_FFFF_0000_0002; - -/// FR-19.4: Manual prefetch hints via /.prefetch/path -pub struct PrefetchOps { - prefetch_engine: Arc, -} - -impl PrefetchOps { - pub fn new(prefetch_engine: Arc) -> Self { - Self { prefetch_engine } - } - - pub fn is_prefetch_path(path: &str) -> bool { - path.starts_with("/.prefetch/") - } - - /// Lookup triggers prefetch for the target file - pub fn lookup(&self, path: &str, file_id: FileId, reply: ReplyEntry) { - debug!("Manual prefetch hint for: {}", path); - - // Queue prefetch with high priority (manual = important) - self.prefetch_engine.prefetch_hint(file_id, 15); - - // Return the original file's attributes - // (actual lookup delegated to main filesystem) - reply.error(libc::ENOENT); // Let main handler resolve - } - - pub fn readdir_prefetch_root(&self, reply: &mut ReplyDirectory) { - reply.add(PREFETCH_DIR_INODE, 1, FileType::Directory, "."); - reply.add(1, 2, FileType::Directory, ".."); - // Empty directory - entries are virtual - } - - pub fn getattr_prefetch_dir(&self, reply: ReplyAttr) { - let attr = fuser::FileAttr { - ino: PREFETCH_DIR_INODE, - size: 0, - blocks: 0, - atime: SystemTime::UNIX_EPOCH, - mtime: SystemTime::UNIX_EPOCH, - ctime: SystemTime::UNIX_EPOCH, - crtime: SystemTime::UNIX_EPOCH, - kind: FileType::Directory, - perm: 0o555, - nlink: 2, - uid: 1000, - gid: 1000, - rdev: 0, - blksize: 512, - flags: 0, - }; - reply.attr(&Duration::from_secs(60), &attr); - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_prefetch_path_detection() { - assert!(PrefetchOps::is_prefetch_path("/.prefetch/Artist/Album/Track.flac")); - assert!(!PrefetchOps::is_prefetch_path("/Artist/Album/Track.flac")); - } -} -``` - -### 4.2 FUSE Integration - -Add to `musicfs-fuse/src/filesystem.rs`: - -```rust -// In lookup() -if name == ".prefetch" && parent == 1 { - self.prefetch_ops.getattr_prefetch_dir(reply); - return; -} - -if let Some(path) = self.inode_to_path(parent) { - if PrefetchOps::is_prefetch_path(&path) { - // Strip /.prefetch/ prefix and lookup actual file - let actual_path = &path[10..]; // "/.prefetch/".len() - if let Some(file_id) = self.path_to_file_id(actual_path) { - self.prefetch_ops.lookup(&path, file_id, reply); - return; - } - } -} - -// In readdir() -if ino == PREFETCH_DIR_INODE { - self.prefetch_ops.readdir_prefetch_root(&mut reply); - reply.ok(); - return; -} -``` - ---- - -## Task 5: API Documentation - -**All APIs must be fully documented with happy and non-happy paths.** - -### 5.1 Create `docs/api/smart-features.md` - -```markdown -# Smart Features API Documentation - -## Overview - -Week 9 implements three smart feature categories: -1. **Smart Collections** - Query-based virtual folders -2. **Artwork** - Embedded album art extraction and caching -3. **Intelligent Prefetching** - Access pattern learning and prediction - ---- - -## 1. Smart Collections - -### Virtual Directory: `/.collections/{name}/` - -Browse query-based collections as virtual directories. - -### Happy Path - -``` -User FUSE - | | - |-- ls /.collections/ ----------->| - |<-- [Recently Added, 80s, Jazz]--| - | | - |-- ls /.collections/Jazz/ ------>| - | (executes: genre:Jazz) | - |<-- [symlinks to jazz tracks] ---| -``` - -### Built-in Collections - -| Name | Query | Description | -|------|-------|-------------| -| Recently Added | `RecentlyAdded { days: 30 }` | Files added in last 30 days | -| Recently Played | `RecentlyPlayed { days: 7 }` | Files played in last 7 days | -| Most Played | `MostPlayed { limit: 100 }` | Top 100 most played | -| 80s Music | `year:[1980 TO 1989]` | Year range filter | -| 90s Music | `year:[1990 TO 1999]` | Year range filter | - -### Collection Query Types - -```rust -enum CollectionQuery { - Match { field, pattern } // field:pattern - DateRange { field, start, end } // field:[start TO end] - RecentlyAdded { days } // Dynamic: mtime > now - days - RecentlyPlayed { days } // Dynamic: from access_log - MostPlayed { limit } // Dynamic: from access_log - Genre { genre } // genre:value - Compound { op, children } // AND/OR combinations -} -``` - -### Error Cases - -| Scenario | Behavior | FUSE Error | -|----------|----------|------------| -| Collection not found | ENOENT | `libc::ENOENT` | -| Invalid query syntax | Empty directory | (none) | -| Database error | EIO | `libc::EIO` | - -### SQLite Schema - -```sql -CREATE TABLE collections ( - id INTEGER PRIMARY KEY, - name TEXT UNIQUE NOT NULL, - query_json TEXT NOT NULL, - created_at INTEGER NOT NULL -); - --- For RecentlyPlayed/MostPlayed queries -CREATE TABLE access_log ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL, - access_time INTEGER NOT NULL, - hour_of_day INTEGER NOT NULL -); -``` - ---- - -## 2. Artwork API - -### Virtual File: `/Artist/Album/cover.jpg` - -Exposes embedded album art as virtual files. - -### Happy Path - -``` -User FUSE ArtworkCache - | | | - |-- open /A/B/cover.jpg --------->| | - | |-- get(file_id, "front")->| - | |<-- chunk_hash -----------| - | |-- CAS.get(hash) -------->| - | |<-- image bytes ----------| - |<-- image data ------------------| | -``` - -### Supported Sizes - -| Size | Max Dimension | Generated | -|------|---------------|-----------| -| `thumbnail` | 150x150 | On-demand | -| `medium` | 300x300 | On-demand | -| `full` | Original | Stored in CAS | - -### Accessing Different Sizes - -``` -/Artist/Album/cover.jpg # Full size (default) -/Artist/Album/cover_thumb.jpg # 150x150 thumbnail -/Artist/Album/cover_medium.jpg # 300x300 medium -``` - -### Error Cases - -| Scenario | Behavior | FUSE Error | -|----------|----------|------------| -| No embedded artwork | ENOENT | `libc::ENOENT` | -| Corrupted image data | ENOENT | `libc::ENOENT` | -| Image too large (>10MB) | Rejected during extraction | (logged) | -| CAS lookup failed | EIO | `libc::EIO` | -| Resize failed | Return full size | (fallback) | - -### SQLite Schema (Architecture 4.3.6) - -```sql -CREATE TABLE artwork ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL REFERENCES files(id), - art_type TEXT NOT NULL, -- 'front', 'back', 'other' - chunk_hash TEXT NOT NULL, -- Reference to CAS - width INTEGER NOT NULL, - height INTEGER NOT NULL, - UNIQUE(file_id, art_type) -); -``` - -**Note**: Only full-size artwork stored. Thumbnail/medium generated on-demand. - ---- - -## 3. Prefetch API - -### Automatic Prefetching - -Prefetch engine learns access patterns and pre-loads likely next files. - -### Pattern Learning Flow - -``` -User plays: Track 1 -> Track 2 -> Track 3 (repeated 5x) - -Pattern Store: - (Track 1 -> Track 2): count = 5 - (Track 2 -> Track 3): count = 5 - -Next time user plays Track 1: - -> Predict Track 2 (high confidence) - -> Queue prefetch for Track 2 -``` - -### FR-19.3: Time-Based Prefetching - -``` -User listens to "Morning Playlist" at 8am every weekday - -Pattern Store: - hour_of_day = 8 -> [track_ids from morning playlist] - -At 7:55am: - -> Predict morning tracks - -> Queue prefetch -``` - -### FR-19.4: Manual Prefetch Hints - -**Virtual Directory**: `/.prefetch/{path}` - -```bash -# Trigger prefetch for an album -ls /.prefetch/Artist/Album/ - -# Prefetch specific file -cat /.prefetch/Artist/Album/Track.flac > /dev/null -``` - -### Happy Path (Manual Prefetch) - -``` -User FUSE PrefetchEngine - | | | - |-- ls /.prefetch/A/B/ ---------->| | - | |-- prefetch_hint() -->| - | | file_id, priority=15 - | | |-- queue task - |<-- (directory listing) ---------| | - | | |-- async fetch -``` - -### Prefetch Priority Levels - -| Source | Priority | Description | -|--------|----------|-------------| -| Manual (/.prefetch/) | 15 | User-initiated, highest | -| Sequence prediction | 10-8 | Based on history patterns | -| Album sequential | 8-6 | Next tracks in album | -| Time-based | 5 | Hour-of-day patterns | - -### Error Cases - -| Scenario | Behavior | -|----------|----------| -| Already pending | Skipped (deduplication) | -| Queue full | try_send fails silently | -| Prefetch fails | Logged, removed from pending | -| Pattern DB error | Logged, prefetch continues | - -### Configuration - -```rust -struct PrefetchConfig { - enabled: bool, // Default: true - max_queue_size: usize, // Default: 100 - lookahead: usize, // Default: 3 tracks - album_aware: bool, // Default: true -} -``` - -### SQLite Schema - -```sql --- Access history for pattern learning -CREATE TABLE access_log ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL, - access_time INTEGER NOT NULL, - hour_of_day INTEGER NOT NULL -); - --- Sequence transition counts -CREATE TABLE sequence_counts ( - from_file_id INTEGER NOT NULL, - to_file_id INTEGER NOT NULL, - count INTEGER NOT NULL DEFAULT 1, - PRIMARY KEY (from_file_id, to_file_id) -); -``` - ---- - -## Performance Targets - -| Metric | Target | Notes | -|--------|--------|-------| -| Cache hit rate (warm) | >95% | FR-16.3 | -| Prefetch accuracy | >50% | Measured as: prefetched files actually accessed | -| Artwork resize latency | <100ms | For thumbnail/medium | -| Pattern prediction latency | <10ms | In-memory lookup | - ---- - -## Integration Examples - -### Creating a Smart Collection - -```rust -let mut store = CollectionStore::new(&db_path)?; - -// Create custom collection -let jazz_80s = store.create( - "80s Jazz", - CollectionQuery::Compound { - op: BoolOp::And, - children: vec![ - CollectionQuery::Genre { genre: "Jazz".into() }, - CollectionQuery::DateRange { - field: "year".into(), - start: 1980, - end: 1989, - }, - ], - }, -)?; - -// List collections -let collections = store.list()?; -``` - -### Accessing Album Art - -```rust -let cache = ArtworkCache::new(cas_store, &db_path)?; - -// Get full-size artwork -let full = cache.get(file_id, "front", ArtSize::Full).await?; - -// Get thumbnail (generated on-demand) -let thumb = cache.get(file_id, "front", ArtSize::Thumbnail).await?; -``` - -### Manual Prefetch via CLI - -```bash -# Prefetch entire album before listening -find /mnt/musicfs/.prefetch/Metallica/BlackAlbum/ -type f | head -n 1 - -# Check prefetch status -musicfs-cli prefetch status -# Output: 3 files pending, 12 completed in last hour -``` -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_collection_crud` | Unit | Create/list/delete collections (FR-15.2) | -| `test_compound_query` | Unit | AND/OR queries work | -| `test_builtin_collections` | Unit | Recently Added, 80s/90s exist | -| `test_recently_played_query` | Unit | RecentlyPlayed from access_log | -| `test_most_played_query` | Unit | MostPlayed from access_log | -| `test_artwork_extraction` | Unit | Extract from FLAC/MP3 (FR-16.1) | -| `test_artwork_resize` | Unit | Thumbnail/medium generation (FR-16.4) | -| `test_artwork_resize_on_demand` | Unit | Full stored, sizes generated | -| `test_artwork_reject_oversized` | Unit | >10MB images rejected | -| `test_artwork_cache` | Unit | Store/retrieve from CAS (FR-16.3) | -| `test_pattern_prediction` | Unit | A->B->C pattern learned (FR-19.1) | -| `test_pattern_persistence` | Unit | Patterns survive restart | -| `test_time_based_prediction` | Unit | Hour-of-day patterns (FR-19.3) | -| `test_prefetch_deduplication` | Unit | Same file not queued twice | -| `test_prefetch_channel` | Unit | Channel-based, no polling | -| `test_prefetch_manual_hint` | Unit | /.prefetch/ handler (FR-19.4) | -| `test_collection_virtual_dir` | E2E | `/.collections/Jazz/` works | -| `test_cover_virtual_file` | E2E | `/Artist/Album/cover.jpg` exists (FR-16.2) | -| `test_prefetch_virtual_dir` | E2E | `/.prefetch/path` triggers prefetch | -| `test_prefetch_reduces_misses` | Integration | >50% miss reduction | - ---- - -## Exit Criteria - -- [ ] Smart collections stored in SQLite -- [ ] Built-in collections (Recently Added, Recently Played, Most Played, 80s, 90s) available -- [ ] `/.collections/Name/` shows matching files -- [ ] RecentlyPlayed/MostPlayed queries use persisted access_log table -- [ ] Album art extracted from embedded FLAC/MP3 data -- [ ] Artwork schema matches architecture.md 4.3.6 exactly (no size column) -- [ ] Thumbnail/medium generated on-demand, only full stored in CAS -- [ ] Oversized images (>10MB) rejected gracefully -- [ ] `cover.jpg` appears in album directories -- [ ] Access patterns recorded in SQLite (survive restarts) -- [ ] Time-based prefetch predicts by hour-of-day (FR-19.3) -- [ ] `/.prefetch/path` triggers manual prefetch hints (FR-19.4) -- [ ] Prefetch engine uses channel-based queue (no busy-wait polling) -- [ ] Prefetch deduplication prevents same file queued twice -- [ ] Prefetch reduces cache misses by >50% on sequential album playback -- [ ] API documentation covers happy/error paths for all features - ---- - -## Architecture Compliance - -| Architecture Section | Requirement | Status | -|---------------------|-------------|--------| -| 4.3.6 | collections table schema | ✅ | -| 4.3.6 | artwork table schema (UNIQUE file_id, art_type) | ✅ Oracle fix | -| 3.2.5 | Cache hit rate >95% | ✅ Benchmark | -| FR-15.1 | Query-based virtual folders | ✅ | -| FR-15.2 | Saved searches as directories | ✅ | -| FR-15.3 | Dynamic playlists (RecentlyPlayed, MostPlayed) | ✅ access_log | -| FR-16.1 | Extract embedded album art | ✅ | -| FR-16.2 | Expose as virtual files | ✅ | -| FR-16.3 | Cache separately from audio | ✅ | -| FR-16.4 | Multiple sizes | ✅ On-demand | -| FR-19.1 | Learn access patterns | ✅ Persistent | -| FR-19.2 | Playlist-aware prefetch | ✅ | -| FR-19.3 | Time-based prefetching | ✅ Task 4 | -| FR-19.4 | Manual prefetch hints | ✅ /.prefetch/ | - -## Oracle Fixes Applied - -| Issue | Fix | Location | -|-------|-----|----------| -| Artwork schema mismatch | Removed `size` column, matches architecture exactly | `artwork.rs` | -| rusqlite in async context | Use `spawn_blocking` for DB operations | `artwork.rs` | -| PatternStore not persisted | Added `access_log` and `sequence_counts` tables | `patterns.rs` | -| FR-19.3 missing | Added time-based prediction by hour | `patterns.rs` | -| FR-19.4 missing | Added `/.prefetch/` FUSE handler | `prefetch.rs` | -| Prefetch busy-wait polling | Switched to `mpsc::channel` | `prefetch.rs` | -| No prefetch deduplication | Added `pending: HashSet` guard | `prefetch.rs` | -| Image resize memory spikes | Added 10MB max input size check | `artwork.rs` | diff --git a/docs/v2/plans/week-10-plugin-system.md b/docs/v2/plans/week-10-plugin-system.md deleted file mode 100644 index 38cbdff..0000000 --- a/docs/v2/plans/week-10-plugin-system.md +++ /dev/null @@ -1,179 +0,0 @@ -# Week 10: Plugin System - -**Phase**: 4 - Plugin System & Polish -**Goal**: Extensibility via native and WASM plugins -**Requirements**: FR-23.1-23.5, FR-24.1-24.3 - ---- - -## Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Plugin traits | musicfs-plugins | `traits.rs` | FR-23.1-23.4 | -| Native host | musicfs-plugins | `native.rs` | FR-23.2 | -| WASM host | musicfs-plugins | `wasm.rs` | FR-23.3 | -| Plugin lifecycle | musicfs-plugins | `manager.rs` | FR-23.5 | -| Example plugins | plugins/ | `example-origin/`, `example-format/` | FR-23.5 | - ---- - -## Plugin Traits (`musicfs-plugins/src/traits.rs`) - -```rust -/// Base plugin interface -pub trait Plugin: Send + Sync { - fn name(&self) -> &str; - fn version(&self) -> Version; - fn init(&mut self, config: Value) -> Result<(), PluginError>; - fn shutdown(&mut self) -> Result<(), PluginError>; -} - -/// Origin plugin interface (per architecture 4.3.4) -pub trait OriginPlugin: Plugin { - fn origin_type(&self) -> &str; - fn create(&self, config: Value) -> Result, PluginError>; -} - -/// Metadata source plugin -pub trait MetadataPlugin: Plugin { - fn lookup(&self, query: &MetadataQuery) -> Result, PluginError>; -} - -/// Format plugin for custom audio formats (FR-24.1) -pub trait FormatPlugin: Plugin { - fn extensions(&self) -> &[&str]; - fn can_handle(&self, extension: &str) -> bool; - fn parse(&self, reader: &mut dyn Read) -> Result; -} -``` - ---- - -## Native Plugin Host (`musicfs-plugins/src/native.rs`) - -```rust -pub struct NativePluginHost { - plugins: HashMap, - search_paths: Vec, -} - -struct LoadedPlugin { - library: libloading::Library, - instance: Box, -} - -impl NativePluginHost { - pub fn new() -> Self; - - /// Load plugin from shared library (.so/.dylib) - pub fn load(&mut self, path: &Path) -> Result; - - /// Unload plugin (FR-23.5) - pub fn unload(&mut self, id: PluginId) -> Result<(), PluginError>; - - /// Hot reload plugin without restart (FR-23.4) - pub fn reload(&mut self, id: PluginId) -> Result<(), PluginError>; - - /// List loaded plugins - pub fn list(&self) -> Vec; -} -``` - ---- - -## WASM Plugin Host (`musicfs-plugins/src/wasm.rs`) - -```rust -pub struct WasmPluginHost { - engine: wasmtime::Engine, - linker: wasmtime::Linker, -} - -impl WasmPluginHost { - pub fn new() -> Result; - - /// Load WASM plugin with sandboxing (FR-23.3) - pub fn load(&mut self, wasm_bytes: &[u8]) -> Result; - - /// Resource limits for sandboxed execution - pub fn set_limits(&mut self, limits: ResourceLimits); -} - -pub struct ResourceLimits { - pub max_memory_mb: u32, - pub max_cpu_time_ms: u32, - pub allow_network: bool, - pub allow_filesystem: bool, -} -``` - ---- - -## Plugin Manager (`musicfs-plugins/src/manager.rs`) - -```rust -pub struct PluginManager { - native_host: NativePluginHost, - wasm_host: WasmPluginHost, - registry: PluginRegistry, -} - -impl PluginManager { - /// Initialize and load plugins from config - pub fn init(config: &PluginConfig) -> Result; - - /// Get all origin plugins - pub fn origin_plugins(&self) -> Vec<&dyn OriginPlugin>; - - /// Get all format plugins - pub fn format_plugins(&self) -> Vec<&dyn FormatPlugin>; - - /// Get all metadata plugins - pub fn metadata_plugins(&self) -> Vec<&dyn MetadataPlugin>; - - /// Reload all plugins (hot reload) - pub fn reload_all(&mut self) -> Result<(), PluginError>; -} -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_native_plugin_load` | Unit | Native plugin loading (FR-23.2) | -| `test_native_plugin_unload` | Unit | Clean unload | -| `test_wasm_plugin_sandbox` | Unit | WASM isolation (FR-23.3) | -| `test_wasm_resource_limits` | Unit | Memory/CPU limits enforced | -| `test_plugin_hot_reload` | Integration | Reload without restart (FR-23.4) | -| `test_example_origin_plugin` | Integration | Custom origin works | -| `test_example_format_plugin` | Integration | Custom format works | - ---- - -## Exit Criteria - -- [ ] Native plugins loadable at runtime -- [ ] WASM plugins sandboxed with resource limits -- [ ] Example plugins functional -- [ ] Plugins hot-reloadable without daemon restart -- [ ] Plugin lifecycle management (load, unload, reload) - ---- - -## Architecture Alignment - -Per architecture.md section 4.3.4: -- Plugin loading: Built-in → Native (.so) → WASM -- Origin plugins create `Box` -- Format plugins register file extensions -- WASM runs in wasmtime sandbox - -Per requirements.md: -- FR-23.1: Loadable plugins ✓ -- FR-23.2: Stable plugin API ✓ -- FR-23.3: Plugins for origins, metadata, formats ✓ -- FR-23.4: WASM sandbox ✓ -- FR-23.5: Plugin lifecycle ✓ diff --git a/docs/v2/plans/week-11-control-api.md b/docs/v2/plans/week-11-control-api.md deleted file mode 100644 index e2bb0fc..0000000 --- a/docs/v2/plans/week-11-control-api.md +++ /dev/null @@ -1,539 +0,0 @@ -# Week 11: Control API & Production - -**Phase**: 4 - Plugin System & Polish -**Goal**: gRPC control API, metrics, and production readiness -**Requirements**: FR-17.1-17.5, FR-18.1-18.4, NFR-6.1-6.4, NFR-10.1-10.4 - ---- - -## Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| gRPC server | musicfs-grpc | `server.rs` | FR-17.1-17.5 | -| Proto codegen | proto/ | `musicfs.proto`, `build.rs` | FR-17.2 | -| Event streaming | musicfs-grpc | `events.rs` | FR-18.1-18.3 | -| Webhook handler | musicfs-grpc | `webhook.rs` | FR-18.2 | -| Metrics export | musicfs-core | `metrics.rs` | NFR-6.1-6.4, NFR-10.2-10.4 | -| CLI completion | musicfs-cli | `main.rs` | FR-17 | -| systemd unit | dist/ | `musicfs.service` | Production | -| Packaging | dist/ | `PKGBUILD`, `musicfs.spec` | Production | -| E2E compatibility | tests/ | `e2e_players.rs` | NFR-12.1-12.3 | - ---- - -## Proto Definitions (`proto/musicfs.proto`) - -Per architecture.md section 4.3.7, implement full gRPC API: - -```protobuf -syntax = "proto3"; -package musicfs.v1; - -service MusicFS { - // Daemon lifecycle - rpc GetStatus(Empty) returns (StatusResponse); - rpc Shutdown(ShutdownRequest) returns (Empty); - - // Cache management - rpc GetCacheStats(Empty) returns (CacheStats); - rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse); - rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress); - - // Origin management - rpc ListOrigins(Empty) returns (OriginsResponse); - rpc GetOriginHealth(OriginRequest) returns (OriginHealth); - rpc RescanOrigin(OriginRequest) returns (stream SyncProgress); - - // Search (already implemented in Week 8) - rpc Search(SearchRequest) returns (SearchResponse); - rpc SearchStream(SearchRequest) returns (stream SearchResult); - - // Events (server-streaming) - rpc SubscribeEvents(EventFilter) returns (stream Event); -} -``` - -Full message definitions in architecture.md section 4.3.7. - ---- - -## gRPC Server (`musicfs-grpc/src/server.rs`) - -```rust -pub struct MusicFsService { - core: Arc, - events: broadcast::Sender, - metrics: Arc, -} - -#[tonic::async_trait] -impl musicfs::v1::music_fs_server::MusicFs for MusicFsService { - // Daemon lifecycle - async fn get_status(&self, _: Request) -> Result, Status>; - async fn shutdown(&self, req: Request) -> Result, Status>; - - // Cache management - async fn get_cache_stats(&self, _: Request) -> Result, Status>; - async fn clear_cache(&self, req: Request) -> Result, Status>; - - type PrefetchStream = ReceiverStream>; - async fn prefetch(&self, req: Request) -> Result, Status>; - - // Origin management - async fn list_origins(&self, _: Request) -> Result, Status>; - async fn get_origin_health(&self, req: Request) -> Result, Status>; - - type RescanOriginStream = ReceiverStream>; - async fn rescan_origin(&self, req: Request) -> Result, Status>; - - // Events - type SubscribeEventsStream = ReceiverStream>; - async fn subscribe_events(&self, req: Request) -> Result, Status>; -} -``` - ---- - -## Event Streaming (`musicfs-grpc/src/events.rs`) - -```rust -pub struct EventStreamer { - bus: Arc, -} - -impl EventStreamer { - /// Convert internal events to gRPC Event messages - pub fn subscribe(&self, filter: EventFilter) -> impl Stream; - - /// Filter events by type and origin - fn matches(event: &Event, filter: &EventFilter) -> bool; -} -``` - ---- - -## Webhook Handler (`musicfs-grpc/src/webhook.rs`) - -HTTP webhook notifications for external integrations (FR-18.2): - -```rust -use reqwest::Client; -use serde::Serialize; -use tokio::sync::broadcast; - -#[derive(Debug, Clone, Serialize)] -pub struct WebhookPayload { - pub event_type: String, - pub timestamp: i64, - pub data: serde_json::Value, -} - -pub struct WebhookConfig { - pub url: String, - pub secret: Option, - pub events: Vec, // Filter: ["file_accessed", "sync_completed", ...] - pub retry_count: u32, - pub timeout_ms: u64, -} - -pub struct WebhookHandler { - client: Client, - configs: Vec, -} - -impl WebhookHandler { - pub fn new(configs: Vec) -> Self; - - /// Start listening to event bus and dispatch webhooks - pub async fn run(&self, mut rx: broadcast::Receiver) { - while let Ok(event) = rx.recv().await { - for config in &self.configs { - if self.matches_filter(&event, config) { - self.dispatch(config, &event).await; - } - } - } - } - - /// Dispatch webhook with retry logic - async fn dispatch(&self, config: &WebhookConfig, event: &Event) { - let payload = WebhookPayload { - event_type: event.event_type(), - timestamp: event.timestamp(), - data: event.to_json(), - }; - - let mut attempts = 0; - loop { - let result = self.client - .post(&config.url) - .timeout(Duration::from_millis(config.timeout_ms)) - .header("X-MusicFS-Signature", self.sign(&payload, config)) - .json(&payload) - .send() - .await; - - match result { - Ok(resp) if resp.status().is_success() => break, - _ if attempts < config.retry_count => { - attempts += 1; - tokio::time::sleep(Duration::from_millis(100 * 2u64.pow(attempts))).await; - } - _ => { - tracing::warn!("Webhook delivery failed after {} attempts", attempts); - break; - } - } - } - } - - /// HMAC-SHA256 signature if secret configured - fn sign(&self, payload: &WebhookPayload, config: &WebhookConfig) -> String; - - fn matches_filter(&self, event: &Event, config: &WebhookConfig) -> bool; -} -``` - -Configuration in `config.toml`: - -```toml -[[webhooks]] -url = "https://example.com/musicfs/events" -secret = "your-webhook-secret" -events = ["file_accessed", "sync_completed", "origin_health_changed"] -retry_count = 3 -timeout_ms = 5000 -``` - ---- - -## E2E Compatibility Tests (`tests/e2e_players.rs`) - -Verify MusicFS works with common media players (NFR-12.1-12.3): - -```rust -//! E2E tests for media player compatibility -//! Requires: mpv, vlc, file manager (nautilus/dolphin) installed - -use std::process::Command; -use std::time::Duration; - -/// Test mpv can play files from MusicFS (NFR-12.1) -#[test] -#[ignore] // Run manually: cargo test --ignored -fn test_mpv_playback() { - let mountpoint = setup_test_mount(); - - // mpv should be able to: - // 1. Open file without hanging - // 2. Read metadata (duration, format) - // 3. Play first few seconds - // 4. Seek forward - // 5. Exit cleanly - - let output = Command::new("mpv") - .args([ - "--no-video", - "--no-audio", // Silent playback - "--length=2", // Play 2 seconds only - "--msg-level=all=debug", - &format!("{}/Artist/Album/01 - Track.flac", mountpoint), - ]) - .output() - .expect("mpv must be installed"); - - assert!(output.status.success(), "mpv playback failed: {:?}", output); -} - -/// Test VLC can browse and play (NFR-12.2) -#[test] -#[ignore] -fn test_vlc_playback() { - let mountpoint = setup_test_mount(); - - // VLC should handle: - // 1. Directory browsing - // 2. Playlist creation from folder - // 3. Metadata display - // 4. Gapless playback (if supported) - - let output = Command::new("cvlc") // Command-line VLC - .args([ - "--play-and-exit", - "--run-time=2", - &format!("{}/Artist/Album/", mountpoint), - ]) - .output() - .expect("vlc must be installed"); - - assert!(output.status.success(), "VLC playback failed"); -} - -/// Test file manager operations (NFR-12.3) -#[test] -#[ignore] -fn test_file_manager_operations() { - let mountpoint = setup_test_mount(); - - // File managers should be able to: - // 1. List directories without timeout - // 2. Show file previews/thumbnails - // 3. Display file properties - // 4. Copy files to local disk - - // Test basic stat operations that file managers use - let entries: Vec<_> = std::fs::read_dir(&mountpoint) - .expect("read_dir failed") - .collect(); - - assert!(!entries.is_empty(), "mountpoint should have entries"); - - // Test stat on each entry (file managers do this for icons) - for entry in entries { - let entry = entry.expect("entry should be valid"); - let metadata = entry.metadata().expect("metadata should work"); - assert!(metadata.is_dir() || metadata.is_file()); - } -} - -/// Test concurrent access from multiple players -#[test] -#[ignore] -fn test_concurrent_player_access() { - let mountpoint = setup_test_mount(); - - // Spawn multiple players accessing different files - let handles: Vec<_> = (0..3) - .map(|i| { - let mp = mountpoint.clone(); - std::thread::spawn(move || { - Command::new("mpv") - .args([ - "--no-video", "--no-audio", "--length=1", - &format!("{}/Artist/Album/0{} - Track.flac", mp, i + 1), - ]) - .output() - }) - }) - .collect(); - - for handle in handles { - let output = handle.join().unwrap().expect("mpv should run"); - assert!(output.status.success()); - } -} - -fn setup_test_mount() -> String { - // Returns path to test mount with sample files - std::env::var("MUSICFS_TEST_MOUNT") - .unwrap_or_else(|_| "/tmp/musicfs-test".to_string()) -} -``` - ---- - -## Metrics (`musicfs-core/src/metrics.rs`) - -Per architecture.md section 5.2: - -```rust -use prometheus::{IntCounterVec, HistogramVec, IntGauge, register_*}; - -lazy_static! { - pub static ref FUSE_OPS: IntCounterVec = register_int_counter_vec!( - "musicfs_fuse_ops_total", - "Total FUSE operations", - &["op"] - ).unwrap(); - - pub static ref FUSE_LATENCY: HistogramVec = register_histogram_vec!( - "musicfs_fuse_latency_seconds", - "FUSE operation latency", - &["op"], - vec![0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0] - ).unwrap(); - - pub static ref CACHE_HITS: IntCounter = register_int_counter!( - "musicfs_cache_hits_total", - "Cache hits" - ).unwrap(); - - pub static ref CACHE_MISSES: IntCounter = register_int_counter!( - "musicfs_cache_misses_total", - "Cache misses" - ).unwrap(); - - pub static ref CACHE_SIZE_BYTES: IntGauge = register_int_gauge!( - "musicfs_cache_size_bytes", - "Current cache size in bytes" - ).unwrap(); - - pub static ref ORIGIN_HEALTH: IntGaugeVec = register_int_gauge_vec!( - "musicfs_origin_health", - "Origin health status (1=healthy, 0=unhealthy)", - &["origin"] - ).unwrap(); -} - -/// Expose metrics on HTTP endpoint -pub async fn serve_metrics(addr: SocketAddr) -> Result<(), MetricsError>; -``` - ---- - -## CLI Commands (`musicfs-cli/src/main.rs`) - -```rust -#[derive(Parser)] -enum Command { - /// Mount filesystem - Mount { - #[arg(short, long)] - config: PathBuf, - mountpoint: PathBuf, - }, - - /// Get daemon status - Status, - - /// Cache management - Cache { - #[command(subcommand)] - command: CacheCommand, - }, - - /// Search library - Search { - query: String, - #[arg(short, long, default_value = "100")] - limit: u32, - }, - - /// Origin management - Origin { - #[command(subcommand)] - command: OriginCommand, - }, - - /// Subscribe to events - Events { - #[arg(short, long)] - r#type: Option, - }, -} - -#[derive(Subcommand)] -enum CacheCommand { - Stats, - Clear { origin: Option }, - Prefetch { paths: Vec }, -} - -#[derive(Subcommand)] -enum OriginCommand { - List, - Health { origin_id: String }, - Rescan { origin_id: String }, -} -``` - ---- - -## systemd Service (`dist/musicfs.service`) - -```ini -[Unit] -Description=MusicFS - Metadata-Organized Music Filesystem -After=network.target - -[Service] -Type=notify -ExecStart=/usr/bin/musicfs mount --config /etc/musicfs/config.toml /mnt/music -ExecStop=/usr/bin/musicfs shutdown -Restart=on-failure -RestartSec=5 -User=musicfs -Group=musicfs - -# Security hardening -NoNewPrivileges=true -ProtectSystem=strict -ProtectHome=read-only -ReadWritePaths=/var/cache/musicfs /mnt/music -PrivateTmp=true - -[Install] -WantedBy=multi-user.target -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_grpc_status` | Unit | GetStatus RPC (FR-17.1) | -| `test_grpc_cache_stats` | Unit | GetCacheStats RPC | -| `test_grpc_cache_clear` | Unit | ClearCache RPC (FR-17.3) | -| `test_grpc_origins_list` | Unit | ListOrigins RPC | -| `test_grpc_origin_rescan` | Integration | RescanOrigin streaming | -| `test_grpc_events_stream` | Integration | Event streaming (FR-18.1) | -| `test_grpc_prefetch_stream` | Integration | Prefetch progress | -| `test_webhook_dispatch` | Unit | Webhook delivery (FR-18.2) | -| `test_webhook_retry` | Unit | Webhook retry on failure | -| `test_webhook_hmac_signature` | Unit | HMAC-SHA256 signing | -| `test_metrics_prometheus` | Unit | Prometheus format (NFR-6.1) | -| `test_metrics_http_endpoint` | Integration | HTTP metrics endpoint | -| `test_cli_commands` | Integration | CLI works | -| `test_systemd_service` | E2E | Service lifecycle | -| `test_mpv_playback` | E2E | mpv compatibility (NFR-12.1) | -| `test_vlc_playback` | E2E | VLC compatibility (NFR-12.2) | -| `test_file_manager_operations` | E2E | File manager browsing (NFR-12.3) | -| `test_concurrent_player_access` | E2E | Multiple players concurrently | - ---- - -## Exit Criteria - -- [ ] gRPC API fully functional (all RPCs from architecture.md 4.3.7) -- [ ] Event streaming works with filtering -- [ ] Webhook notifications delivered with HMAC signing -- [ ] Prometheus metrics exported on HTTP endpoint -- [ ] CLI feature-complete with all commands -- [ ] systemd service works (start, stop, restart) -- [ ] mpv, VLC playback verified (E2E tests) -- [ ] File manager browsing verified -- [ ] All acceptance tests pass - ---- - -## Architecture Alignment - -Per architecture.md section 4.3.7: -- gRPC over Unix socket ✓ -- Protocol Buffers for type safety ✓ -- Server-streaming for events, sync progress, prefetch ✓ -- CLI wraps gRPC client ✓ - -Per architecture.md section 5.2: -- Prometheus metrics format ✓ -- Golden signals: latency, traffic, errors, saturation ✓ - -Per requirements.md: -- FR-17.1: Unix socket control ✓ -- FR-17.2: gRPC with Protocol Buffers ✓ -- FR-17.3: Cache management commands ✓ -- FR-17.4: Runtime configuration ✓ -- FR-17.5: Graceful shutdown ✓ -- FR-18.1: File access events ✓ -- FR-18.2: Webhook notifications ✓ (HTTP webhooks with HMAC) -- FR-18.3: Event streaming ✓ -- FR-18.4: Access pattern logging ✓ -- NFR-10.1: Configurable logging ✓ -- NFR-10.2: Metrics exposure ✓ -- NFR-10.3: Health check ✓ -- NFR-10.4: Prometheus integration ✓ -- NFR-12.1: mpv compatibility ✓ (E2E tests) -- NFR-12.2: VLC compatibility ✓ (E2E tests) -- NFR-12.3: File manager compatibility ✓ (E2E tests) diff --git a/docs/v2/plans/week-12-external-metadata.md b/docs/v2/plans/week-12-external-metadata.md deleted file mode 100644 index 6d309ca..0000000 --- a/docs/v2/plans/week-12-external-metadata.md +++ /dev/null @@ -1,624 +0,0 @@ -# Week 12: External Metadata Integration - -**Phase**: 5 - P1 Feature Completion -**Goal**: Integrate external metadata sources for automatic tagging and artwork -**Requirements**: FR-21.1-21.4, FR-16.5 - ---- - -## Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| MusicBrainz client | musicfs-external | `musicbrainz.rs` | FR-21.1 | -| Discogs client | musicfs-external | `discogs.rs` | FR-21.2 | -| Last.fm client | musicfs-external | `lastfm.rs` | FR-21.3 | -| AcoustID/Chromaprint | musicfs-external | `acoustid.rs` | FR-21.4 | -| Online artwork fetch | musicfs-external | `artwork_fetch.rs` | FR-16.5 | -| Metadata enrichment | musicfs-external | `enrichment.rs` | All | -| Plugin integration | musicfs-plugins | `metadata_plugin.rs` | FR-21.5 | - ---- - -## Task 1: Create `musicfs-external` Crate - -### 1.1 `Cargo.toml` - -```toml -[package] -name = "musicfs-external" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -reqwest = { version = "0.11", features = ["json"] } -serde = { workspace = true, features = ["derive"] } -serde_json.workspace = true -tokio.workspace = true -tracing.workspace = true -thiserror.workspace = true -chromaprint = "0.6" # Audio fingerprinting -base64 = "0.21" - -[dev-dependencies] -wiremock = "0.5" # Mock HTTP responses -tokio-test = "0.4" -``` - -### 1.2 `src/lib.rs` - -```rust -pub mod musicbrainz; -pub mod discogs; -pub mod lastfm; -pub mod acoustid; -pub mod artwork_fetch; -pub mod enrichment; - -pub use enrichment::MetadataEnricher; -``` - ---- - -## Task 2: MusicBrainz Client (`musicfs-external/src/musicbrainz.rs`) - -```rust -use serde::Deserialize; - -const MB_API: &str = "https://musicbrainz.org/ws/2"; -const USER_AGENT: &str = "MusicFS/0.1.0 (https://github.com/user/musicfs)"; - -#[derive(Debug, Deserialize)] -pub struct MbRecording { - pub id: String, - pub title: String, - pub length: Option, - #[serde(rename = "artist-credit")] - pub artist_credit: Vec, - pub releases: Option>, -} - -#[derive(Debug, Deserialize)] -pub struct MbRelease { - pub id: String, - pub title: String, - pub date: Option, - #[serde(rename = "release-group")] - pub release_group: Option, -} - -#[derive(Debug, Deserialize)] -pub struct MbReleaseGroup { - pub id: String, - #[serde(rename = "primary-type")] - pub primary_type: Option, -} - -#[derive(Debug, Deserialize)] -pub struct ArtistCredit { - pub artist: MbArtist, -} - -#[derive(Debug, Deserialize)] -pub struct MbArtist { - pub id: String, - pub name: String, - #[serde(rename = "sort-name")] - pub sort_name: String, -} - -pub struct MusicBrainzClient { - client: reqwest::Client, - rate_limiter: RateLimiter, // 1 req/sec per MB guidelines -} - -impl MusicBrainzClient { - pub fn new() -> Self { - let client = reqwest::Client::builder() - .user_agent(USER_AGENT) - .build() - .expect("client build"); - - Self { - client, - rate_limiter: RateLimiter::new(Duration::from_secs(1)), - } - } - - /// Search by recording title + artist (FR-21.1) - pub async fn search_recording( - &self, - title: &str, - artist: Option<&str>, - ) -> Result, ExternalError> { - self.rate_limiter.wait().await; - - let mut query = format!("recording:{}", title); - if let Some(artist) = artist { - query.push_str(&format!(" AND artist:{}", artist)); - } - - let resp = self.client - .get(format!("{}/recording", MB_API)) - .query(&[ - ("query", query.as_str()), - ("fmt", "json"), - ("limit", "5"), - ]) - .send() - .await?; - - let body: SearchResponse = resp.json().await?; - Ok(body.recordings) - } - - /// Get release artwork from Cover Art Archive - pub async fn get_cover_art(&self, release_id: &str) -> Result>, ExternalError> { - let url = format!("https://coverartarchive.org/release/{}/front-500", release_id); - - let resp = self.client.get(&url).send().await?; - if resp.status() == 404 { - return Ok(None); - } - - let bytes = resp.bytes().await?; - Ok(Some(bytes.to_vec())) - } - - /// Lookup recording by MusicBrainz ID - pub async fn get_recording(&self, mbid: &str) -> Result { - self.rate_limiter.wait().await; - - let resp = self.client - .get(format!("{}/recording/{}", MB_API, mbid)) - .query(&[ - ("inc", "artist-credits+releases+release-groups"), - ("fmt", "json"), - ]) - .send() - .await?; - - Ok(resp.json().await?) - } -} - -struct RateLimiter { - interval: Duration, - last_request: Mutex, -} - -impl RateLimiter { - fn new(interval: Duration) -> Self { - Self { - interval, - last_request: Mutex::new(Instant::now() - interval), - } - } - - async fn wait(&self) { - let mut last = self.last_request.lock().await; - let elapsed = last.elapsed(); - if elapsed < self.interval { - tokio::time::sleep(self.interval - elapsed).await; - } - *last = Instant::now(); - } -} -``` - ---- - -## Task 3: Discogs Client (`musicfs-external/src/discogs.rs`) - -```rust -const DISCOGS_API: &str = "https://api.discogs.com"; - -pub struct DiscogsClient { - client: reqwest::Client, - token: Option, - rate_limiter: RateLimiter, // 60 req/min authenticated -} - -impl DiscogsClient { - pub fn new(token: Option) -> Self; - - /// Search releases (FR-21.2) - pub async fn search( - &self, - query: &str, - artist: Option<&str>, - ) -> Result, ExternalError>; - - /// Get master release details - pub async fn get_master(&self, id: u64) -> Result; - - /// Get release images - pub async fn get_images(&self, release_id: u64) -> Result, ExternalError>; -} - -#[derive(Debug, Deserialize)] -pub struct DiscogsRelease { - pub id: u64, - pub title: String, - pub year: Option, - pub thumb: Option, - pub master_id: Option, -} - -#[derive(Debug, Deserialize)] -pub struct DiscogsImage { - pub uri: String, - pub width: u32, - pub height: u32, - #[serde(rename = "type")] - pub image_type: String, // "primary" or "secondary" -} -``` - ---- - -## Task 4: Last.fm Client (`musicfs-external/src/lastfm.rs`) - -```rust -const LASTFM_API: &str = "https://ws.audioscrobbler.com/2.0"; - -pub struct LastFmClient { - client: reqwest::Client, - api_key: String, -} - -impl LastFmClient { - pub fn new(api_key: String) -> Self; - - /// Get track info with play counts, tags (FR-21.3) - pub async fn get_track_info( - &self, - track: &str, - artist: &str, - ) -> Result; - - /// Get album info with artwork - pub async fn get_album_info( - &self, - album: &str, - artist: &str, - ) -> Result; - - /// Get artist info - pub async fn get_artist_info(&self, artist: &str) -> Result; -} - -#[derive(Debug, Deserialize)] -pub struct LastFmTrack { - pub name: String, - pub playcount: Option, - pub listeners: Option, - pub duration: Option, - pub toptags: Option, - pub album: Option, -} - -#[derive(Debug, Deserialize)] -pub struct LastFmAlbum { - pub name: String, - pub artist: String, - pub image: Vec, - pub tracks: Option, -} - -#[derive(Debug, Deserialize)] -pub struct LastFmImage { - #[serde(rename = "#text")] - pub url: String, - pub size: String, // "small", "medium", "large", "extralarge", "mega" -} -``` - ---- - -## Task 5: AcoustID/Chromaprint (`musicfs-external/src/acoustid.rs`) - -```rust -use chromaprint::{Fingerprinter, Configuration}; - -const ACOUSTID_API: &str = "https://api.acoustid.org/v2/lookup"; - -pub struct AcoustIdClient { - client: reqwest::Client, - api_key: String, -} - -impl AcoustIdClient { - pub fn new(api_key: String) -> Self; - - /// Generate fingerprint from audio data (FR-21.4) - pub fn fingerprint(&self, samples: &[i16], sample_rate: u32) -> Result { - let config = Configuration::preset_test1(); - let mut fp = Fingerprinter::new(&config); - - fp.start(sample_rate, 1)?; // mono - fp.feed(samples)?; - fp.finish()?; - - Ok(fp.fingerprint().to_string()) - } - - /// Lookup fingerprint on AcoustID database - pub async fn lookup( - &self, - fingerprint: &str, - duration: u32, - ) -> Result, ExternalError> { - let resp = self.client - .get(ACOUSTID_API) - .query(&[ - ("client", self.api_key.as_str()), - ("fingerprint", fingerprint), - ("duration", &duration.to_string()), - ("meta", "recordings+releasegroups"), - ]) - .send() - .await?; - - let body: AcoustIdResponse = resp.json().await?; - Ok(body.results) - } -} - -#[derive(Debug, Deserialize)] -pub struct AcoustIdResult { - pub id: String, - pub score: f32, - pub recordings: Option>, -} - -#[derive(Debug, Deserialize)] -pub struct AcoustIdRecording { - pub id: String, // MusicBrainz recording ID - pub title: Option, - pub artists: Option>, -} -``` - ---- - -## Task 6: Online Artwork Fetch (`musicfs-external/src/artwork_fetch.rs`) - -```rust -pub struct ArtworkFetcher { - musicbrainz: MusicBrainzClient, - discogs: Option, - lastfm: Option, -} - -impl ArtworkFetcher { - /// Fetch missing artwork from online sources (FR-16.5) - /// Tries sources in order: MusicBrainz Cover Art Archive → Discogs → Last.fm - pub async fn fetch_artwork( - &self, - artist: &str, - album: &str, - size: ArtworkSize, - ) -> Result, ExternalError> { - // 1. Try MusicBrainz release search → Cover Art Archive - if let Some(art) = self.try_musicbrainz(artist, album, size).await? { - return Ok(Some(art)); - } - - // 2. Try Discogs - if let Some(discogs) = &self.discogs { - if let Some(art) = self.try_discogs(discogs, artist, album, size).await? { - return Ok(Some(art)); - } - } - - // 3. Try Last.fm - if let Some(lastfm) = &self.lastfm { - if let Some(art) = self.try_lastfm(lastfm, artist, album, size).await? { - return Ok(Some(art)); - } - } - - Ok(None) - } - - async fn try_musicbrainz( - &self, - artist: &str, - album: &str, - size: ArtworkSize, - ) -> Result, ExternalError> { - // Search for release, get cover art from Cover Art Archive - let releases = self.musicbrainz.search_release(album, Some(artist)).await?; - - for release in releases.iter().take(3) { - if let Some(art) = self.musicbrainz.get_cover_art(&release.id).await? { - return Ok(Some(ArtworkData { - data: art, - source: ArtworkSource::MusicBrainz, - mime_type: "image/jpeg".to_string(), - })); - } - } - - Ok(None) - } -} - -#[derive(Debug)] -pub struct ArtworkData { - pub data: Vec, - pub source: ArtworkSource, - pub mime_type: String, -} - -#[derive(Debug)] -pub enum ArtworkSource { - MusicBrainz, - Discogs, - LastFm, - Embedded, -} - -pub enum ArtworkSize { - Small, // 150px - Medium, // 300px - Large, // 500px - Original, -} -``` - ---- - -## Task 7: Metadata Enrichment (`musicfs-external/src/enrichment.rs`) - -```rust -pub struct MetadataEnricher { - musicbrainz: MusicBrainzClient, - acoustid: Option, - artwork_fetcher: ArtworkFetcher, -} - -impl MetadataEnricher { - /// Enrich metadata from external sources - pub async fn enrich(&self, meta: &AudioMeta) -> Result { - let mut enriched = EnrichedMetadata::from(meta); - - // If we have title + artist, search MusicBrainz - if let (Some(title), Some(artist)) = (&meta.title, &meta.artist) { - let recordings = self.musicbrainz.search_recording(title, Some(artist)).await?; - - if let Some(best) = recordings.first() { - enriched.musicbrainz_recording_id = Some(best.id.clone()); - - // Enrich with release info - if let Some(releases) = &best.releases { - if let Some(release) = releases.first() { - enriched.musicbrainz_release_id = Some(release.id.clone()); - } - } - } - } - - Ok(enriched) - } - - /// Identify unknown track by audio fingerprint - pub async fn identify_by_fingerprint( - &self, - samples: &[i16], - sample_rate: u32, - duration: u32, - ) -> Result, ExternalError> { - let acoustid = self.acoustid.as_ref() - .ok_or(ExternalError::ServiceNotConfigured("AcoustID"))?; - - let fingerprint = acoustid.fingerprint(samples, sample_rate)?; - let results = acoustid.lookup(&fingerprint, duration).await?; - - // Return best match above threshold - results.into_iter() - .filter(|r| r.score > 0.8) - .flat_map(|r| r.recordings) - .flatten() - .next() - .map(|rec| IdentifiedTrack { - title: rec.title, - musicbrainz_id: Some(rec.id), - artists: rec.artists.map(|a| a.into_iter().map(|x| x.name).collect()), - }) - .pipe(Ok) - } -} - -#[derive(Debug)] -pub struct EnrichedMetadata { - pub original: AudioMeta, - pub musicbrainz_recording_id: Option, - pub musicbrainz_release_id: Option, - pub musicbrainz_artist_id: Option, - pub genres: Vec, - pub play_count: Option, -} - -#[derive(Debug)] -pub struct IdentifiedTrack { - pub title: Option, - pub musicbrainz_id: Option, - pub artists: Option>, -} -``` - ---- - -## Configuration - -```toml -[external] -# MusicBrainz (no auth required, rate limited to 1 req/sec) -musicbrainz.enabled = true - -# Discogs (optional, requires token for higher rate limits) -discogs.enabled = true -discogs.token = "your_discogs_token" - -# Last.fm (requires API key) -lastfm.enabled = true -lastfm.api_key = "your_lastfm_api_key" - -# AcoustID (requires API key) -acoustid.enabled = true -acoustid.api_key = "your_acoustid_api_key" - -# Artwork fetching behavior -artwork.fetch_missing = true -artwork.cache_fetched = true -artwork.preferred_size = "large" # small, medium, large, original -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_musicbrainz_search` | Integration | Recording search (FR-21.1) | -| `test_musicbrainz_cover_art` | Integration | Cover Art Archive | -| `test_discogs_search` | Integration | Release search (FR-21.2) | -| `test_lastfm_track_info` | Integration | Track metadata (FR-21.3) | -| `test_acoustid_fingerprint` | Unit | Chromaprint generation | -| `test_acoustid_lookup` | Integration | Fingerprint lookup (FR-21.4) | -| `test_artwork_fetch_cascade` | Integration | Multi-source artwork (FR-16.5) | -| `test_metadata_enrichment` | Integration | Full enrichment flow | -| `test_rate_limiting` | Unit | Rate limiter works | -| `test_mock_responses` | Unit | Offline testing with mocks | - ---- - -## Exit Criteria - -- [ ] MusicBrainz search returns relevant recordings -- [ ] Cover Art Archive artwork downloads work -- [ ] Discogs integration retrieves release info -- [ ] Last.fm integration retrieves track/artist info -- [ ] AcoustID fingerprinting identifies tracks -- [ ] Artwork fetcher tries all sources in cascade -- [ ] Metadata enricher adds external IDs -- [ ] Rate limiting prevents API abuse -- [ ] All tests pass with mock HTTP responses - ---- - -## Architecture Alignment - -Per requirements.md: -- FR-21.1: MusicBrainz for canonical metadata ✓ -- FR-21.2: Discogs for release info, artwork ✓ -- FR-21.3: Last.fm for play counts, tags ✓ -- FR-21.4: AcoustID for audio fingerprinting ✓ -- FR-16.5: Fetch missing artwork from online ✓ - -Per architecture.md section 4.3.4: -- External metadata via `MetadataPlugin` trait ✓ -- Plugin architecture allows adding more sources ✓ diff --git a/docs/v2/plans/week-13-import-export.md b/docs/v2/plans/week-13-import-export.md deleted file mode 100644 index 63a6e28..0000000 --- a/docs/v2/plans/week-13-import-export.md +++ /dev/null @@ -1,699 +0,0 @@ -# Week 13: Import & Export - -**Phase**: 5 - P1 Feature Completion -**Goal**: Import metadata from existing library managers, export library data -**Requirements**: FR-22.1-22.3 - ---- - -## Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Beets database import | musicfs-import | `beets.rs` | FR-22.1 | -| iTunes/Apple Music import | musicfs-import | `itunes.rs` | FR-22.2 | -| Library export | musicfs-import | `export.rs` | FR-22.3 | -| Import CLI | musicfs-cli | `import.rs` | All | - ---- - -## Task 1: Create `musicfs-import` Crate - -### 1.1 `Cargo.toml` - -```toml -[package] -name = "musicfs-import" -version.workspace = true -edition.workspace = true - -[dependencies] -musicfs-core = { path = "../musicfs-core" } -musicfs-cache = { path = "../musicfs-cache" } -rusqlite = { workspace = true, features = ["bundled"] } -serde = { workspace = true, features = ["derive"] } -serde_json.workspace = true -plist = "1.5" # For iTunes XML parsing -tokio.workspace = true -tracing.workspace = true -thiserror.workspace = true -csv = "1.3" -url = "2.4" -percent-encoding = "2.3" -chrono = { version = "0.4", features = ["serde"] } - -[dev-dependencies] -tempfile.workspace = true -``` - -### 1.2 `src/lib.rs` - -```rust -pub mod beets; -pub mod itunes; -pub mod export; - -use musicfs_core::Result; - -/// Common import result -#[derive(Debug, Default)] -pub struct ImportResult { - pub imported: usize, - pub skipped: usize, - pub errors: Vec, -} - -#[derive(Debug)] -pub struct ImportError { - pub path: String, - pub reason: String, -} - -/// Import progress callback -pub type ProgressCallback = Box; - -#[derive(Debug, Clone)] -pub struct ImportProgress { - pub current: usize, - pub total: usize, - pub current_file: String, -} -``` - ---- - -## Task 2: Beets Database Import (`musicfs-import/src/beets.rs`) - -```rust -use rusqlite::{Connection, params}; -use std::path::Path; - -/// Beets database schema (simplified) -/// Full schema: https://beets.readthedocs.io/en/stable/dev/db.html -#[derive(Debug)] -pub struct BeetsItem { - pub id: i64, - pub path: String, - pub title: Option, - pub artist: Option, - pub album: Option, - pub album_artist: Option, - pub genre: Option, - pub year: Option, - pub track: Option, - pub disc: Option, - pub length: Option, - pub bitrate: Option, - pub sample_rate: Option, - pub format: Option, - pub mb_trackid: Option, - pub mb_albumid: Option, - pub mb_artistid: Option, - pub mtime: f64, -} - -pub struct BeetsImporter { - beets_db: Connection, - target_db: Arc, -} - -impl BeetsImporter { - /// Open beets database for import (FR-22.1) - pub fn new(beets_db_path: &Path, target_db: Arc) -> Result { - let conn = Connection::open_with_flags( - beets_db_path, - rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, - )?; - - // Verify this is a beets database - let tables: Vec = conn - .prepare("SELECT name FROM sqlite_master WHERE type='table'")? - .query_map([], |row| row.get(0))? - .filter_map(|r| r.ok()) - .collect(); - - if !tables.contains(&"items".to_string()) { - return Err(ImportError::InvalidDatabase("Not a beets database")); - } - - Ok(Self { - beets_db: conn, - target_db, - }) - } - - /// Count items to import - pub fn count_items(&self) -> Result { - self.beets_db - .query_row("SELECT COUNT(*) FROM items", [], |row| row.get(0)) - .map_err(Into::into) - } - - /// Import all items with progress callback - pub fn import_all(&self, progress: Option) -> Result { - let total = self.count_items()?; - let mut result = ImportResult::default(); - - let mut stmt = self.beets_db.prepare(r#" - SELECT id, path, title, artist, album, albumartist, genre, - year, track, disc, length, bitrate, samplerate, format, - mb_trackid, mb_albumid, mb_artistid, mtime - FROM items - "#)?; - - let items = stmt.query_map([], |row| { - Ok(BeetsItem { - id: row.get(0)?, - path: row.get(1)?, - title: row.get(2)?, - artist: row.get(3)?, - album: row.get(4)?, - album_artist: row.get(5)?, - genre: row.get(6)?, - year: row.get(7)?, - track: row.get(8)?, - disc: row.get(9)?, - length: row.get(10)?, - bitrate: row.get(11)?, - sample_rate: row.get(12)?, - format: row.get(13)?, - mb_trackid: row.get(14)?, - mb_albumid: row.get(15)?, - mb_artistid: row.get(16)?, - mtime: row.get(17)?, - }) - })?; - - for (idx, item) in items.enumerate() { - match item { - Ok(item) => { - if let Some(ref cb) = progress { - cb(ImportProgress { - current: idx + 1, - total, - current_file: item.path.clone(), - }); - } - - match self.import_item(&item) { - Ok(_) => result.imported += 1, - Err(e) => { - result.errors.push(ImportError { - path: item.path, - reason: e.to_string(), - }); - } - } - } - Err(e) => { - result.skipped += 1; - result.errors.push(ImportError { - path: format!("item_{}", idx), - reason: e.to_string(), - }); - } - } - } - - Ok(result) - } - - fn import_item(&self, item: &BeetsItem) -> Result<(), ImportError> { - let path = Path::new(&item.path); - - // Convert to our AudioMeta - let audio_meta = AudioMeta { - title: item.title.clone(), - artist: item.artist.clone(), - album: item.album.clone(), - album_artist: item.album_artist.clone(), - genre: item.genre.clone(), - year: item.year.map(|y| y as u32), - track: item.track.map(|t| t as u32), - disc: item.disc.map(|d| d as u32), - duration_ms: item.length.map(|l| (l * 1000.0) as u64), - bitrate: item.bitrate.map(|b| b as u32), - sample_rate: item.sample_rate.map(|s| s as u32), - format: AudioFormat::from_extension( - path.extension().and_then(|e| e.to_str()).unwrap_or("") - ), - ..Default::default() - }; - - // Generate virtual path using our resolver - let virtual_path = VirtualPath::from_metadata(&audio_meta, path); - - // Import to our database - self.target_db.upsert_file( - &OriginId::from("beets-import"), - path, - &virtual_path, - &audio_meta, - std::time::UNIX_EPOCH + std::time::Duration::from_secs_f64(item.mtime), - std::fs::metadata(path).map(|m| m.len()).unwrap_or(0), - )?; - - Ok(()) - } -} -``` - ---- - -## Task 3: iTunes/Apple Music Import (`musicfs-import/src/itunes.rs`) - -```rust -use plist::Value; -use std::collections::HashMap; -use url::Url; - -/// iTunes Library XML format -#[derive(Debug)] -pub struct ItunesTrack { - pub track_id: u64, - pub name: Option, - pub artist: Option, - pub album: Option, - pub album_artist: Option, - pub genre: Option, - pub year: Option, - pub track_number: Option, - pub disc_number: Option, - pub total_time: Option, // milliseconds - pub bit_rate: Option, - pub sample_rate: Option, - pub location: Option, // file:// URL - pub date_added: Option, -} - -pub struct ItunesImporter { - tracks: Vec, - target_db: Arc, -} - -impl ItunesImporter { - /// Parse iTunes Library.xml (FR-22.2) - pub fn from_xml(xml_path: &Path, target_db: Arc) -> Result { - let file = std::fs::File::open(xml_path)?; - let plist: Value = plist::from_reader(file)?; - - let dict = plist.as_dictionary() - .ok_or(ImportError::InvalidFormat("Expected dictionary at root"))?; - - let tracks_dict = dict.get("Tracks") - .and_then(|v| v.as_dictionary()) - .ok_or(ImportError::InvalidFormat("Missing Tracks dictionary"))?; - - let mut tracks = Vec::new(); - - for (_, track_value) in tracks_dict { - if let Some(track_dict) = track_value.as_dictionary() { - tracks.push(Self::parse_track(track_dict)?); - } - } - - Ok(Self { tracks, target_db }) - } - - fn parse_track(dict: &plist::Dictionary) -> Result { - Ok(ItunesTrack { - track_id: dict.get("Track ID") - .and_then(|v| v.as_unsigned_integer()) - .unwrap_or(0), - name: dict.get("Name").and_then(|v| v.as_string()).map(String::from), - artist: dict.get("Artist").and_then(|v| v.as_string()).map(String::from), - album: dict.get("Album").and_then(|v| v.as_string()).map(String::from), - album_artist: dict.get("Album Artist").and_then(|v| v.as_string()).map(String::from), - genre: dict.get("Genre").and_then(|v| v.as_string()).map(String::from), - year: dict.get("Year").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32), - track_number: dict.get("Track Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32), - disc_number: dict.get("Disc Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32), - total_time: dict.get("Total Time").and_then(|v| v.as_unsigned_integer()), - bit_rate: dict.get("Bit Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32), - sample_rate: dict.get("Sample Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32), - location: dict.get("Location").and_then(|v| v.as_string()).map(String::from), - date_added: dict.get("Date Added").and_then(|v| v.as_string()).map(String::from), - }) - } - - /// Convert file:// URL to path - fn url_to_path(url_str: &str) -> Option { - Url::parse(url_str).ok() - .filter(|u| u.scheme() == "file") - .and_then(|u| u.to_file_path().ok()) - } - - pub fn count_tracks(&self) -> usize { - self.tracks.len() - } - - /// Import all tracks - pub fn import_all(&self, progress: Option) -> Result { - let total = self.tracks.len(); - let mut result = ImportResult::default(); - - for (idx, track) in self.tracks.iter().enumerate() { - if let Some(ref cb) = progress { - cb(ImportProgress { - current: idx + 1, - total, - current_file: track.name.clone().unwrap_or_default(), - }); - } - - // Skip tracks without location - let Some(ref location) = track.location else { - result.skipped += 1; - continue; - }; - - let Some(path) = Self::url_to_path(location) else { - result.skipped += 1; - result.errors.push(ImportError { - path: location.clone(), - reason: "Invalid file URL".to_string(), - }); - continue; - }; - - match self.import_track(track, &path) { - Ok(_) => result.imported += 1, - Err(e) => { - result.errors.push(ImportError { - path: path.display().to_string(), - reason: e.to_string(), - }); - } - } - } - - Ok(result) - } - - fn import_track(&self, track: &ItunesTrack, path: &Path) -> Result<(), ImportError> { - let audio_meta = AudioMeta { - title: track.name.clone(), - artist: track.artist.clone(), - album: track.album.clone(), - album_artist: track.album_artist.clone(), - genre: track.genre.clone(), - year: track.year, - track: track.track_number, - disc: track.disc_number, - duration_ms: track.total_time, - bitrate: track.bit_rate, - sample_rate: track.sample_rate, - format: AudioFormat::from_extension( - path.extension().and_then(|e| e.to_str()).unwrap_or("") - ), - ..Default::default() - }; - - let virtual_path = VirtualPath::from_metadata(&audio_meta, path); - - let mtime = std::fs::metadata(path) - .map(|m| m.modified().unwrap_or(std::time::UNIX_EPOCH)) - .unwrap_or(std::time::UNIX_EPOCH); - - let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0); - - self.target_db.upsert_file( - &OriginId::from("itunes-import"), - path, - &virtual_path, - &audio_meta, - mtime, - size, - )?; - - Ok(()) - } -} -``` - ---- - -## Task 4: Library Export (`musicfs-import/src/export.rs`) - -```rust -use csv::Writer; -use serde::Serialize; - -#[derive(Debug, Serialize)] -pub struct ExportedTrack { - pub virtual_path: String, - pub real_path: String, - pub title: String, - pub artist: String, - pub album: String, - pub album_artist: String, - pub genre: String, - pub year: Option, - pub track: Option, - pub disc: Option, - pub duration_ms: Option, - pub format: String, - pub musicbrainz_id: Option, -} - -pub struct LibraryExporter { - db: Arc, -} - -impl LibraryExporter { - pub fn new(db: Arc) -> Self { - Self { db } - } - - /// Export library to CSV (FR-22.3) - pub fn export_csv(&self, output: &Path) -> Result { - let files = self.db.list_all_files()?; - let mut writer = Writer::from_path(output)?; - - let mut count = 0; - for file in files { - let audio = file.audio.as_ref(); - - writer.serialize(ExportedTrack { - virtual_path: file.virtual_path.as_str().to_string(), - real_path: file.real_path.path.display().to_string(), - title: audio.and_then(|a| a.title.clone()).unwrap_or_default(), - artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(), - album: audio.and_then(|a| a.album.clone()).unwrap_or_default(), - album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(), - genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(), - year: audio.and_then(|a| a.year), - track: audio.and_then(|a| a.track), - disc: audio.and_then(|a| a.disc), - duration_ms: audio.and_then(|a| a.duration_ms), - format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(), - musicbrainz_id: None, // TODO: Include if enriched - })?; - - count += 1; - } - - writer.flush()?; - Ok(count) - } - - /// Export library to JSON - pub fn export_json(&self, output: &Path) -> Result { - let files = self.db.list_all_files()?; - - let tracks: Vec = files.iter() - .map(|file| { - let audio = file.audio.as_ref(); - ExportedTrack { - virtual_path: file.virtual_path.as_str().to_string(), - real_path: file.real_path.path.display().to_string(), - title: audio.and_then(|a| a.title.clone()).unwrap_or_default(), - artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(), - album: audio.and_then(|a| a.album.clone()).unwrap_or_default(), - album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(), - genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(), - year: audio.and_then(|a| a.year), - track: audio.and_then(|a| a.track), - disc: audio.and_then(|a| a.disc), - duration_ms: audio.and_then(|a| a.duration_ms), - format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(), - musicbrainz_id: None, - } - }) - .collect(); - - let json = serde_json::to_string_pretty(&tracks)?; - std::fs::write(output, json)?; - - Ok(tracks.len()) - } - - /// Export to M3U playlist format - pub fn export_m3u(&self, output: &Path, base_path: Option<&Path>) -> Result { - let files = self.db.list_all_files()?; - - let mut content = String::from("#EXTM3U\n"); - - for file in &files { - let duration = file.audio.as_ref() - .and_then(|a| a.duration_ms) - .map(|d| d / 1000) - .unwrap_or(0); - - let title = file.audio.as_ref() - .and_then(|a| a.title.clone()) - .unwrap_or_else(|| file.virtual_path.as_str().to_string()); - - let artist = file.audio.as_ref() - .and_then(|a| a.artist.clone()) - .unwrap_or_default(); - - content.push_str(&format!( - "#EXTINF:{},{} - {}\n", - duration, artist, title - )); - - // Use virtual path relative to base, or absolute real path - let path = if let Some(base) = base_path { - base.join(file.virtual_path.as_str().trim_start_matches('/')) - .display().to_string() - } else { - file.real_path.path.display().to_string() - }; - - content.push_str(&path); - content.push('\n'); - } - - std::fs::write(output, content)?; - Ok(files.len()) - } -} -``` - ---- - -## Task 5: Import CLI Commands (`musicfs-cli/src/import.rs`) - -```rust -#[derive(Subcommand)] -pub enum ImportCommand { - /// Import from beets database - Beets { - /// Path to beets library.db - #[arg(short, long)] - db: PathBuf, - }, - - /// Import from iTunes Library.xml - Itunes { - /// Path to iTunes Library.xml - #[arg(short, long)] - xml: PathBuf, - }, - - /// Export library - Export { - /// Output file path - #[arg(short, long)] - output: PathBuf, - - /// Format: csv, json, m3u - #[arg(short, long, default_value = "csv")] - format: String, - }, -} - -pub async fn handle_import(cmd: ImportCommand, db: Arc) -> Result<()> { - match cmd { - ImportCommand::Beets { db: beets_path } => { - println!("Importing from beets database: {:?}", beets_path); - - let importer = BeetsImporter::new(&beets_path, db)?; - let total = importer.count_items()?; - println!("Found {} items to import", total); - - let pb = ProgressBar::new(total as u64); - let result = importer.import_all(Some(Box::new(move |p| { - pb.set_position(p.current as u64); - })))?; - - println!("\nImport complete:"); - println!(" Imported: {}", result.imported); - println!(" Skipped: {}", result.skipped); - println!(" Errors: {}", result.errors.len()); - } - - ImportCommand::Itunes { xml } => { - println!("Importing from iTunes Library: {:?}", xml); - - let importer = ItunesImporter::from_xml(&xml, db)?; - let total = importer.count_tracks(); - println!("Found {} tracks to import", total); - - let pb = ProgressBar::new(total as u64); - let result = importer.import_all(Some(Box::new(move |p| { - pb.set_position(p.current as u64); - })))?; - - println!("\nImport complete:"); - println!(" Imported: {}", result.imported); - println!(" Skipped: {}", result.skipped); - println!(" Errors: {}", result.errors.len()); - } - - ImportCommand::Export { output, format } => { - let exporter = LibraryExporter::new(db); - - let count = match format.as_str() { - "csv" => exporter.export_csv(&output)?, - "json" => exporter.export_json(&output)?, - "m3u" => exporter.export_m3u(&output, None)?, - _ => return Err(anyhow::anyhow!("Unknown format: {}", format)), - }; - - println!("Exported {} tracks to {:?}", count, output); - } - } - - Ok(()) -} -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_beets_import_valid` | Integration | Beets database parsing (FR-22.1) | -| `test_beets_import_missing_fields` | Unit | Handle incomplete metadata | -| `test_itunes_xml_parsing` | Unit | iTunes XML parsing (FR-22.2) | -| `test_itunes_url_to_path` | Unit | file:// URL conversion | -| `test_itunes_import_tracks` | Integration | Full iTunes import | -| `test_export_csv` | Unit | CSV export (FR-22.3) | -| `test_export_json` | Unit | JSON export | -| `test_export_m3u` | Unit | M3U playlist export | -| `test_import_preserves_musicbrainz_ids` | Integration | External IDs preserved | -| `test_import_deduplication` | Integration | No duplicates on re-import | - ---- - -## Exit Criteria - -- [ ] Beets database import works with real beets.db -- [ ] iTunes Library.xml import parses all tracks -- [ ] CSV/JSON/M3U export generates valid files -- [ ] Progress reporting works during import -- [ ] Errors are reported without crashing -- [ ] Import is idempotent (re-import updates, doesn't duplicate) -- [ ] MusicBrainz IDs from beets are preserved - ---- - -## Architecture Alignment - -Per requirements.md: -- FR-22.1: Import from beets database ✓ -- FR-22.2: Import from iTunes/Apple Music ✓ -- FR-22.3: Export library metadata ✓ diff --git a/docs/v2/plans/week-14-extended-formats.md b/docs/v2/plans/week-14-extended-formats.md deleted file mode 100644 index 21d12c0..0000000 --- a/docs/v2/plans/week-14-extended-formats.md +++ /dev/null @@ -1,633 +0,0 @@ -# Week 14: Extended Formats & Audio Fingerprinting - -**Phase**: 5 - P1 Feature Completion -**Goal**: Audio fingerprint search and audiobook format support -**Requirements**: FR-14.4, FR-24.2 - ---- - -## Deliverables - -| Task | Crate | Files | Requirements | -|------|-------|-------|--------------| -| Fingerprint indexing | musicfs-search | `fingerprint.rs` | FR-14.4 | -| Fingerprint search | musicfs-search | `fingerprint_search.rs` | FR-14.4 | -| M4B audiobook support | musicfs-metadata | `formats/m4b.rs` | FR-24.2 | -| Chapter extraction | musicfs-metadata | `chapters.rs` | FR-24.2 | -| Virtual chapter files | musicfs-fuse | `ops/chapters.rs` | FR-24.2 | - ---- - -## Task 1: Audio Fingerprint Generation - -### 1.1 Add Dependencies - -```toml -# In musicfs-search/Cargo.toml -[dependencies] -chromaprint = "0.6" -symphonia = { version = "0.5", features = ["all"] } -``` - -### 1.2 Fingerprint Generation (`musicfs-search/src/fingerprint.rs`) - -```rust -use chromaprint::{Configuration, Fingerprinter}; -use symphonia::core::audio::SampleBuffer; -use symphonia::core::codecs::DecoderOptions; -use std::path::Path; - -/// Audio fingerprint using Chromaprint algorithm -#[derive(Debug, Clone)] -pub struct AudioFingerprint { - pub raw: Vec, - pub duration_secs: u32, -} - -impl AudioFingerprint { - /// Generate fingerprint from audio file (FR-14.4) - pub fn from_file(path: &Path) -> Result { - let file = std::fs::File::open(path)?; - let mss = MediaSourceStream::new(Box::new(file), Default::default()); - - let probed = symphonia::default::get_probe() - .format(&Hint::new(), mss, &FormatOptions::default(), &MetadataOptions::default())?; - - let mut format = probed.format; - let track = format.tracks() - .iter() - .find(|t| t.codec_params.codec != CODEC_TYPE_NULL) - .ok_or(FingerprintError::NoAudioTrack)?; - - let sample_rate = track.codec_params.sample_rate - .ok_or(FingerprintError::NoSampleRate)?; - - let mut decoder = symphonia::default::get_codecs() - .make(&track.codec_params, &DecoderOptions::default())?; - - // Chromaprint configuration - let config = Configuration::preset_test1(); - let mut fingerprinter = Fingerprinter::new(&config); - fingerprinter.start(sample_rate, 1)?; // Mono - - let mut samples: Vec = Vec::new(); - let mut duration_samples = 0u64; - - // Decode and collect samples (first 120 seconds max) - let max_samples = sample_rate as u64 * 120; - - loop { - match format.next_packet() { - Ok(packet) => { - let decoded = decoder.decode(&packet)?; - let mut sample_buf = SampleBuffer::::new( - decoded.capacity() as u64, - *decoded.spec(), - ); - sample_buf.copy_interleaved_ref(decoded); - - // Convert to mono if stereo - let mono: Vec = if decoded.spec().channels.count() > 1 { - sample_buf.samples() - .chunks(decoded.spec().channels.count()) - .map(|chunk| (chunk.iter().map(|&s| s as i32).sum::() / chunk.len() as i32) as i16) - .collect() - } else { - sample_buf.samples().to_vec() - }; - - samples.extend(&mono); - duration_samples += mono.len() as u64; - - if duration_samples >= max_samples { - break; - } - } - Err(symphonia::core::errors::Error::IoError(e)) - if e.kind() == std::io::ErrorKind::UnexpectedEof => break, - Err(e) => return Err(e.into()), - } - } - - // Feed samples to fingerprinter - fingerprinter.feed(&samples)?; - fingerprinter.finish()?; - - let raw = fingerprinter.fingerprint().to_vec(); - let duration_secs = (duration_samples / sample_rate as u64) as u32; - - Ok(Self { raw, duration_secs }) - } - - /// Compress fingerprint for storage - pub fn to_bytes(&self) -> Vec { - // Use chromaprint's compressed format - chromaprint::encode_fingerprint(&self.raw, chromaprint::Algorithm::Test1) - } - - /// Decompress fingerprint - pub fn from_bytes(bytes: &[u8]) -> Result { - let (raw, _) = chromaprint::decode_fingerprint(bytes)?; - Ok(Self { raw, duration_secs: 0 }) - } -} - -#[derive(Debug, thiserror::Error)] -pub enum FingerprintError { - #[error("No audio track found")] - NoAudioTrack, - #[error("No sample rate")] - NoSampleRate, - #[error("IO error: {0}")] - Io(#[from] std::io::Error), - #[error("Decode error: {0}")] - Decode(String), - #[error("Chromaprint error: {0}")] - Chromaprint(String), -} -``` - ---- - -## Task 2: Fingerprint Search (`musicfs-search/src/fingerprint_search.rs`) - -```rust -use crate::fingerprint::AudioFingerprint; - -/// Fingerprint similarity search using bit-level comparison -pub struct FingerprintIndex { - db: Arc, -} - -impl FingerprintIndex { - pub fn new(db: Arc) -> Self { - Self { db } - } - - /// Index a file's fingerprint - pub fn index(&self, file_id: FileId, fingerprint: &AudioFingerprint) -> Result<(), SearchError> { - let bytes = fingerprint.to_bytes(); - self.db.store_fingerprint(file_id, &bytes, fingerprint.duration_secs)?; - Ok(()) - } - - /// Search by fingerprint similarity (FR-14.4) - pub fn search( - &self, - query: &AudioFingerprint, - threshold: f32, // 0.0-1.0, higher = more similar - limit: usize, - ) -> Result, SearchError> { - let candidates = self.db.get_fingerprints_by_duration( - query.duration_secs.saturating_sub(10), - query.duration_secs + 10, - )?; - - let mut matches: Vec = candidates - .into_iter() - .filter_map(|(file_id, fp_bytes, duration)| { - let fp = AudioFingerprint::from_bytes(&fp_bytes).ok()?; - let similarity = self.compare(&query.raw, &fp.raw); - - if similarity >= threshold { - Some(FingerprintMatch { file_id, similarity, duration }) - } else { - None - } - }) - .collect(); - - // Sort by similarity descending - matches.sort_by(|a, b| b.similarity.partial_cmp(&a.similarity).unwrap()); - matches.truncate(limit); - - Ok(matches) - } - - /// Compare two fingerprints using bit error rate - fn compare(&self, a: &[u32], b: &[u32]) -> f32 { - let len = a.len().min(b.len()); - if len == 0 { - return 0.0; - } - - let mut matching_bits = 0u32; - let mut total_bits = 0u32; - - for i in 0..len { - let xor = a[i] ^ b[i]; - matching_bits += 32 - xor.count_ones(); - total_bits += 32; - } - - matching_bits as f32 / total_bits as f32 - } - - /// Find duplicates by fingerprint - pub fn find_duplicates(&self, threshold: f32) -> Result, SearchError> { - let all_fps = self.db.get_all_fingerprints()?; - let mut groups: Vec = Vec::new(); - let mut processed: HashSet = HashSet::new(); - - for (file_id, fp_bytes, duration) in &all_fps { - if processed.contains(file_id) { - continue; - } - - let fp = AudioFingerprint::from_bytes(fp_bytes)?; - let matches = self.search(&fp, threshold, 100)?; - - if matches.len() > 1 { - let group = DuplicateGroup { - files: matches.iter().map(|m| m.file_id).collect(), - similarity: matches.iter().map(|m| m.similarity).sum::() / matches.len() as f32, - }; - - for m in &matches { - processed.insert(m.file_id); - } - - groups.push(group); - } - } - - Ok(groups) - } -} - -#[derive(Debug)] -pub struct FingerprintMatch { - pub file_id: FileId, - pub similarity: f32, - pub duration: u32, -} - -#[derive(Debug)] -pub struct DuplicateGroup { - pub files: Vec, - pub similarity: f32, -} -``` - ---- - -## Task 3: M4B Audiobook Support (`musicfs-metadata/src/formats/m4b.rs`) - -```rust -use symphonia::core::meta::StandardTagKey; - -/// M4B audiobook metadata (FR-24.2) -#[derive(Debug, Clone, Default)] -pub struct AudiobookMeta { - pub title: Option, - pub author: Option, // Maps to "artist" in audio - pub narrator: Option, - pub series: Option, - pub series_part: Option, - pub description: Option, - pub publisher: Option, - pub year: Option, - pub duration_ms: Option, - pub chapters: Vec, -} - -#[derive(Debug, Clone)] -pub struct Chapter { - pub index: u32, - pub title: String, - pub start_ms: u64, - pub end_ms: u64, -} - -impl Chapter { - pub fn duration_ms(&self) -> u64 { - self.end_ms - self.start_ms - } -} - -pub struct M4bParser; - -impl M4bParser { - /// Parse M4B audiobook with chapters - pub fn parse(&self, path: &Path) -> Result { - let file = std::fs::File::open(path)?; - let mss = MediaSourceStream::new(Box::new(file), Default::default()); - - let mut hint = Hint::new(); - hint.with_extension("m4b"); - - let probed = symphonia::default::get_probe() - .format(&hint, mss, &FormatOptions::default(), &MetadataOptions::default())?; - - let mut meta = AudiobookMeta::default(); - let format = probed.format; - - // Extract metadata - if let Some(metadata) = format.metadata().current() { - for tag in metadata.tags() { - if let Some(std_key) = tag.std_key { - let value = tag.value.to_string(); - match std_key { - StandardTagKey::TrackTitle | StandardTagKey::Album => { - meta.title = Some(value); - } - StandardTagKey::Artist => { - meta.author = Some(value); - } - StandardTagKey::Composer => { - meta.narrator = Some(value); - } - StandardTagKey::Description => { - meta.description = Some(value); - } - StandardTagKey::Label => { - meta.publisher = Some(value); - } - StandardTagKey::Date => { - meta.year = value.chars().take(4).collect::().parse().ok(); - } - _ => {} - } - } - } - } - - // Extract chapters from MP4 chpl atom - meta.chapters = self.extract_chapters(&format)?; - - // Get total duration - if let Some(track) = format.tracks().first() { - if let (Some(n_frames), Some(sample_rate)) = - (track.codec_params.n_frames, track.codec_params.sample_rate) - { - meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64); - } - } - - Ok(meta) - } - - fn extract_chapters(&self, format: &dyn FormatReader) -> Result, MetadataError> { - let mut chapters = Vec::new(); - - // Symphonia exposes chapters via cues - if let Some(cues) = format.cues() { - for (idx, cue) in cues.iter().enumerate() { - let start_ms = (cue.start_ts as f64 / cue.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64; - - // End time is start of next chapter or track end - let end_ms = cues.get(idx + 1) - .map(|next| (next.start_ts as f64 / next.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64) - .unwrap_or(u64::MAX); // Will be clamped to duration - - chapters.push(Chapter { - index: idx as u32, - title: cue.tags.iter() - .find(|t| t.std_key == Some(StandardTagKey::TrackTitle)) - .map(|t| t.value.to_string()) - .unwrap_or_else(|| format!("Chapter {}", idx + 1)), - start_ms, - end_ms, - }); - } - } - - Ok(chapters) - } -} -``` - ---- - -## Task 4: Chapter Extraction (`musicfs-metadata/src/chapters.rs`) - -```rust -/// Generic chapter support for various formats -pub trait ChapterSource { - fn chapters(&self) -> &[Chapter]; - fn chapter_at(&self, position_ms: u64) -> Option<&Chapter>; -} - -impl ChapterSource for AudiobookMeta { - fn chapters(&self) -> &[Chapter] { - &self.chapters - } - - fn chapter_at(&self, position_ms: u64) -> Option<&Chapter> { - self.chapters.iter() - .find(|c| position_ms >= c.start_ms && position_ms < c.end_ms) - } -} - -/// Virtual chapter file generator -pub struct ChapterFileGenerator; - -impl ChapterFileGenerator { - /// Generate virtual files for each chapter - /// Example: book.m4b -> book/01 - Introduction.m4b.chapter - pub fn generate_virtual_files(&self, meta: &AudiobookMeta, base_path: &VirtualPath) -> Vec { - meta.chapters.iter() - .map(|chapter| { - let filename = format!( - "{:02} - {}.chapter", - chapter.index + 1, - sanitize_filename(&chapter.title) - ); - - VirtualChapterFile { - path: base_path.join(&filename), - chapter_index: chapter.index, - start_ms: chapter.start_ms, - end_ms: chapter.end_ms, - title: chapter.title.clone(), - } - }) - .collect() - } -} - -#[derive(Debug)] -pub struct VirtualChapterFile { - pub path: VirtualPath, - pub chapter_index: u32, - pub start_ms: u64, - pub end_ms: u64, - pub title: String, -} - -fn sanitize_filename(name: &str) -> String { - name.chars() - .map(|c| match c { - '/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_', - _ => c, - }) - .collect() -} -``` - ---- - -## Task 5: Virtual Chapter Files (`musicfs-fuse/src/ops/chapters.rs`) - -```rust -use crate::VirtualFs; - -impl VirtualFs { - /// Handle reads from virtual chapter files - /// These return a byte-range reference to the parent M4B file - pub async fn read_chapter( - &self, - chapter_file: &VirtualChapterFile, - offset: u64, - size: usize, - ) -> Result, FuseError> { - // Get the parent audiobook file - let parent = self.get_parent_audiobook(&chapter_file.path)?; - - // Calculate byte range for this chapter - // This requires knowing the audio bitrate to convert ms -> bytes - let meta = self.get_audiobook_meta(&parent)?; - let bitrate_bps = meta.bitrate.unwrap_or(128_000); // Default 128kbps - let bytes_per_ms = bitrate_bps / 8 / 1000; - - let chapter_start_bytes = chapter_file.start_ms * bytes_per_ms; - let chapter_end_bytes = chapter_file.end_ms * bytes_per_ms; - - // Adjust offset to be within chapter - let actual_offset = chapter_start_bytes + offset; - let max_size = (chapter_end_bytes - actual_offset) as usize; - let read_size = size.min(max_size); - - // Read from the actual file - self.read_file(&parent, actual_offset, read_size).await - } - - /// List chapter files for an audiobook - pub fn list_chapters(&self, audiobook_path: &VirtualPath) -> Result, FuseError> { - let meta = self.get_audiobook_meta(audiobook_path)?; - let generator = ChapterFileGenerator; - - let chapters = generator.generate_virtual_files(&meta, audiobook_path); - - Ok(chapters.into_iter() - .map(|c| DirEntry { - name: c.path.filename().to_string(), - kind: FileType::RegularFile, - size: self.estimate_chapter_size(&c), - }) - .collect()) - } - - fn estimate_chapter_size(&self, chapter: &VirtualChapterFile) -> u64 { - // Estimate based on duration and typical bitrate - let duration_secs = (chapter.end_ms - chapter.start_ms) / 1000; - duration_secs * 128_000 / 8 // 128kbps assumption - } -} -``` - ---- - -## Task 6: Fingerprint Search Virtual Directory - -```rust -/// Virtual directory for fingerprint search -/// /.search/fingerprint/{base64_fingerprint} -> matching files - -impl SearchOps { - pub async fn search_by_fingerprint( - &self, - fingerprint_path: &str, - ) -> Result, SearchError> { - // Path format: /.search/fingerprint/{base64_encoded_fingerprint} - let fp_bytes = base64::decode(fingerprint_path) - .map_err(|_| SearchError::InvalidQuery)?; - - let fingerprint = AudioFingerprint::from_bytes(&fp_bytes)?; - let matches = self.fingerprint_index.search(&fingerprint, 0.8, 20)?; - - let mut results = Vec::new(); - for m in matches { - if let Some(file) = self.db.get_file_by_id(m.file_id)? { - results.push(SearchResult { - path: file.virtual_path, - score: m.similarity, - snippet: format!("Similarity: {:.1}%", m.similarity * 100.0), - }); - } - } - - Ok(results) - } -} -``` - ---- - -## Database Schema Additions - -```sql --- Fingerprint storage -CREATE TABLE IF NOT EXISTS fingerprints ( - file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE, - fingerprint BLOB NOT NULL, -- Compressed chromaprint - duration INTEGER NOT NULL, -- Duration in seconds - indexed_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')) -); - -CREATE INDEX IF NOT EXISTS idx_fingerprints_duration ON fingerprints(duration); - --- Audiobook chapters -CREATE TABLE IF NOT EXISTS chapters ( - id INTEGER PRIMARY KEY, - file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE, - chapter_idx INTEGER NOT NULL, - title TEXT NOT NULL, - start_ms INTEGER NOT NULL, - end_ms INTEGER NOT NULL, - UNIQUE(file_id, chapter_idx) -); - -CREATE INDEX IF NOT EXISTS idx_chapters_file ON chapters(file_id); -``` - ---- - -## Tests - -| Test | Type | Validates | -|------|------|-----------| -| `test_fingerprint_generation` | Unit | Chromaprint from audio (FR-14.4) | -| `test_fingerprint_similarity` | Unit | Bit comparison algorithm | -| `test_fingerprint_search` | Integration | Find similar tracks | -| `test_fingerprint_duplicates` | Integration | Detect duplicate audio | -| `test_m4b_parsing` | Unit | M4B metadata extraction (FR-24.2) | -| `test_chapter_extraction` | Unit | Chapter list from M4B | -| `test_virtual_chapter_files` | Integration | Chapter files appear in listing | -| `test_chapter_read` | Integration | Read chapter content | -| `test_audiobook_navigation` | E2E | Browse audiobook chapters | - ---- - -## Exit Criteria - -- [ ] Audio fingerprints generated from audio files -- [ ] Fingerprint similarity search finds matching tracks -- [ ] Duplicate detection works across library -- [ ] M4B files parsed with full metadata -- [ ] Chapters extracted and stored -- [ ] Virtual chapter files appear in directory listing -- [ ] Chapter files are readable (return correct byte range) -- [ ] All tests pass - ---- - -## Architecture Alignment - -Per requirements.md: -- FR-14.4: Audio fingerprint search ✓ -- FR-24.2: Audiobook formats with chapters ✓ - -Per architecture.md section 4.3.4: -- FormatPlugin trait for M4B support ✓ -- Chapter extraction via symphonia ✓ diff --git a/docs/v2/requirements.md b/docs/v2/requirements.md deleted file mode 100644 index 353c985..0000000 --- a/docs/v2/requirements.md +++ /dev/null @@ -1,649 +0,0 @@ -# Music Library FUSE Filesystem - Requirements Specification - -**Version**: 1.0 -**Date**: 2026-05-12 -**Status**: Draft - -## 1. Introduction - -### 1.1 Purpose - -This document specifies the requirements for a FUSE-based virtual filesystem that presents a music library organized by metadata. The system overlays metadata onto audio files without modifying originals and operates as a read-only client against the origin storage. - -### 1.2 Scope - -The system provides: -- Virtual filesystem accessible via standard POSIX operations -- Metadata-based directory structure (artist/album/track) -- Local caching with delta synchronization -- Support for local and remote origin storage - -### 1.3 Definitions - -| Term | Definition | -|------|------------| -| **Origin** | The source storage containing original audio files (local FS, NFS, S3, etc.) | -| **Virtual path** | The metadata-derived path shown to users (e.g., `/Artist/Album/Track.flac`) | -| **Real path** | The actual path on origin storage | -| **Metadata overlay** | Serving synthesized file headers from cached metadata | -| **CDC** | Content-Defined Chunking - algorithm for stable file segmentation | - ---- - -## 2. System Overview - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ User Applications │ -│ (mpv, Rhythmbox, Plex, etc.) │ -└─────────────────────────────┬───────────────────────────────────┘ - │ POSIX (read-only) - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ FUSE Interface │ -├─────────────────────────────────────────────────────────────────┤ -│ Plugin Host │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ Origin │ │ Metadata │ │ Format │ │ -│ │ Plugins │ │ Plugins │ │ Plugins │ │ -│ └─────────────┘ └─────────────┘ └─────────────┘ │ -├─────────────────────────────────────────────────────────────────┤ -│ Core Services │ -│ ┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐ │ -│ │ Virtual │ │ Event │ │ Search │ │ Control │ │ -│ │ Path │ │ Bus │ │ Index │ │ API │ │ -│ │ Resolver │ │ │ │ │ │ │ │ -│ └───────────┘ └───────────┘ └───────────┘ └───────────┘ │ -├─────────────────────────────────────────────────────────────────┤ -│ Storage Layer │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Content-Addressable Chunk Store │ │ -│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ -│ │ │ Metadata │ │ Content │ │ Tree │ │ │ -│ │ │ Cache │ │ Chunks │ │ Cache │ │ │ -│ │ │ (SQLite) │ │ (CAS) │ │ │ │ │ -│ │ └──────────┘ └──────────┘ └──────────┘ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -├─────────────────────────────────────────────────────────────────┤ -│ Origin Federation │ -│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ -│ │ Local │ │ NFS │ │ S3 │ │ SFTP │ │ -│ │ FS │ │ │ │ │ │ │ │ -│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ -└─────────────────────────────────────────────────────────────────┘ - │ read-only - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Origin Storage(s) │ -│ (original audio files) │ -└─────────────────────────────────────────────────────────────────┘ -``` - ---- - -## 3. Functional Requirements - -### 3.1 Filesystem Operations - -#### FR-1: Mount/Unmount - -| ID | Requirement | -|----|-------------| -| FR-1.1 | The system SHALL mount as a FUSE filesystem at a user-specified mountpoint | -| FR-1.2 | The system SHALL return control to the caller within 500ms of mount initiation | -| FR-1.3 | The system SHALL unmount cleanly via `fusermount -u` | -| FR-1.4 | The system SHALL release all resources (file handles, connections) on unmount | - -#### FR-2: Directory Operations - -| ID | Requirement | -|----|-------------| -| FR-2.1 | The system SHALL present files organized by metadata path format | -| FR-2.2 | The system SHALL support configurable path templates (e.g., `$artist/$album/$track - $title.$format`) | -| FR-2.3 | The system SHALL return directory listings via `readdir()` | -| FR-2.4 | The system SHALL support nested directory traversal to arbitrary depth | -| FR-2.5 | The system SHALL handle directories with 100,000+ entries | - -#### FR-3: File Operations (Read) - -| ID | Requirement | -|----|-------------| -| FR-3.1 | The system SHALL support `open()` for reading | -| FR-3.2 | The system SHALL support `read()` with arbitrary offset and size | -| FR-3.3 | The system SHALL support `seek()` operations for random access | -| FR-3.4 | The system SHALL return file attributes via `stat()` / `fstat()` | -| FR-3.5 | The system SHALL support concurrent reads from multiple processes | - -#### FR-4: Read-Only Constraint - -| ID | Requirement | -|----|-------------| -| FR-4.1 | The system SHALL NOT modify original files on the origin storage | -| FR-4.2 | The system SHALL NOT push any changes to the origin server | -| FR-4.3 | The system SHALL return `EROFS` (Read-only filesystem) for write operations | -| FR-4.4 | The system SHALL return `EROFS` for `create()`, `mkdir()`, `unlink()`, `rmdir()` | -| FR-4.5 | The system SHALL return `EROFS` for `rename()`, `chmod()`, `chown()`, `truncate()` | - -### 3.2 Metadata Handling - -#### FR-5: Metadata Overlay - -| ID | Requirement | -|----|-------------| -| FR-5.1 | The system SHALL extract metadata from audio files on first access | -| FR-5.2 | The system SHALL cache extracted metadata in a local database | -| FR-5.3 | The system SHALL serve file headers with metadata from cache | -| FR-5.4 | The system SHALL support FLAC Vorbis comments | -| FR-5.5 | The system SHALL support MP3 ID3v2 tags | -| FR-5.6 | The system SHOULD support additional formats (OGG, M4A, OPUS) | - -#### FR-6: Metadata Fields - -| ID | Requirement | -|----|-------------| -| FR-6.1 | The system SHALL extract and cache: title, artist, album, genre | -| FR-6.2 | The system SHALL extract and cache: year, track number, disc number | -| FR-6.3 | The system SHALL extract and cache: duration, bitrate, sample rate | -| FR-6.4 | The system SHOULD extract: composer, album artist, lyrics | -| FR-6.5 | The system SHALL handle missing metadata gracefully with defaults | - -### 3.3 Caching - -#### FR-7: Metadata Cache - -| ID | Requirement | -|----|-------------| -| FR-7.1 | The system SHALL persist metadata cache across restarts | -| FR-7.2 | The system SHALL store metadata in SQLite database | -| FR-7.3 | The system SHALL index by both virtual path and real path | -| FR-7.4 | The system SHALL invalidate cache entries when origin file changes | - -#### FR-8: Content Cache - -| ID | Requirement | -|----|-------------| -| FR-8.1 | The system SHALL cache file content in fixed-size chunks | -| FR-8.2 | The system SHALL use content-defined chunking for cache efficiency | -| FR-8.3 | The system SHALL store chunk hashes for delta detection | -| FR-8.4 | The system SHALL evict chunks under memory/disk pressure | - -#### FR-9: Directory Tree Cache - -| ID | Requirement | -|----|-------------| -| FR-9.1 | The system SHALL cache directory listings locally | -| FR-9.2 | The system SHALL serve `readdir()` from cache without origin access | -| FR-9.3 | The system SHALL refresh tree cache based on configurable policy | -| FR-9.4 | The system SHALL support forced refresh via signal or special file | - -### 3.4 Synchronization - -#### FR-10: Change Detection - -| ID | Requirement | -|----|-------------| -| FR-10.1 | The system SHALL detect changes to origin files | -| FR-10.2 | The system SHALL use inotify for local filesystem origins | -| FR-10.3 | The system SHALL use polling for remote origins without push support | -| FR-10.4 | The system SHALL compare mtime and size for change detection | -| FR-10.5 | The system SHALL support content-hash verification on demand | - -#### FR-11: Delta Sync - -| ID | Requirement | -|----|-------------| -| FR-11.1 | The system SHALL download only changed portions of files | -| FR-11.2 | The system SHALL use CDC to identify changed chunks | -| FR-11.3 | The system SHALL preserve unchanged chunks in cache | -| FR-11.4 | The system SHALL handle file additions and deletions | - -### 3.5 Origin Support - -#### FR-12: Origin Types - -| ID | Requirement | -|----|-------------| -| FR-12.1 | The system SHALL support local filesystem as origin | -| FR-12.2 | The system SHOULD support NFS mounted filesystems | -| FR-12.3 | The system SHOULD support SMB/CIFS shares | -| FR-12.4 | The system SHOULD support S3-compatible object storage | -| FR-12.5 | The system SHOULD support SFTP servers | -| FR-12.6 | The system SHALL provide pluggable origin interface | - -#### FR-13: Multiple Origins [P0] - -| ID | Requirement | -|----|-------------| -| FR-13.1 | The system SHALL support multiple simultaneous origins | -| FR-13.2 | The system SHALL present unified virtual tree across origins | -| FR-13.3 | The system SHALL support origin priority/preference ordering | -| FR-13.4 | The system SHALL handle duplicate files across origins | -| FR-13.5 | The system SHALL support per-origin configuration | - -### 3.6 Search & Discovery - -#### FR-14: Full-Text Search [P1] - -| ID | Requirement | -|----|-------------| -| FR-14.1 | The system SHALL index metadata for full-text search | -| FR-14.2 | The system SHALL expose search via virtual directory (`/.search/query/`) | -| FR-14.3 | The system SHALL support fuzzy matching | -| FR-14.4 | The system SHOULD support search by audio fingerprint | - -#### FR-15: Smart Collections [P1] - -| ID | Requirement | -|----|-------------| -| FR-15.1 | The system SHALL support query-based virtual folders | -| FR-15.2 | The system SHALL support saved searches as directories | -| FR-15.3 | The system SHALL support dynamic playlists (recently played, most played) | -| FR-15.4 | The system SHOULD support user-defined metadata fields | - -### 3.7 Album Art - -#### FR-16: Cover Art Handling [P1] - -| ID | Requirement | -|----|-------------| -| FR-16.1 | The system SHALL extract embedded album art | -| FR-16.2 | The system SHALL expose art as virtual files (`/Artist/Album/cover.jpg`) | -| FR-16.3 | The system SHALL cache artwork separately from audio | -| FR-16.4 | The system SHALL support multiple art sizes (thumbnail, medium, full) | -| FR-16.5 | The system SHOULD fetch missing art from online sources | - -### 3.8 Control & API - -#### FR-17: Control Interface [P0] - -| ID | Requirement | -|----|-------------| -| FR-17.1 | The system SHALL expose control via Unix socket (gRPC) | -| FR-17.2 | The system SHALL use gRPC with Protocol Buffers for all control APIs | -| FR-17.3 | The system SHALL support cache management commands (clear, refresh, stats) | -| FR-17.4 | The system SHALL support runtime configuration changes | -| FR-17.5 | The system SHALL support graceful shutdown with drain | - -#### FR-18: Event System [P0] - -| ID | Requirement | -|----|-------------| -| FR-18.1 | The system SHALL emit events for file access | -| FR-18.2 | The system SHALL support webhook notifications | -| FR-18.3 | The system SHOULD support event streaming (SSE/WebSocket) | -| FR-18.4 | The system SHALL log access patterns for analysis | - -### 3.9 Caching Enhancements - -#### FR-19: Intelligent Prefetching [P1] - -| ID | Requirement | -|----|-------------| -| FR-19.1 | The system SHALL learn access patterns | -| FR-19.2 | The system SHALL support playlist-aware prefetching | -| FR-19.3 | The system SHOULD support time-based prefetching | -| FR-19.4 | The system SHALL support manual prefetch hints (`/.prefetch/path/`) | - -#### FR-20: Content-Addressable Storage [P0] - -| ID | Requirement | -|----|-------------| -| FR-20.1 | The system SHALL store chunks by content hash | -| FR-20.2 | The system SHALL detect identical files across library | -| FR-20.3 | The system SHALL report deduplication statistics | -| FR-20.4 | The system SHALL enable cache sharing via content addressing | - -### 3.10 Integration - -#### FR-21: Metadata Sources [P1] - -| ID | Requirement | -|----|-------------| -| FR-21.1 | The system SHOULD integrate with MusicBrainz | -| FR-21.2 | The system SHOULD integrate with Discogs | -| FR-21.3 | The system SHOULD integrate with Last.fm | -| FR-21.4 | The system SHOULD support AcoustID fingerprinting | -| FR-21.5 | The system SHALL support custom metadata plugins | - -#### FR-22: Import & Migration [P1] - -| ID | Requirement | -|----|-------------| -| FR-22.1 | The system SHALL import from beets database | -| FR-22.2 | The system SHOULD import from iTunes/Apple Music library | -| FR-22.3 | The system SHALL export library metadata | - -### 3.11 Extensibility - -#### FR-23: Plugin System [P0] - -| ID | Requirement | -|----|-------------| -| FR-23.1 | The system SHALL support loadable plugins | -| FR-23.2 | The system SHALL define stable plugin API | -| FR-23.3 | The system SHALL support plugins for: origins, metadata extractors, formats | -| FR-23.4 | The system SHOULD support WASM plugins for sandboxed execution | -| FR-23.5 | The system SHALL provide plugin lifecycle management (load, unload, reload) | - -#### FR-24: Format Extensibility [P1] - -| ID | Requirement | -|----|-------------| -| FR-24.1 | The system SHALL support pluggable codec modules | -| FR-24.2 | The system SHOULD support audiobook formats (M4B, chapters) | -| FR-24.3 | The system SHALL allow format plugins to register file extensions | - -### 3.12 High Availability [P3] - -#### FR-25: Resilience - -| ID | Requirement | -|----|-------------| -| FR-25.1 | The system SHOULD support active-passive failover | -| FR-25.2 | The system SHOULD support read replicas | -| FR-25.3 | The system SHALL support zero-downtime upgrades | -| FR-25.4 | The system SHALL support cache backup/restore | -| FR-25.5 | The system SHALL validate cache integrity on startup | - ---- - -## 4. Non-Functional Requirements - -### 4.1 Performance - -#### NFR-1: Latency - -| ID | Requirement | Target | Maximum | -|----|-------------|--------|---------| -| NFR-1.1 | `stat()` on cached file | <1ms | 5ms | -| NFR-1.2 | `readdir()` on cached directory | <10ms | 50ms | -| NFR-1.3 | `open()` on cached file | <5ms | 20ms | -| NFR-1.4 | `read()` from cache | <1ms | 5ms | -| NFR-1.5 | `read()` cache miss (local origin) | <50ms | 200ms | -| NFR-1.6 | `read()` cache miss (remote origin) | <200ms | 1000ms | -| NFR-1.7 | Mount completion | <100ms | 500ms | - -#### NFR-2: Throughput - -| ID | Requirement | Target | -|----|-------------|--------| -| NFR-2.1 | Sequential read throughput (cached) | >500 MB/s | -| NFR-2.2 | Sequential read throughput (local origin) | >200 MB/s | -| NFR-2.3 | Metadata operations per second | >1000 ops/s | -| NFR-2.4 | Concurrent file handles | >1000 | - -#### NFR-3: Scalability - -| ID | Requirement | -|----|-------------| -| NFR-3.1 | The system SHALL handle libraries with 1,000,000+ files | -| NFR-3.2 | The system SHALL handle directories with 100,000+ entries | -| NFR-3.3 | The system SHALL maintain O(1) mount time regardless of library size | -| NFR-3.4 | The system SHALL maintain O(log n) lookup time for paths | -| NFR-3.5 | The system SHOULD handle libraries with 10,000,000+ files [P3] | -| NFR-3.6 | The system SHOULD support 100+ concurrent clients [P3] | -| NFR-3.7 | The system SHOULD achieve <100μs cached stat for high-performance use [P3] | - -### 4.2 Resource Usage - -#### NFR-4: Memory - -| ID | Requirement | Limit | -|----|-------------|-------| -| NFR-4.1 | Idle memory usage | <50 MB | -| NFR-4.2 | Active usage (1000 files accessed) | <200 MB | -| NFR-4.3 | Peak usage under load | <500 MB | -| NFR-4.4 | Per-file metadata overhead | <1 KB | -| NFR-4.5 | The system SHALL NOT load entire files into memory | - -#### NFR-5: Disk - -| ID | Requirement | -|----|-------------| -| NFR-5.1 | Metadata cache size SHALL be configurable (default: 100 MB) | -| NFR-5.2 | Content cache size SHALL be configurable (default: 10 GB) | -| NFR-5.3 | The system SHALL evict cache entries under disk pressure | -| NFR-5.4 | The system SHALL function with cache disabled (passthrough mode) | - -#### NFR-6: Network - -| ID | Requirement | -|----|-------------| -| NFR-6.1 | The system SHALL minimize network round-trips via batching | -| NFR-6.2 | The system SHALL use connection pooling for remote origins | -| NFR-6.3 | The system SHALL support bandwidth limiting (configurable) | -| NFR-6.4 | Delta sync SHALL achieve >90% bandwidth reduction vs full copy | - -### 4.3 Reliability - -#### NFR-7: Availability - -| ID | Requirement | -|----|-------------| -| NFR-7.1 | The system SHALL serve cached data when origin is unavailable | -| NFR-7.2 | The system SHALL gracefully degrade with network failures | -| NFR-7.3 | The system SHALL retry failed operations with exponential backoff | -| NFR-7.4 | The system SHALL not crash on malformed audio files | - -#### NFR-8: Data Integrity - -| ID | Requirement | -|----|-------------| -| NFR-8.1 | The system SHALL verify chunk integrity via checksums | -| NFR-8.2 | The system SHALL use ACID transactions for cache database | -| NFR-8.3 | The system SHALL recover from interrupted synchronization | -| NFR-8.4 | The system SHALL detect and report cache corruption | - -### 4.4 Usability - -#### NFR-9: Configuration - -| ID | Requirement | -|----|-------------| -| NFR-9.1 | The system SHALL support configuration via file (TOML/YAML) | -| NFR-9.2 | The system SHALL support configuration via command-line arguments | -| NFR-9.3 | The system SHALL support configuration via environment variables | -| NFR-9.4 | The system SHALL provide sensible defaults for all options | - -#### NFR-10: Observability - -| ID | Requirement | -|----|-------------| -| NFR-10.1 | The system SHALL log operations at configurable verbosity | -| NFR-10.2 | The system SHALL expose metrics (cache hit rate, latency, etc.) | -| NFR-10.3 | The system SHALL support health check endpoint/signal | -| NFR-10.4 | The system SHOULD support integration with Prometheus/StatsD | - -### 4.5 Compatibility - -#### NFR-11: Platform Support - -| ID | Requirement | -|----|-------------| -| NFR-11.1 | The system SHALL run on Linux (kernel 4.x+) | -| NFR-11.2 | The system SHOULD run on macOS (via macFUSE) | -| NFR-11.3 | The system SHALL require FUSE kernel module | -| NFR-11.4 | The system SHALL run without root privileges (user-space FUSE) | - -#### NFR-12: Application Compatibility - -| ID | Requirement | -|----|-------------| -| NFR-12.1 | The system SHALL work with standard media players (mpv, VLC, etc.) | -| NFR-12.2 | The system SHALL work with media servers (Plex, Jellyfin) | -| NFR-12.3 | The system SHALL work with file managers (Nautilus, Dolphin) | -| NFR-12.4 | The system SHALL correctly report file sizes and timestamps | - -### 4.6 Security - -#### NFR-13: Access Control - -| ID | Requirement | -|----|-------------| -| NFR-13.1 | The system SHALL respect origin file permissions | -| NFR-13.2 | The system SHALL run as unprivileged user | -| NFR-13.3 | The system SHALL support credential storage for remote origins | -| NFR-13.4 | The system SHALL NOT expose credentials in logs or process list | - -### 4.7 Maintainability - -#### NFR-14: Code Quality - -| ID | Requirement | -|----|-------------| -| NFR-14.1 | The system SHALL be implemented in a memory-safe language | -| NFR-14.2 | The system SHALL have no global interpreter lock (no Python/Ruby) | -| NFR-14.3 | The system SHALL use async I/O for concurrent operations | -| NFR-14.4 | The system SHALL have modular architecture with pluggable components | - ---- - -## 5. Constraints - -### 5.1 Technical Constraints - -| ID | Constraint | -|----|------------| -| C-1 | Must use FUSE for filesystem interface | -| C-2 | Must not require kernel module development | -| C-3 | Must work with existing audio file formats (no transcoding) | -| C-4 | Cache database must be portable (no external database server) | - -### 5.2 Operational Constraints - -| ID | Constraint | -|----|------------| -| C-5 | Client is read-only; no writes propagate to origin | -| C-6 | Must function offline with cached data | -| C-7 | Must not corrupt origin files under any circumstances | - ---- - -## 6. Assumptions - -| ID | Assumption | -|----|------------| -| A-1 | Origin storage is accessible via supported protocol | -| A-2 | Audio files contain valid metadata headers | -| A-3 | Sufficient local disk space for caching is available | -| A-4 | FUSE kernel module is installed and accessible | -| A-5 | Network connectivity is intermittent but generally available | - ---- - -## 7. Dependencies - -| ID | Dependency | Purpose | -|----|------------|---------| -| D-1 | FUSE library (fuser/libfuse) | Filesystem interface | -| D-2 | SQLite | Metadata and tree cache | -| D-3 | Audio parsing library (symphonia) | Metadata extraction | -| D-4 | Async runtime (tokio) | Concurrent I/O | -| D-5 | CDC library (fastcdc) | Content chunking | -| D-6 | Full-text search (tantivy) | Search index [P1] | -| D-7 | Image processing (image) | Album art thumbnails [P1] | -| D-8 | HTTP client (reqwest) | Remote origins, metadata APIs | -| D-9 | WASM runtime (wasmtime) | Plugin sandboxing [P0] | -| D-10 | Hash library (xxhash/blake3) | Content addressing [P0] | - ---- - -## 8. Acceptance Criteria - -### 8.1 Functional Acceptance - -| ID | Criterion | -|----|-----------| -| AC-1 | Mount filesystem and browse directories via `ls` | -| AC-2 | Play audio file through mounted filesystem with media player | -| AC-3 | Seek within audio file without full download | -| AC-4 | Directory listing completes without network access (when cached) | -| AC-5 | Confirm write operations return EROFS | -| AC-6 | Detect and sync changes from origin within configured interval | - -### 8.2 Performance Acceptance - -| ID | Criterion | -|----|-----------| -| AC-7 | Mount completes in <500ms for library of any size | -| AC-8 | Cached stat() completes in <5ms (p99) | -| AC-9 | Memory stays under 500MB with 10,000 files accessed | -| AC-10 | Tag-only change syncs <10KB of data | - -### 8.3 Reliability Acceptance - -| ID | Criterion | -|----|-----------| -| AC-11 | Filesystem remains accessible when origin is offline | -| AC-12 | No data corruption after unclean unmount | -| AC-13 | Recovers automatically when origin comes back online | - -### 8.4 Multi-Origin Acceptance [P0] - -| ID | Criterion | -|----|-----------| -| AC-14 | Configure and mount multiple origins simultaneously | -| AC-15 | Browse unified tree showing content from all origins | -| AC-16 | Access same file from preferred origin when duplicated | - -### 8.5 Search & Discovery Acceptance [P1] - -| ID | Criterion | -|----|-----------| -| AC-17 | Search for tracks by partial artist/album/title match | -| AC-18 | Browse smart collection (e.g., "Jazz from 1960s") | -| AC-19 | View album art via virtual cover.jpg file | - -### 8.6 Plugin Acceptance [P0] - -| ID | Criterion | -|----|-----------| -| AC-20 | Load custom origin plugin at runtime | -| AC-21 | Control daemon via Unix socket (cache stats, refresh) | -| AC-22 | Receive webhook on file access event | - -### 8.7 Deduplication Acceptance [P0] - -| ID | Criterion | -|----|-----------| -| AC-23 | Identical chunks stored once regardless of file count | -| AC-24 | Deduplication stats visible via control API | - ---- - -## 9. Appendix - -### 9.1 Comparison with beetfs - -| Requirement Area | beetfs | This Specification | -|------------------|--------|-------------------| -| Mount time | O(N), 5-120s | O(1), <500ms (NFR-1.7) | -| Memory per file | Full file size | <1KB (NFR-4.4) | -| Write to origin | Yes (DB updates) | No (FR-4.1, FR-4.2) | -| Delta sync | None | Required (FR-11) | -| Remote origins | None | Required (FR-12) | -| Offline access | No | Required (NFR-7.1) | -| Cache persistence | No | Required (FR-7.1) | - -### 9.2 Path Template Variables - -| Variable | Description | Example | -|----------|-------------|---------| -| `$artist` | Track artist | "Metallica" | -| `$album` | Album name | "72 Seasons" | -| `$title` | Track title | "Lux Æterna" | -| `$track` | Track number (zero-padded) | "03" | -| `$disc` | Disc number | "1" | -| `$year` | Release year | "2023" | -| `$genre` | Genre | "Metal" | -| `$format` | File extension | "flac" | -| `$format_upper` | File extension (uppercase) | "FLAC" | - -### 9.3 Error Codes - -| Operation | Error | Code | -|-----------|-------|------| -| Any write operation | Read-only filesystem | EROFS (30) | -| File not found | No such file | ENOENT (2) | -| Origin unavailable | I/O error | EIO (5) | -| Permission denied | Access denied | EACCES (13) | diff --git a/docs/v2/week-07-performance-review.md b/docs/v2/week-07-performance-review.md deleted file mode 100644 index 6717838..0000000 --- a/docs/v2/week-07-performance-review.md +++ /dev/null @@ -1,179 +0,0 @@ -# MusicFS Week 7 Performance Review - -**Date**: 2026-05-12 -**Commit**: `09f0197` (Week 7 Remote Origins) -**Baseline**: `d5ef68c` (Week 6 Origin Federation) -**System**: Linux, NixOS -**Test**: Synthetic benchmarks (CDC chunking, hashing, chunk reuse) - ---- - -## Executive Summary - -**Week 7 Remote Origins adds no performance regression.** The core CDC and hashing algorithms remain unchanged; Week 7 adds I/O wrappers (NFS, SMB, S3, SFTP) that are network-bound, not CPU-bound. All NFR targets continue to be met or exceeded. - ---- - -## Benchmark Results - -### CDC Chunker Throughput - -| Metric | Week 6 | Week 7 | Delta | NFR Target | Status | -|--------|--------|--------|-------|------------|--------| -| CDC Throughput | 3148.7 MB/s | 3007.9 MB/s | -4.5% | N/A* | ✅ | -| Chunks per 10MB | 137 | 137 | 0% | — | ✅ | - -*CDC throughput is internal; NFR-2.1/2.2 measure end-to-end read throughput (>500 MB/s cached, >200 MB/s local origin). CDC at ~3 GB/s confirms chunking is not a bottleneck. - -### Hash Computation Throughput - -| Metric | Week 6 | Week 7 | Delta | Status | -|--------|--------|--------|-------|--------| -| xxHash64 Throughput | 16330.7 MB/s | 16274.6 MB/s | -0.3% | ✅ | - -Hash computation at ~16 GB/s is CPU-limited and far exceeds any I/O bottleneck. - -### Chunk Reuse (NFR-6.4) - -| Metric | Week 6 | Week 7 | NFR-6.4 Target | Status | -|--------|--------|--------|----------------|--------| -| Chunk Reuse | 99.1% | 99.1% | >90% | ✅ PASS | -| Reused Chunks | 107/108 | 107/108 | — | — | -| Edit Size | 100 bytes | 100 bytes | — | — | - -**NFR-6.4**: *"Delta sync SHALL achieve >90% bandwidth reduction vs full copy"* - -Result: **99.1% bandwidth reduction** for mid-file metadata edits (100 bytes changed in 2MB file). This exceeds the >90% requirement by 9.1 percentage points. - ---- - -## Requirements Compliance - -### NFR-2: Throughput - -| ID | Requirement | Target | Measured | Status | -|----|-------------|--------|----------|--------| -| NFR-2.1 | Sequential read (cached) | >500 MB/s | ~3000 MB/s* | ✅ | -| NFR-2.2 | Sequential read (local origin) | >200 MB/s | ~3000 MB/s* | ✅ | - -*Measured at CDC layer. End-to-end throughput demonstrated in MVP review (2-3 GB/s). - -### NFR-6: Network - -| ID | Requirement | Target | Measured | Status | -|----|-------------|--------|----------|--------| -| NFR-6.4 | Delta sync bandwidth reduction | >90% | 99.1% | ✅ | - -### NFR-7: Availability (Week 7 Additions) - -| ID | Requirement | Implementation | Status | -|----|-------------|----------------|--------| -| NFR-7.3 | Retry with exponential backoff | NFS: ESTALE retry (100ms→200ms→400ms) | ✅ | -| NFR-7.3 | Retry with exponential backoff | SMB: ENOTCONN retry (100ms fixed) | ✅ | - ---- - -## Week 7 Changes Analysis - -### What Changed (No Performance Impact Expected) - -| Component | Change | Performance Impact | -|-----------|--------|-------------------| -| `credentials.rs` | New CredentialStore with redacted Debug | None (startup only) | -| `nfs.rs` | NfsOrigin with ESTALE retry, 5s health timeout | None (error path only) | -| `smb.rs` | SmbOrigin with ENOTCONN retry, 5s health timeout | None (error path only) | -| `s3.rs` | Feature-gated stub | None (not compiled) | -| `sftp.rs` | Feature-gated stub | None (not compiled) | -| `error.rs` | New error variants | None (enum extension) | - -### Why ~4.5% CDC Variance is Noise - -The 4.5% difference (3148.7 → 3007.9 MB/s) is within expected benchmark noise: - -1. **No code path changed** — FastCDC algorithm unchanged -2. **CPU frequency variation** — Turbo boost, thermal throttling -3. **Memory subsystem** — Cache line evictions, NUMA effects -4. **OS scheduler** — Process placement, interrupt handling - -A 4.5% variance over 10 iterations of 10MB data is statistically insignificant. To detect real regressions, we'd need: -- Warmup iterations (discard first N) -- Statistical analysis (mean, stddev, p-value) -- Dedicated benchmark infrastructure (criterion.rs) - ---- - -## Comparison with MVP Performance Review - -| Metric | MVP Review | Week 7 | Change | -|--------|-----------|--------|--------| -| Single file read | 3.2 GB/s (warm) | N/A | — | -| CDC Throughput | Not measured | 3.0 GB/s | Baseline | -| Chunk Reuse | Not measured | 99.1% | Baseline | -| Mount time | ~8ms | N/A | — | -| stat() latency | 3ms | N/A | — | - -MVP review focused on end-to-end FUSE operations. Week 7 review focuses on CDC/sync layer since remote origins add I/O wrappers, not CPU-bound logic. - ---- - -## Test Details - -``` -Test Type: Synthetic microbenchmarks -Data Size: 10 MB (CDC), 64 KB × 10000 (hash), 2 MB (reuse) -Iterations: 10 (CDC), 10000 (hash), 1 (reuse) -Build: cargo build --release -Rust: stable (via nix develop) -``` - -### Benchmark Code - -CDC and hash throughput measured with in-memory data to isolate algorithm performance from I/O. Chunk reuse measured with simulated metadata edit (100 bytes changed mid-file). - ---- - -## Recommendations - -### 1. Add Formal Benchmarks (Priority: Medium) - -Current benchmarks are ad-hoc. Add criterion.rs for: -- Reproducible measurements with statistical analysis -- Regression detection in CI -- Historical tracking - -```toml -[dev-dependencies] -criterion = "0.5" -``` - -### 2. Add Integration Benchmarks (Priority: Low) - -Week 7 adds NFS/SMB wrappers. Add benchmarks for: -- ESTALE retry overhead -- Health check timeout behavior -- Connection pool performance (when S3/SFTP implemented) - -### 3. Test with Real Network Origins (Priority: High for Week 8+) - -Current benchmarks use local mounts. Before deploying: -- Benchmark against real NFS server -- Measure latency distribution (p50, p95, p99) -- Test failure scenarios (network partition, slow origin) - ---- - -## Conclusion - -**Week 7 introduces no performance regression.** The 4.5% CDC throughput variance is within noise margin. NFR-6.4 (>90% bandwidth reduction) continues to be exceeded at 99.1%. - -Remote origin wrappers (NFS, SMB) are I/O-bound and will only affect performance when accessing remote storage. The retry logic (ESTALE, ENOTCONN) and health timeouts are error-path-only and have no impact on happy-path performance. - -**All 102 tests pass with 0 warnings.** - ---- - -## References - -- [Requirements Specification](requirements.md) — NFR-2 (Throughput), NFR-6 (Network), NFR-7 (Availability) -- [MVP Performance Review](mvp-performance-review.md) — Baseline end-to-end measurements -- [Week 7 Plan](plans/week-07-remote-origins.md) — Remote origins implementation