- Add SyncedFile message and subdir scoping to RescanOrigin proto
- Add label, album_type, cover_url fields to UpdateMetadataRequest/MetadataResponse
- Implement OriginScanner: walk, hash, diff, ingest with live FUSE tree and content fetcher registration
- Add enrichment DB columns: enrichment_source, enriched_at, enrichment_attempts, genres_json, label, album_type, cover_url
- Add EnrichmentUpdate struct and update_enrichment DB method
- Wire BatchUpdateMetadata to write enrichment fields alongside audio metadata
- Wire gRPC server into CLI mount command with --grpc-port flag
- Pass VirtualTree and ContentFetcher to scanner so rescanned files are immediately visible and readable via FUSE
The metadata overlay feature was implemented but not connected to the
CLI daemon. Files were being served with original metadata instead of
synthesized headers from the database.
- Import OverlayReader from musicfs-cache
- Create OverlayReader with db, format_registry, and reader
- Call .with_overlay() on MusicFs builder
Tested: ffprobe now shows modified metadata from database updates.
- Add MetadataService with 5 RPCs (Get, Update, Clear, Batch, Import)
- Add 11 message types for requests/responses
- Use optional fields and map for custom_tags
- Proto codegen successful, all tests pass
- Update read() to use OverlayReader when available
- Map OverlayError to libc error codes
- Maintain 30s timeout and backward compatibility
- Fallback to FileReader for non-overlay files
- All tests pass, full workspace compiles
- Add overlay_reader field to MusicFs struct
- Add with_overlay() builder method
- Update getattr() to call estimate_virtual_size()
- Graceful fallback to original size
- All tests pass, backward compatible
- Create FormatHandlerRegistry in CLI initialization
- Register Id3v2Handler and FlacHandler
- Add analyze_format_layout() helper to read file headers
- Update scan functions to call handler.analyze()
- Use upsert_file_with_layout() when format_layout available
- Graceful degradation for unsupported formats
- Full workspace compiles successfully
- Update upsert_file() to include all 26 new AudioMeta fields
- Update get_file_by_virtual_path() to read all new columns
- Add get_file_metadata_row() for overlay synthesis
- Add update_metadata() for partial metadata updates
- Add clear_overlay() to reset metadata to NULL
- Handle format_layout BLOB with msgpack serialization
- Handle custom_tags JSON with serde_json
- Add 8 comprehensive unit tests
- All 92 tests pass, LSP diagnostics clean
- Implement all 8 FormatHandler trait methods
- Use lofty 0.24 for ID3v2.4 tag creation/parsing
- Map all 36 AudioMeta fields to ID3v2 frames
- Handle ID3v2 header parsing for audio_start
- Detect ID3v1 tags at EOF for audio_end
- Add 13 comprehensive unit tests
- Fix test-utils AudioMeta construction with ..Default::default()
- All tests pass, LSP diagnostics clean
Add mkdir and mv (rename) FUSE operations to the virtual filesystem:
- mkdir: Create directories that persist across remounts via SQLite
- mv: Move/rename files and directories with database persistence
Changes:
- Add directories table to schema for user-created empty dirs
- Add tree operations: mkdir, rename_file, rename_directory
- Add DB methods for path updates and directory CRUD
- Remove MountOption::RO to allow write syscalls
- Load stored virtual_path from DB instead of regenerating
- Restore user directories on mount from directories table
- Upsert files to DB during origin scan
POSIX compliant: mv fails with ENOENT if parent doesn't exist
(use mkdir first, shell handles -p flag and brace expansion)
- CLI mountpoint argument is now Option<PathBuf>
- Falls back to config.mount_point when --config is provided
- CLI mountpoint still overrides config if both are given
- Expanded config.example.toml with all available options
Decision: SQLite (Option A) — existing schema, CRUD, row mapping,
and chunk_manifest column are already built but not wired into mount.
8-day plan to transform mount from O(N×origin_latency) to O(N×SQLite_read):
1. Database bulk load + manifest CRUD methods
2. Rewrite run_mount() with DB-load vs first-mount-scan paths
3. Persist chunk manifests via ManifestCached event
4. Wire tantivy + PatternStore + CollectionStore into mount
5. Background delta sync (origin vs DB reconciliation)
6. Shutdown WAL checkpoint
7-8. Integration testing + buffer
Implements all 6 critical resilience fixes from phase-a-stop-dying.md:
- Issue 2.9: Migrate std::sync::RwLock → parking_lot::RwLock (7 files)
Prevents lock poisoning cascade on writer panic
- Issue 2.2: Add install_panic_hook() to log panics via tracing
Ensures panics are captured in logs/journald before process death
- Issue 3.7: Add ExecStopPost to systemd service
Cleans up stale FUSE mounts on service stop
- Issue 2.7: Add check_stale_mount() detection on startup
Auto-cleans leftover mounts from previous crashes
- Issue 2.10: Integrate sd_notify for systemd lifecycle
Sends READY=1 after mount, STOPPING on shutdown
- Issue 2.1: Add signal handling with spawn_mount
Catches SIGTERM/SIGINT for clean shutdown instead of instant death
All 7 Phase A tests pass:
- test_poisoned_tree_lock_returns_eio_not_panic
- test_parking_lot_rwlock_survives_panic
- test_panic_hook_logs_to_tracing
- test_systemd_service_has_execstoppost
- test_stale_mount_check_function_exists
- test_sd_notify_ready_sent
- test_sigterm_triggers_shutdown
Phase 1 of resilience testing design doc implementation:
- New musicfs-test-utils crate with FaultyOrigin, FaultyCasStore, fixtures
- Failpoints instrumented in musicfs-cas/store.rs
- 16 resilience tests (13 RED for missing features, 3 GREEN for existing)
- 3 Docker/Toxiproxy network tests (RED until docker-compose up)
- docker-compose.yml for Toxiproxy + MinIO + SFTP test infrastructure
Tests properly fail-first (TDD): check_all() sequential, no health timeout,
missing corruption detection, no passthrough mode, etc.
Replace original resilience-testing.md with BlueDoc-structured version.
All code examples from original preserved in Appendix A (17 sections).
Added: Abstract, Background, Goals/Non-Goals, Cross-Cutting Concerns,
Alternatives Considered (Jepsen, proptest, loom, mockall), phased
implementation plan with rollout order. Removed v2 suffix.
Covers: Nix dev shell, build/test commands, 11-crate workspace layout,
architecture essentials, code conventions, golden documents, documentation
rules (BlueDoc/GreenDoc templates + when to propose new templates),
current state with known critical issues, and key dependencies.
Comprehensive fault tolerance analysis covering 34 issues across 6 phases:
signal handling, crash recovery, cache corruption, network failures,
resource exhaustion, and the critical finding that no persistent state
is used on mount (every restart is a full origin rescan).
Persistent state plan covers storage engine options, mount flow redesign,
background delta sync, and the in-memory state inventory.
- Add tracing-appender and tracing-journald for production logging
- Add LoggingConfig with trace_sample_rate, json_output, journald options
- Expand init_logging() with file rotation, journald, and stderr layers
- Add sanitize_path() helper for PII protection in logs
- Instrument FUSE operations with #[instrument] and trace decision points
- Instrument gRPC handlers (10 methods) with span correlation
- Add spawn instrumentation for health monitor, indexer, watcher tasks
- Add broadcast lag handling (RecvError::Lagged) in event subscribers
- Fix webhook.rs expect() calls with proper error handling
- Add logging to patterns.rs, collections.rs, artwork.rs database ops
- Add Drop impl logging for PluginManager and WatchHandle
- Update systemd service with rate limiting and journal output
- Add logrotate config and example config.toml with logging section
Week 10 - Plugin System (FR-19):
- Plugin traits: Plugin, OriginPlugin, MetadataPlugin, FormatPlugin
- NativePluginHost with libloading for dynamic loading
- WasmPluginHost (feature-gated) with wasmtime runtime
- PluginManager coordinating both hosts with version checks
- OriginInstance::watch() with WatchHandle, WatchEvent for live updates
- FormatPlugin::synthesize_header() for metadata overlay
Week 11 - Control API & Production (FR-17, FR-18, NFR-6, NFR-10):
- gRPC server with full MusicFS service (status, cache, origins, events)
- Proto extended: MountState enum, TierStats, full StatusResponse/CacheStats
- WebhookHandler with HMAC-SHA256 signing and exponential retry
- Metrics with latency histograms (p50/p95/p99) and origin health gauges
- CLI with mount, status, cache, search, origin, events, shutdown commands
- E2E player compatibility tests (mpv, VLC, file manager)
- systemd service, PKGBUILD, RPM spec for packaging
Plans added for Weeks 10-14 covering P1 features.
All 154 tests passing.
- Proto definitions for SearchRequest/SearchResponse/SearchResult
- SearchService implementing MusicFs trait
- Query validation (empty check, 256 char max, limit cap at 10k)
- Streaming support via tokio mpsc channel
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- SearchOps with moka LRU cache for results and inode mapping
- 256 char query length limit with truncation
- Safe symlink targets with path normalization
- Integrates with filesystem.rs for lookup/getattr/readdir/readlink
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- SearchIndex with Arc<RwLock<IndexWriter>> pattern
- Fuzzy query support via custom term~N parsing
- Indexer with EventBus integration and MetadataLookup trait
- SearchQueryBuilder for programmatic query construction
- remove_by_path fallback for FileRemoved handling
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- Add CdcChunker using FastCDC v3 (16KB/64KB/256KB chunks)
- Add DeltaDetector with scan_origin() returning ScannedFile (no FileId assignment)
- Add OriginWatcher with inotify and 200ms debounce using tokio::spawn
- Fix LocalOrigin::read() to loop until all bytes read
- Add read_full() method to Origin trait
- Add mtime field to ChunkManifest
- Update ContentFetcher to use CDC chunking
- Update bandwidth reduction test to assert >90% (NFR-6.4)
Tests: 71 pass (+11 new)
Alexander