Commit Graph

94 Commits

Author SHA1 Message Date
Alexander d49018bfc7 Implement write method to support metadata update 2026-06-27 00:16:04 +02:00
Alexander 9737e26bc9 Read and store flac metadata 2026-06-26 23:27:33 +02:00
Alexander 4f52549e46 Add rename of the files with keeping the state 2026-06-26 20:54:01 +02:00
Alexander 68aba52362 Handle files as tree on fuse side 2026-06-26 20:25:33 +02:00
Alexander a5023a6441 Integrate with db 2026-06-26 19:53:50 +02:00
Alexander 3d1f9a206f Hash the source files diff 2026-06-26 17:56:56 +02:00
Alexander 536e117576 Parse music metadata 2026-06-25 21:04:47 +02:00
Alexander b51c680af4 Handle updates of source files 2026-06-24 21:56:12 +02:00
Alexander f84061a30d Flat representation of localsource 2026-06-20 17:49:35 +02:00
Alexander 9349d29200 Start implementing LocalSource 2026-06-20 13:36:15 +02:00
Alexander 872580602d Added read files + lookup 2026-06-19 21:37:32 +02:00
Alexander 50cb21b9e5 Implement read and stat for dir 2026-06-19 18:45:08 +02:00
Alexander b1f32372f7 Migrate to denenv 2026-06-19 00:07:35 +02:00
Alexander 83ff042b91 Start anew 2026-06-18 23:15:11 +02:00
Alexander 05b83364df feat: add NixOS module
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-20 12:38:04 +02:00
Alexander 154f85bd9b chore(flake): add embedme to dev shell and pre-commit hooks
Keeps README code blocks in sync with source files (config.example.toml, dist/musicfs.service) on every commit.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-18 13:43:08 +02:00
Alexander 61457e1f89 docs: add comprehensive project README
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-18 13:43:03 +02:00
Alexander 4a1b68981e Forgotten fixes 2026-05-18 13:31:31 +02:00
Alexander b88583707d feat: add metadata enrichment integration with music-agregator
- Add SyncedFile message and subdir scoping to RescanOrigin proto
- Add label, album_type, cover_url fields to UpdateMetadataRequest/MetadataResponse
- Implement OriginScanner: walk, hash, diff, ingest with live FUSE tree and content fetcher registration
- Add enrichment DB columns: enrichment_source, enriched_at, enrichment_attempts, genres_json, label, album_type, cover_url
- Add EnrichmentUpdate struct and update_enrichment DB method
- Wire BatchUpdateMetadata to write enrichment fields alongside audio metadata
- Wire gRPC server into CLI mount command with --grpc-port flag
- Pass VirtualTree and ContentFetcher to scanner so rescanned files are immediately visible and readable via FUSE
2026-05-17 23:32:18 +02:00
Alexander 18024dbc62 fix(cli): wire OverlayReader into mount command
The metadata overlay feature was implemented but not connected to the
CLI daemon. Files were being served with original metadata instead of
synthesized headers from the database.

- Import OverlayReader from musicfs-cache
- Create OverlayReader with db, format_registry, and reader
- Call .with_overlay() on MusicFs builder

Tested: ffprobe now shows modified metadata from database updates.
2026-05-17 18:23:15 +02:00
Alexander b0c41e3fa0 feat(cli): add metadata subcommands for overlay management
- Add 6 subcommands: get, set, clear, diff, import, export
- Connect to gRPC MetadataService
- Support JSON and CSV formats
- All subcommands functional, help output correct
2026-05-17 17:59:35 +02:00
Alexander 1a7f70ae1c feat(grpc): implement MetadataService handlers
- Implement all 5 RPCs (Get, Update, Clear, Batch, Import)
- Add MetadataServiceImpl with database integration
- Add 10 comprehensive unit tests
- All 19 tests pass, full workspace compiles
2026-05-17 17:53:44 +02:00
Alexander 391f556286 feat(grpc): add MetadataService proto definition
- Add MetadataService with 5 RPCs (Get, Update, Clear, Batch, Import)
- Add 11 message types for requests/responses
- Use optional fields and map for custom_tags
- Proto codegen successful, all tests pass
2026-05-17 17:46:53 +02:00
Alexander 9623644263 feat(fuse): integrate OverlayReader in read path
- Update read() to use OverlayReader when available
- Map OverlayError to libc error codes
- Maintain 30s timeout and backward compatibility
- Fallback to FileReader for non-overlay files
- All tests pass, full workspace compiles
2026-05-17 17:44:29 +02:00
Alexander 487b119935 feat(fuse): return virtual size in getattr for overlay files
- Add overlay_reader field to MusicFs struct
- Add with_overlay() builder method
- Update getattr() to call estimate_virtual_size()
- Graceful fallback to original size
- All tests pass, backward compatible
2026-05-17 17:41:50 +02:00
Alexander c826bcf35f feat(cache): implement OverlayReader for header/audio splice
- Implement three-region splice logic (header, audio, boundary)
- Add passthrough mode for files without format_layout
- Add estimate_virtual_size() for getattr
- Create OverlayError enum with proper error conversions
- Add 8 comprehensive unit tests
- All tests pass, LSP diagnostics clean
2026-05-17 17:38:03 +02:00
Alexander ebf4044a01 feat(sync): populate format_layout during origin scan
- Create FormatHandlerRegistry in CLI initialization
- Register Id3v2Handler and FlacHandler
- Add analyze_format_layout() helper to read file headers
- Update scan functions to call handler.analyze()
- Use upsert_file_with_layout() when format_layout available
- Graceful degradation for unsupported formats
- Full workspace compiles successfully
2026-05-17 17:31:34 +02:00
Alexander 4f4a4169f8 feat(cache): update database layer for expanded metadata
- Update upsert_file() to include all 26 new AudioMeta fields
- Update get_file_by_virtual_path() to read all new columns
- Add get_file_metadata_row() for overlay synthesis
- Add update_metadata() for partial metadata updates
- Add clear_overlay() to reset metadata to NULL
- Handle format_layout BLOB with msgpack serialization
- Handle custom_tags JSON with serde_json
- Add 8 comprehensive unit tests
- All 92 tests pass, LSP diagnostics clean
2026-05-17 17:27:24 +02:00
Alexander 84bbd8f630 feat(cache): implement FlacHandler for FLAC metadata synthesis
- Implement all 8 FormatHandler trait methods
- Parse FLAC metadata blocks and extract STREAMINFO
- Preserve original STREAMINFO in synthesized headers
- Map all 36 AudioMeta fields to Vorbis comment tags
- Binary serialization of Vorbis comments with little-endian lengths
- Add 16 comprehensive unit tests including STREAMINFO preservation
- All tests pass, LSP diagnostics clean
2026-05-17 17:21:11 +02:00
Alexander 128a6e079e feat(cache): implement Id3v2Handler for MP3 metadata synthesis
- Implement all 8 FormatHandler trait methods
- Use lofty 0.24 for ID3v2.4 tag creation/parsing
- Map all 36 AudioMeta fields to ID3v2 frames
- Handle ID3v2 header parsing for audio_start
- Detect ID3v1 tags at EOF for audio_end
- Add 13 comprehensive unit tests
- Fix test-utils AudioMeta construction with ..Default::default()
- All tests pass, LSP diagnostics clean
2026-05-17 17:14:23 +02:00
Alexander 693b4f067b chore: add .sisyphus/ to gitignore 2026-05-17 15:44:31 +02:00
Alexander 66cd4e945c feat(fuse): implement rm with virtual .trash/ directory
- Add trashed/original_path/trashed_at columns to files table
- Implement FUSE unlink: moves files to /.trash/ preserving path structure
- Implement FUSE rmdir: removes empty directories
- Add trash CLI commands: list, restore, empty
- Add SIGHUP handler for CLI-triggered restore
- Fix upsert_file returning 0 on UPDATE (query actual ID)
- Auto-clear trashed flag when moving files out of /.trash/
2026-05-17 15:44:31 +02:00
Alexander 9d74f1a7a3 feat(fuse): implement mkdir and mv with persistence
Add mkdir and mv (rename) FUSE operations to the virtual filesystem:

- mkdir: Create directories that persist across remounts via SQLite
- mv: Move/rename files and directories with database persistence

Changes:
- Add directories table to schema for user-created empty dirs
- Add tree operations: mkdir, rename_file, rename_directory
- Add DB methods for path updates and directory CRUD
- Remove MountOption::RO to allow write syscalls
- Load stored virtual_path from DB instead of regenerating
- Restore user directories on mount from directories table
- Upsert files to DB during origin scan

POSIX compliant: mv fails with ENOENT if parent doesn't exist
(use mkdir first, shell handles -p flag and brace expansion)
2026-05-17 15:44:27 +02:00
Alexander 6e20ffe939 Make mount point optional when config file provides it
- CLI mountpoint argument is now Option<PathBuf>
- Falls back to config.mount_point when --config is provided
- CLI mountpoint still overrides config if both are given
- Expanded config.example.toml with all available options
2026-05-17 13:55:41 +02:00
Alexander daffd518d1 Update flake 2026-05-17 13:44:20 +02:00
Alexander a705d4d3b9 Add opencode 2026-05-17 13:43:12 +02:00
Alexander e4bf557151 Fix the nix package build 2026-05-13 23:22:26 +02:00
Alexander 39622be117 Package the app with nix 2026-05-13 22:17:01 +02:00
Alexander 265f4958f0 Implement configu use 2026-05-13 21:50:25 +02:00
Alexander 305d027c8b Move the files around 2026-05-13 20:34:14 +02:00
Alexander 90e9683076 Add persistent state implementation plan (SQLite)
Decision: SQLite (Option A) — existing schema, CRUD, row mapping,
and chunk_manifest column are already built but not wired into mount.

8-day plan to transform mount from O(N×origin_latency) to O(N×SQLite_read):
1. Database bulk load + manifest CRUD methods
2. Rewrite run_mount() with DB-load vs first-mount-scan paths
3. Persist chunk manifests via ManifestCached event
4. Wire tantivy + PatternStore + CollectionStore into mount
5. Background delta sync (origin vs DB reconciliation)
6. Shutdown WAL checkpoint
7-8. Integration testing + buffer
2026-05-13 16:02:25 +02:00
Alexander 0ff2a17ab7 Implement Phase C: Production Hardening
Implements phase-c-hardening.md to fix 6 RED resilience tests:

- D1/D2: Health check timeout (1.5s) + parallel execution via join_all
- C6: Recursive CAS calculate_size() to scan shard subdirectories
- C7: FUSE read timeout (30s) returns EIO instead of hanging
- 6.4: Auto-re-fetch corrupt/missing chunks from origin
- 6.6: Passthrough mode - continue even when CAS write fails
- C9: PID file with flock prevents concurrent mounts
- 5.3: fd exhaustion handling test

All 27 resilience tests now pass. Full test suite green.

Files changed:
- musicfs-origins/src/health.rs: timeout + join_all
- musicfs-origins/Cargo.toml: add futures dependency
- musicfs-cas/src/store.rs: recursive calculate_size
- musicfs-cas/src/reader.rs: auto-re-fetch on IntegrityError/NotFound
- musicfs-cas/src/fetcher.rs: passthrough fallback
- musicfs-fuse/src/filesystem.rs: 30s read timeout
- musicfs-cli/src/main.rs: PID file with flock
- musicfs-test-utils/tests/resilience.rs: updated tests
2026-05-13 15:55:22 +02:00
Alexander 3038c94b8c Add Phase C implementation plan (Production Hardening)
Merges practical items from resilience Phases C+D+E+F into one pass.
Turns all 6 remaining RED tests GREEN:
- D1/D2: Health check timeout + parallel join_all
- C6: Fix recursive CAS calculate_size()
- C7: FUSE read 30s timeout wrapper
- 6.4: Auto-re-fetch corrupt/missing chunks from origin
- 6.6: Passthrough fallback when CAS write fails
- C9: PID file with flock
- 5.3: fd exhaustion graceful handling
~4 days estimated.
2026-05-13 15:42:18 +02:00
Alexander 5da96ffab2 Implement Phase B: Crash Recovery
Add startup integrity checks, corruption recovery, CAS size limits,
graceful shutdown orchestration, and a task supervisor — turning 5
previously-RED resilience tests GREEN and adding 5 new tests.

- CAS: pre-check size limit in put(), add StoreFull error variant
- CAS: sled corruption recovery in open() (retry then recreate)
- SQLite: open_with_integrity_check() via PRAGMA integrity_check(1)
- tantivy: open_with_recovery() deletes and rebuilds corrupt index
- CLI: CancellationToken-based ordered shutdown sequence
- Core: TaskSupervisor with spawn_supervised/spawn_critical + backoff
- Tests: replace 4 todo!() stubs, add 5 new shutdown/supervisor tests
2026-05-13 15:33:23 +02:00
Alexander 4e394c60ec Add Phase B implementation plan (Crash Recovery)
BlueDoc covering 6 issues with TDD flow:
- 2.8: CAS size pre-check (StoreFull error variant)
- 2.4: SQLite PRAGMA integrity_check on open
- 2.4: tantivy open_with_recovery (detect + rebuild)
- 3.5: sled corruption repair + fallback recreate
- 2.3: Graceful shutdown with CancellationToken
- 2.6: TaskSupervisor (monitor, detect panic, restart)
Turns 5 RED tests GREEN, adds 4 new tests. ~5 days.
2026-05-13 14:56:43 +02:00
Alexander 6285eeb6c0 Implement Phase A: Stop Dying resilience fixes
Implements all 6 critical resilience fixes from phase-a-stop-dying.md:

- Issue 2.9: Migrate std::sync::RwLock → parking_lot::RwLock (7 files)
  Prevents lock poisoning cascade on writer panic

- Issue 2.2: Add install_panic_hook() to log panics via tracing
  Ensures panics are captured in logs/journald before process death

- Issue 3.7: Add ExecStopPost to systemd service
  Cleans up stale FUSE mounts on service stop

- Issue 2.7: Add check_stale_mount() detection on startup
  Auto-cleans leftover mounts from previous crashes

- Issue 2.10: Integrate sd_notify for systemd lifecycle
  Sends READY=1 after mount, STOPPING on shutdown

- Issue 2.1: Add signal handling with spawn_mount
  Catches SIGTERM/SIGINT for clean shutdown instead of instant death

All 7 Phase A tests pass:
- test_poisoned_tree_lock_returns_eio_not_panic
- test_parking_lot_rwlock_survives_panic
- test_panic_hook_logs_to_tracing
- test_systemd_service_has_execstoppost
- test_stale_mount_check_function_exists
- test_sd_notify_ready_sent
- test_sigterm_triggers_shutdown
2026-05-13 14:48:32 +02:00
Alexander 24086cc744 Add Phase A implementation plan (Stop Dying)
BlueDoc covering 6 critical resilience fixes with TDD flow:
- 2.9: RwLock → parking_lot (poison-free locks)
- 2.2: Panic hook with tracing integration
- 3.7+2.7: systemd ExecStopPost + stale mount cleanup
- 2.10: sd_notify READY/STOPPING
- 2.1: Signal handling via spawn_mount2 + tokio signals
Each with: stubs → RED tests → implementation → GREEN verify.
~5 days estimated, exact files and code patterns specified.
2026-05-13 14:00:46 +02:00
Alexander e3eeba4650 Add musicfs-test-utils crate with RED resilience tests
Phase 1 of resilience testing design doc implementation:
- New musicfs-test-utils crate with FaultyOrigin, FaultyCasStore, fixtures
- Failpoints instrumented in musicfs-cas/store.rs
- 16 resilience tests (13 RED for missing features, 3 GREEN for existing)
- 3 Docker/Toxiproxy network tests (RED until docker-compose up)
- docker-compose.yml for Toxiproxy + MinIO + SFTP test infrastructure

Tests properly fail-first (TDD): check_all() sequential, no health timeout,
missing corruption detection, no passthrough mode, etc.
2026-05-13 13:49:25 +02:00
Alexander 00f14930cd Consolidate resilience testing into BlueDoc format
Replace original resilience-testing.md with BlueDoc-structured version.
All code examples from original preserved in Appendix A (17 sections).
Added: Abstract, Background, Goals/Non-Goals, Cross-Cutting Concerns,
Alternatives Considered (Jepsen, proptest, loom, mockall), phased
implementation plan with rollout order. Removed v2 suffix.
2026-05-13 12:54:20 +02:00
Alexander c6aa47f440 Add resilience testing BlueDoc (v2)
Restructured resilience testing strategy into BlueDoc template format
with proper sections: Abstract, Background, Goals/Non-Goals, Proposed
Design, Cross-Cutting Concerns, Alternatives Considered, Implementation
Plan, and Glossary. Original resilience-testing.md preserved.
2026-05-13 12:46:25 +02:00