Commit Graph

13 Commits

Author SHA1 Message Date
Alexander 5ac33987c0 Add comprehensive logging with tracing, file rotation, and systemd integration
- Add tracing-appender and tracing-journald for production logging
- Add LoggingConfig with trace_sample_rate, json_output, journald options
- Expand init_logging() with file rotation, journald, and stderr layers
- Add sanitize_path() helper for PII protection in logs
- Instrument FUSE operations with #[instrument] and trace decision points
- Instrument gRPC handlers (10 methods) with span correlation
- Add spawn instrumentation for health monitor, indexer, watcher tasks
- Add broadcast lag handling (RecvError::Lagged) in event subscribers
- Fix webhook.rs expect() calls with proper error handling
- Add logging to patterns.rs, collections.rs, artwork.rs database ops
- Add Drop impl logging for PluginManager and WatchHandle
- Update systemd service with rate limiting and journal output
- Add logrotate config and example config.toml with logging section
2026-05-13 11:21:51 +02:00
Alexander bc9fa36646 Add Week 10 Plugin System and Week 11 Control API
Week 10 - Plugin System (FR-19):
- Plugin traits: Plugin, OriginPlugin, MetadataPlugin, FormatPlugin
- NativePluginHost with libloading for dynamic loading
- WasmPluginHost (feature-gated) with wasmtime runtime
- PluginManager coordinating both hosts with version checks
- OriginInstance::watch() with WatchHandle, WatchEvent for live updates
- FormatPlugin::synthesize_header() for metadata overlay

Week 11 - Control API & Production (FR-17, FR-18, NFR-6, NFR-10):
- gRPC server with full MusicFS service (status, cache, origins, events)
- Proto extended: MountState enum, TierStats, full StatusResponse/CacheStats
- WebhookHandler with HMAC-SHA256 signing and exponential retry
- Metrics with latency histograms (p50/p95/p99) and origin health gauges
- CLI with mount, status, cache, search, origin, events, shutdown commands
- E2E player compatibility tests (mpv, VLC, file manager)
- systemd service, PKGBUILD, RPM spec for packaging

Plans added for Weeks 10-14 covering P1 features.
All 154 tests passing.
2026-05-13 10:34:01 +02:00
Alexander 34d05b7a49 Add Week 9 Smart Features: collections, artwork, predictive prefetch
Smart Collections (musicfs-search/src/collections.rs):
- CollectionStore with thread-safe Mutex<Connection>
- CollectionQuery enum: Match, DateRange, RecentlyAdded/Played, MostPlayed, Genre, Compound
- Builtin collections for Recently Added, 80s/90s Music

Artwork Extraction & Caching:
- ArtworkExtractor using symphonia Visual (musicfs-metadata)
- ArtworkCache with CAS storage + on-demand resize (musicfs-cache)
- ArtType: Front/Back/Other, ArtSize: Thumbnail/Medium/Full

Predictive Prefetching:
- PatternStore tracks access patterns with sequence prediction
- PrefetchEngine listens to FileAccessed events, prefetches predictions
- PrefetchOps exposes /.prefetch/ virtual directory with status/hints

Oracle review fixes applied:
- CollectionStore uses Mutex for thread safety
- FileAccessed event now includes file_id for canonical correlation
- JSON parse warnings in collection deserialization

130 tests pass (15 new tests added)
2026-05-13 07:21:28 +02:00
Alexander 6bae6ca67b Add tantivy, moka, tonic workspace dependencies for Week 8 search
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/claude-agent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-05-12 23:23:04 +02:00
Alexander 09f019730f Implement Week 7 Remote Origins with Oracle fixes
- Add credentials.rs with CredentialStore, redacted Debug (session_token shows [REDACTED])
- Add nfs.rs with ESTALE retry using Fn closure, 5s health timeout
- Add smb.rs with ENOTCONN retry handling, 5s health timeout
- Add s3.rs/sftp.rs feature-gated stubs with security documentation
- Add error variants: S3, Sftp, Timeout, Credential, NfsStaleHandle
- Fix delta.rs unused imports

Oracle fixes applied:
- SMB retry_on_disconnect for ENOTCONN (errno 107)
- session_token Debug shows [REDACTED] when Some, None otherwise
- NFS/SMB health checks wrapped with tokio::time::timeout(5s)

102 tests pass, 0 warnings.
2026-05-12 22:26:19 +02:00
Alexander d5ef68c9c9 Implement Week 6 Origin Federation with Oracle fixes
New files:
- musicfs-core/src/config.rs: Config, OriginConfig, HealthConfig
- musicfs-origins/src/registry.rs: OriginRegistry with watch cleanup
- musicfs-origins/src/router.rs: Priority router with (priority, latency) ordering
- musicfs-origins/src/health.rs: HealthMonitor with per-origin-type thresholds
- musicfs-origins/src/failover.rs: FailoverExecutor with NFR-7.3 backoff

Oracle fixes applied:
- Per-OriginType threshold: Local=1, Remote=3 (check_one uses threshold_for)
- AllOriginsUnhealthy event: Added to events.rs, emitted in select_with_fallback
- Unified OriginType: Removed duplicate from traits.rs, use musicfs_core::OriginType
- Watch handle cleanup: Tracked and dropped on unregister()
- Retry delays: 100ms, 500ms, 2000ms (NFR-7.3 compliant)

Tests: 91 pass (+20 new)
2026-05-12 20:15:56 +02:00
Alexander 32c96701c8 Implement Week 5 CDC & Delta Detection with Oracle fixes
- Add CdcChunker using FastCDC v3 (16KB/64KB/256KB chunks)
- Add DeltaDetector with scan_origin() returning ScannedFile (no FileId assignment)
- Add OriginWatcher with inotify and 200ms debounce using tokio::spawn
- Fix LocalOrigin::read() to loop until all bytes read
- Add read_full() method to Origin trait
- Add mtime field to ChunkManifest
- Update ContentFetcher to use CDC chunking
- Update bandwidth reduction test to assert >90% (NFR-6.4)

Tests: 71 pass (+11 new)
2026-05-12 20:05:44 +02:00
Alexander 7ad554f8d5 Add CLI implementation and MVP performance review
- Implement functional CLI with clap argument parsing
- Add directory scanning and metadata extraction at startup
- Fix filesystem.rs to store tokio Handle for async/sync bridge
- Fix flake.nix with LD_LIBRARY_PATH for libfuse3
- Add MVP performance review with real-world benchmark results

Benchmarks show:
- Mount time: 8ms (target <500ms)
- Throughput: 2-3 GB/s (target >500 MB/s)
- Identifies critical gap: incomplete file caching (only ~2MB per file)
- Identifies missing CDC chunking per architecture spec
2026-05-12 19:28:13 +02:00
Alexander c46750b1ec Implement Week 4b Origin-CAS connector for cache-miss handling
- Add ContentFetcher bridging Origin→CAS on cache miss
- Integrate fetcher into FileReader via with_fetcher() constructor
- Add get_or_fetch_manifest() for lazy manifest loading
- Emit FileAccessed events per FR-18.1 via EventBus
- Add 2 integration tests for e2e fetch flow
- Test count: 60 (was 54)
2026-05-12 19:04:48 +02:00
Alexander ffbb238633 Implement Week 4 CAS store with chunk deduplication and LRU eviction
- Add musicfs-cas crate: CasStore, ChunkHash, FileReader, ChunkManifest
- Add LruEviction policy to musicfs-cache for cache size management
- Integrate FileReader into FUSE filesystem for actual file reads
- Use xxHash64 for content hashing, sled for index, msgpack serialization
- Default cache path: ~/.cache/musicfs/chunks/ with 256 subdirs sharding
- 20 new tests (14 CAS unit + 3 integration + 3 eviction), 54 total
2026-05-12 18:43:39 +02:00
Alexander d9e5e06166 Implement Week 3 virtual tree with path resolver and FUSE integration 2026-05-12 18:25:24 +02:00
Alexander d664439746 Implement Week 2 metadata extraction and cache database
Week 1 fixes:
- Move hex to workspace dependencies
- Add cargo-criterion, protobuf, grpcurl to flake.nix

Week 2 implementation:
- musicfs-metadata: MetadataParser with symphonia 0.5 for FLAC, MP3,
  Opus/Vorbis, M4A/AAC (2 tests)
- musicfs-cache: SQLite schema per architecture 4.3.6 with track/disc
  columns, TEXT content_hash, all required indexes
- musicfs-cache/db.rs: Database with upsert, CRUD, mtime lookup (9 tests)
- musicfs-cache/metadata.rs: MetadataCache with store/lookup/is_fresh/
  invalidate (2 tests)
- musicfs-core: Added Metadata error variant

22 tests pass total. Oracle-verified against architecture doc.
2026-05-12 18:15:44 +02:00
Alexander 76856b893a Implement Week 1 foundation: workspace, core types, FUSE skeleton, LocalOrigin
- musicfs-core: OriginId, FileId, VirtualPath, ContentHash, AudioMeta,
  FileMeta, EventBus with FileAccessed event (5 tests)
- musicfs-fuse: FUSE skeleton with EROFS handlers for write ops
- musicfs-origins: Origin trait with watch(), LocalOrigin impl (6 tests)
- flake.nix: Nix dev shell with rust toolchain, clang, lld, fuse3

All 11 tests pass. Build produces no warnings.
2026-05-12 18:01:47 +02:00