MusicFS/AGENTS.md

AGENTS.md — beetfs/musicfs

Everything an AI agent needs to get work done on this project.

---

Quick Start

cd beetfs/musicfs
nix develop                    # Enter dev shell (ALL tooling provided)
cargo check                    # Verify compilation
cargo test                     # Run all tests (162 tests, ~10s)
cargo build --release          # Build release binary

No rustup, no apt install, no manual dependency management. The Nix flake provides everything: Rust stable + rust-analyzer + clippy + rustfmt, FUSE3, SQLite, OpenSSL, protobuf, grpcurl, cargo-nextest, cargo-criterion, lld linker.

---

Project Overview

MusicFS is a read-only FUSE filesystem that presents music libraries organized by metadata (artist/album/track) instead of physical file paths. It supports multiple storage backends (local, NFS, S3, SFTP), content-addressable caching with delta sync, and full-text search.

Key constraint: Read-only. Never modifies origin files. Never pushes changes to the origin server.

---

Repository Layout

beetfs/
├── musicfs/                        # Rust implementation (active)
│   ├── Cargo.toml                  # Workspace root
│   ├── flake.nix                   # Nix dev shell
│   ├── .cargo/config.toml          # LLD linker, aliases (t/c/b)
│   ├── crates/                     # 11 workspace crates
│   │   ├── musicfs-cli/            # Binary entry point (clap)
│   │   ├── musicfs-core/           # Types, errors, config, events
│   │   ├── musicfs-fuse/           # FUSE ops (fuser)
│   │   ├── musicfs-metadata/       # Audio parsing (symphonia)
│   │   ├── musicfs-cache/          # Cache: tree, metadata, patterns, eviction
│   │   ├── musicfs-cas/            # Content-addressable store (sled + chunks)
│   │   ├── musicfs-origins/        # Origin backends: local, NFS, SMB, S3, SFTP
│   │   ├── musicfs-sync/           # Delta sync, CDC chunking (fastcdc), watcher
│   │   ├── musicfs-search/         # Full-text search (tantivy)
│   │   ├── musicfs-grpc/           # gRPC control API (tonic + prost)
│   │   └── musicfs-plugins/        # Plugin system (native .so + WASM)
│   ├── tests/
│   │   ├── e2e/e2e_players.rs      # E2E: mpv/VLC playback (manual, #[ignore])
│   │   └── integration/            # (placeholder)
│   └── dist/                       # Deployment
│       ├── musicfs.service          # systemd unit
│       ├── config.example.toml      # Example config
│       ├── logrotate.d/musicfs      # Log rotation
│       ├── PKGBUILD                 # Arch package
│       └── musicfs.spec             # RPM spec
├── docs/
│   ├── templates/
│   │   ├── bluedoc.md              # Full design doc (5-20+ pages)
│   │   └── greendoc.md             # One-pager (1-2 pages)
│   ├── v2/
│   │   ├── architecture.md         # System design (GOLDEN — source of truth)
│   │   ├── requirements.md         # Functional + non-functional requirements
│   │   ├── development-plan.md     # Implementation roadmap (weeks 1-14)
│   │   └── plans/                  # Weekly plans, feature plans, research
│   └── v1/                         # Original Python beetfs docs (reference only)
└── beetsplug/beetFs.py             # Original Python implementation (archived)

---

Build & Test Commands

# Cargo aliases (.cargo/config.toml)
cargo t                            # cargo test
cargo c                            # cargo check
cargo b                            # cargo build

# Common workflows
cargo check                        # Fast compile check
cargo test                         # All tests
cargo test -p musicfs-core         # Single crate
cargo clippy                       # Lint
cargo fmt                          # Format
cargo nextest run                  # Parallel test runner (faster)

# gRPC
cargo build -p musicfs-grpc        # Triggers proto codegen via build.rs
grpcurl -unix /run/musicfs.sock musicfs.v1.MusicFS/GetStatus

# Watch mode
cargo watch -x 'check' -x 'test'

# Release
cargo build --release

Proto file location: crates/musicfs-grpc/proto/musicfs.proto (codegen via tonic-build in build.rs)

---

Architecture Essentials

Crate Dependency Graph

                    musicfs-cli
                         │
              ┌──────────┼──────────┐
              │          │          │
              ▼          ▼          ▼
        musicfs-grpc  musicfs-fuse  musicfs-search
              │          │               │
              └────┬─────┴───────────────┘
                   │
                   ▼
             musicfs-core
              /    |    \
             /     |     \
            ▼      ▼      ▼
    musicfs-cache  musicfs-origins  musicfs-metadata
         │              │
         ▼              │
    musicfs-cas ◄───────┘
         │
         ▼
    musicfs-sync

Core Concepts

Concept	What	Where
Virtual Tree	In-memory directory structure from metadata	musicfs-cache/src/tree.rs
CAS	Content-addressable chunk storage (xxHash64 + sled index)	musicfs-cas/src/store.rs
Origins	Storage backends with failover + health monitoring	musicfs-origins/src/
CDC	Content-defined chunking for delta sync (FastCDC)	musicfs-sync/src/cdc.rs
Event Bus	tokio::broadcast for cross-component notifications	musicfs-core/src/events.rs

Performance Targets (from requirements.md)

Operation	Target	Max
Mount	<100ms	500ms
stat() cached	<1ms	5ms
readdir() cached	<10ms	50ms
read() cached	<1ms	5ms
Search (1M files)	<500ms	1s

---

Code Conventions

Rust

• Edition: 2021, MSRV: 1.75+
• Linker: LLD via clang (configured in .cargo/config.toml)
• Error handling: thiserror for library errors, anyhow for CLI
• Async: tokio runtime, async-trait for trait objects
• Concurrency: parking_lot for hot-path locks, dashmap for concurrent maps, std::sync::RwLock elsewhere
• Logging: tracing with structured fields (#[instrument], info!, debug!, etc.)
• Serialization: serde + toml for config, rmp-serde (msgpack) for binary data, prost for protobuf

Never Do

• as any, @ts-ignore equivalents — no unsafe without justification
• Empty catch / let _ = result on operations that can fail meaningfully
• Suppress type errors
• Commit secrets or credentials

Testing Patterns

• Fixtures: TempDir::new().unwrap() for isolated storage (used in 29 files)
• In-memory DB: Database::open_memory() for fast SQLite tests
• No mocking framework — tests use real implementations with temp directories
• Async tests: #[tokio::test]
• Helper functions: make_file_meta(), mock_health() — currently duplicated per module

---

Golden Documents

These are the authoritative references. All implementations must match them.

Document	Path	Role
Architecture	docs/v2/architecture.md	System design — THE source of truth
Requirements	docs/v2/requirements.md	What to build (FR-, NFR-)
Development Plan	docs/v2/development-plan.md	How to build it (week-by-week)
Proto Definition	crates/musicfs-grpc/proto/musicfs.proto	API contract

If code contradicts architecture.md, the architecture doc wins (unless explicitly superseded by a newer plan document).

---

Documentation Rules

Templates

Two templates exist in docs/templates/:

Template	When to Use	Length	Review Level
BlueDoc	New systems, major architecture changes, new services	5-20+ pages	Cross-functional
GreenDoc	Bug fixes, small features, optimizations, config changes	1-2 pages	Peer review

Decision rule: If any GreenDoc section needs more than 3 paragraphs, upgrade to a BlueDoc.

When Neither Template Fits

If the work doesn't fit BlueDoc or GreenDoc (e.g., research summaries, audit reports, testing strategies, runbooks):

1. Stop — do not force-fit content into wrong template
2. Propose a new template format to the user with: name, intended use case, suggested structure
3. Get approval before writing the document
4. Save approved template to docs/templates/{name}.md for future use

Document Metadata

Every document MUST have at the top:

**Date**: YYYY-MM-DD
**Status**: [Draft / In-Review / Approved / Shipped / Obsolete]
**Prerequisites**: [links to dependent docs]

BlueDoc additionally requires: Authors, Reviewers, Approvers.

Writing Conventions

• Tables: Use for requirements mapping, deliverables tracking, comparisons
• Code blocks: Include for implementation examples, config samples, commands
• Checklists: [ ] for exit criteria and success metrics
• Section numbering: Hierarchical (1., 1.1, 1.2)
• Cross-references: Relative markdown links ([architecture](../architecture.md))
• Requirement tracing: Reference FR-X.Y / NFR-X.Y from requirements.md
• Diagrams: PlantUML or Mermaid (architecture.md uses PlantUML)

File Naming

docs/v2/plans/week-NN-{feature}.md        # Weekly implementation plans
docs/v2/plans/{feature}-{type}.md          # Feature plans, research, proposals
docs/v2/{topic}.md                         # Top-level docs (architecture, requirements)
docs/templates/{name}.md                   # Document templates

---

Current State & Known Issues

What's Implemented (Weeks 1-11)

• FUSE filesystem with local origin
• Metadata extraction (symphonia: FLAC, MP3, AAC, OGG, Opus)
• Virtual tree with configurable path templates
• CAS with CDC chunking and deduplication
• Multi-origin federation with failover and health monitoring
• NFS/SMB origin wrappers with retry logic
• Full-text search (tantivy) with .search/ virtual directory
• Smart collections, artwork caching, predictive prefetch
• Plugin system (native + WASM)
• gRPC control API with event streaming
• Comprehensive tracing/logging with journald integration

Critical Open Issues

Detailed in docs/v2/plans/resilience-fault-tolerance.md and docs/v2/plans/persistent-state.md:

1. No persistent state on mount — every restart does full origin scan (O(N) instead of O(1)). SQLite, tantivy, and manifests persist on disk but are never loaded.
2. No signal handling — SIGTERM kills the daemon instantly, no graceful shutdown
3. No crash recovery — corrupted cache = crash on startup, no repair
4. FUSE↔tokio deadlock risk — block_on() in sync FUSE callback can hang under load
5. Fire-and-forget tasks — background tasks (health monitor, watcher, indexer) not supervised
6. RwLock poison — single panic in a writer kills all FUSE operations

S3/SFTP Origins

s3.rs and sftp.rs are feature-gated stubs (not implemented). The Origin trait and failover infrastructure work, but only local, nfs, and smb origins have real implementations.

---

Running the Filesystem

# Development
nix develop
cargo build
./target/debug/musicfs mount /mnt/music --origin /path/to/music

# Production (systemd)
sudo cp dist/musicfs.service /etc/systemd/system/
sudo systemctl enable --now musicfs

# E2E tests (requires mounted filesystem)
MUSICFS_TEST_MOUNT=/mnt/music cargo test --test e2e_players -- --ignored

---

Key Dependencies

Crate	Version	Purpose
fuser	0.14	FUSE interface
tokio	1.x	Async runtime (full features)
rusqlite	0.31	SQLite (bundled)
sled	0.34	Embedded KV (CAS chunk index)
tantivy	0.22	Full-text search
symphonia	0.5	Audio metadata extraction
fastcdc	3.x	Content-defined chunking
tonic	0.11	gRPC server
tracing	0.1	Structured logging
clap	4.x	CLI argument parsing
parking_lot	0.12	Fast locks
dashmap	5.x	Concurrent HashMap