MusicFS/AGENTS.md

# AGENTS.md — beetfs/musicfs

> Everything an AI agent needs to get work done on this project.

---

## Quick Start

```bash
cd beetfs/musicfs
nix develop                    # Enter dev shell (ALL tooling provided)
cargo check                    # Verify compilation
cargo test                     # Run all tests (162 tests, ~10s)
cargo build --release          # Build release binary
```

**No `rustup`, no `apt install`, no manual dependency management.** The Nix flake provides everything: Rust stable + rust-analyzer + clippy + rustfmt, FUSE3, SQLite, OpenSSL, protobuf, grpcurl, cargo-nextest, cargo-criterion, lld linker.

---

## Project Overview

MusicFS is a **read-only FUSE filesystem** that presents music libraries organized by metadata (artist/album/track) instead of physical file paths. It supports multiple storage backends (local, NFS, S3, SFTP), content-addressable caching with delta sync, and full-text search.

**Key constraint**: Read-only. Never modifies origin files. Never pushes changes to the origin server.

---

## Repository Layout

```
beetfs/
├── musicfs/                        # Rust implementation (active)
│   ├── Cargo.toml                  # Workspace root
│   ├── flake.nix                   # Nix dev shell
│   ├── .cargo/config.toml          # LLD linker, aliases (t/c/b)
│   ├── crates/                     # 11 workspace crates
│   │   ├── musicfs-cli/            # Binary entry point (clap)
│   │   ├── musicfs-core/           # Types, errors, config, events
│   │   ├── musicfs-fuse/           # FUSE ops (fuser)
│   │   ├── musicfs-metadata/       # Audio parsing (symphonia)
│   │   ├── musicfs-cache/          # Cache: tree, metadata, patterns, eviction
│   │   ├── musicfs-cas/            # Content-addressable store (sled + chunks)
│   │   ├── musicfs-origins/        # Origin backends: local, NFS, SMB, S3, SFTP
│   │   ├── musicfs-sync/           # Delta sync, CDC chunking (fastcdc), watcher
│   │   ├── musicfs-search/         # Full-text search (tantivy)
│   │   ├── musicfs-grpc/           # gRPC control API (tonic + prost)
│   │   └── musicfs-plugins/        # Plugin system (native .so + WASM)
│   ├── tests/
│   │   ├── e2e/e2e_players.rs      # E2E: mpv/VLC playback (manual, #[ignore])
│   │   └── integration/            # (placeholder)
│   └── dist/                       # Deployment
│       ├── musicfs.service          # systemd unit
│       ├── config.example.toml      # Example config
│       ├── logrotate.d/musicfs      # Log rotation
│       ├── PKGBUILD                 # Arch package
│       └── musicfs.spec             # RPM spec
├── docs/
│   ├── templates/
│   │   ├── bluedoc.md              # Full design doc (5-20+ pages)
│   │   └── greendoc.md             # One-pager (1-2 pages)
│   ├── v2/
│   │   ├── architecture.md         # System design (GOLDEN — source of truth)
│   │   ├── requirements.md         # Functional + non-functional requirements
│   │   ├── development-plan.md     # Implementation roadmap (weeks 1-14)
│   │   └── plans/                  # Weekly plans, feature plans, research
│   └── v1/                         # Original Python beetfs docs (reference only)
└── beetsplug/beetFs.py             # Original Python implementation (archived)
```

---

## Build & Test Commands

```bash
# Cargo aliases (.cargo/config.toml)
cargo t                            # cargo test
cargo c                            # cargo check
cargo b                            # cargo build

# Common workflows
cargo check                        # Fast compile check
cargo test                         # All tests
cargo test -p musicfs-core         # Single crate
cargo clippy                       # Lint
cargo fmt                          # Format
cargo nextest run                  # Parallel test runner (faster)

# gRPC
cargo build -p musicfs-grpc        # Triggers proto codegen via build.rs
grpcurl -unix /run/musicfs.sock musicfs.v1.MusicFS/GetStatus

# Watch mode
cargo watch -x 'check' -x 'test'

# Release
cargo build --release
```

**Proto file location**: `crates/musicfs-grpc/proto/musicfs.proto` (codegen via `tonic-build` in `build.rs`)

---

## Architecture Essentials

### Crate Dependency Graph

```
                    musicfs-cli
                         │
              ┌──────────┼──────────┐
              │          │          │
              ▼          ▼          ▼
        musicfs-grpc  musicfs-fuse  musicfs-search
              │          │               │
              └────┬─────┴───────────────┘
                   │
                   ▼
             musicfs-core
              /    |    \
             /     |     \
            ▼      ▼      ▼
    musicfs-cache  musicfs-origins  musicfs-metadata
         │              │
         ▼              │
    musicfs-cas ◄───────┘
         │
         ▼
    musicfs-sync
```

### Core Concepts

| Concept | What | Where |
|---------|------|-------|
| **Virtual Tree** | In-memory directory structure from metadata | `musicfs-cache/src/tree.rs` |
| **CAS** | Content-addressable chunk storage (xxHash64 + sled index) | `musicfs-cas/src/store.rs` |
| **Origins** | Storage backends with failover + health monitoring | `musicfs-origins/src/` |
| **CDC** | Content-defined chunking for delta sync (FastCDC) | `musicfs-sync/src/cdc.rs` |
| **Event Bus** | `tokio::broadcast` for cross-component notifications | `musicfs-core/src/events.rs` |

### Performance Targets (from requirements.md)

| Operation | Target | Max |
|-----------|--------|-----|
| Mount | <100ms | 500ms |
| `stat()` cached | <1ms | 5ms |
| `readdir()` cached | <10ms | 50ms |
| `read()` cached | <1ms | 5ms |
| Search (1M files) | <500ms | 1s |

---

## Code Conventions

### Rust

- **Edition**: 2021, **MSRV**: 1.75+
- **Linker**: LLD via clang (configured in `.cargo/config.toml`)
- **Error handling**: `thiserror` for library errors, `anyhow` for CLI
- **Async**: `tokio` runtime, `async-trait` for trait objects
- **Concurrency**: `parking_lot` for hot-path locks, `dashmap` for concurrent maps, `std::sync::RwLock` elsewhere
- **Logging**: `tracing` with structured fields (`#[instrument]`, `info!`, `debug!`, etc.)
- **Serialization**: `serde` + `toml` for config, `rmp-serde` (msgpack) for binary data, `prost` for protobuf

### Never Do

- `as any`, `@ts-ignore` equivalents — no `unsafe` without justification
- Empty `catch` / `let _ = result` on operations that can fail meaningfully
- Suppress type errors
- Commit secrets or credentials

### Testing Patterns

- **Fixtures**: `TempDir::new().unwrap()` for isolated storage (used in 29 files)
- **In-memory DB**: `Database::open_memory()` for fast SQLite tests
- **No mocking framework** — tests use real implementations with temp directories
- **Async tests**: `#[tokio::test]`
- **Helper functions**: `make_file_meta()`, `mock_health()` — currently duplicated per module

---

## Golden Documents

These are the authoritative references. All implementations must match them.

| Document | Path | Role |
|----------|------|------|
| **Architecture** | `docs/v2/architecture.md` | System design — THE source of truth |
| **Requirements** | `docs/v2/requirements.md` | What to build (FR-*, NFR-*) |
| **Development Plan** | `docs/v2/development-plan.md` | How to build it (week-by-week) |
| **Proto Definition** | `crates/musicfs-grpc/proto/musicfs.proto` | API contract |

If code contradicts architecture.md, the architecture doc wins (unless explicitly superseded by a newer plan document).

---

## Documentation Rules

### Templates

Two templates exist in `docs/templates/`:

| Template | When to Use | Length | Review Level |
|----------|-------------|--------|-------------|
| **BlueDoc** | New systems, major architecture changes, new services | 5-20+ pages | Cross-functional |
| **GreenDoc** | Bug fixes, small features, optimizations, config changes | 1-2 pages | Peer review |

**Decision rule**: If any GreenDoc section needs more than 3 paragraphs, upgrade to a BlueDoc.

### When Neither Template Fits

If the work doesn't fit BlueDoc or GreenDoc (e.g., research summaries, audit reports, testing strategies, runbooks):
1. **Stop** — do not force-fit content into wrong template
2. **Propose** a new template format to the user with: name, intended use case, suggested structure
3. **Get approval** before writing the document
4. Save approved template to `docs/templates/{name}.md` for future use

### Document Metadata

Every document MUST have at the top:

```markdown
**Date**: YYYY-MM-DD
**Status**: [Draft / In-Review / Approved / Shipped / Obsolete]
**Prerequisites**: [links to dependent docs]
```

BlueDoc additionally requires: Authors, Reviewers, Approvers.

### Writing Conventions

- **Tables**: Use for requirements mapping, deliverables tracking, comparisons
- **Code blocks**: Include for implementation examples, config samples, commands
- **Checklists**: `[ ]` for exit criteria and success metrics
- **Section numbering**: Hierarchical (1., 1.1, 1.2)
- **Cross-references**: Relative markdown links (`[architecture](../architecture.md)`)
- **Requirement tracing**: Reference FR-X.Y / NFR-X.Y from requirements.md
- **Diagrams**: PlantUML or Mermaid (architecture.md uses PlantUML)

### File Naming

```
docs/v2/plans/week-NN-{feature}.md        # Weekly implementation plans
docs/v2/plans/{feature}-{type}.md          # Feature plans, research, proposals
docs/v2/{topic}.md                         # Top-level docs (architecture, requirements)
docs/templates/{name}.md                   # Document templates
```

---

## Current State & Known Issues

### What's Implemented (Weeks 1-11)

- FUSE filesystem with local origin
- Metadata extraction (symphonia: FLAC, MP3, AAC, OGG, Opus)
- Virtual tree with configurable path templates
- CAS with CDC chunking and deduplication
- Multi-origin federation with failover and health monitoring
- NFS/SMB origin wrappers with retry logic
- Full-text search (tantivy) with `.search/` virtual directory
- Smart collections, artwork caching, predictive prefetch
- Plugin system (native + WASM)
- gRPC control API with event streaming
- Comprehensive tracing/logging with journald integration

### Critical Open Issues

Detailed in `docs/v2/plans/resilience-fault-tolerance.md` and `docs/v2/plans/persistent-state.md`:

1. **No persistent state on mount** — every restart does full origin scan (O(N) instead of O(1)). SQLite, tantivy, and manifests persist on disk but are never loaded.
2. **No signal handling** — SIGTERM kills the daemon instantly, no graceful shutdown
3. **No crash recovery** — corrupted cache = crash on startup, no repair
4. **FUSE↔tokio deadlock risk** — `block_on()` in sync FUSE callback can hang under load
5. **Fire-and-forget tasks** — background tasks (health monitor, watcher, indexer) not supervised
6. **RwLock poison** — single panic in a writer kills all FUSE operations

### S3/SFTP Origins

`s3.rs` and `sftp.rs` are **feature-gated stubs** (not implemented). The `Origin` trait and failover infrastructure work, but only `local`, `nfs`, and `smb` origins have real implementations.

---

## Running the Filesystem

```bash
# Development
nix develop
cargo build
./target/debug/musicfs mount /mnt/music --origin /path/to/music

# Production (systemd)
sudo cp dist/musicfs.service /etc/systemd/system/
sudo systemctl enable --now musicfs

# E2E tests (requires mounted filesystem)
MUSICFS_TEST_MOUNT=/mnt/music cargo test --test e2e_players -- --ignored
```

---

## Key Dependencies

| Crate | Version | Purpose |
|-------|---------|---------|
| `fuser` | 0.14 | FUSE interface |
| `tokio` | 1.x | Async runtime (full features) |
| `rusqlite` | 0.31 | SQLite (bundled) |
| `sled` | 0.34 | Embedded KV (CAS chunk index) |
| `tantivy` | 0.22 | Full-text search |
| `symphonia` | 0.5 | Audio metadata extraction |
| `fastcdc` | 3.x | Content-defined chunking |
| `tonic` | 0.11 | gRPC server |
| `tracing` | 0.1 | Structured logging |
| `clap` | 4.x | CLI argument parsing |
| `parking_lot` | 0.12 | Fast locks |
| `dashmap` | 5.x | Concurrent HashMap |