Add Week 10 Plugin System and Week 11 Control API

Week 10 - Plugin System (FR-19):
- Plugin traits: Plugin, OriginPlugin, MetadataPlugin, FormatPlugin
- NativePluginHost with libloading for dynamic loading
- WasmPluginHost (feature-gated) with wasmtime runtime
- PluginManager coordinating both hosts with version checks
- OriginInstance::watch() with WatchHandle, WatchEvent for live updates
- FormatPlugin::synthesize_header() for metadata overlay

Week 11 - Control API & Production (FR-17, FR-18, NFR-6, NFR-10):
- gRPC server with full MusicFS service (status, cache, origins, events)
- Proto extended: MountState enum, TierStats, full StatusResponse/CacheStats
- WebhookHandler with HMAC-SHA256 signing and exponential retry
- Metrics with latency histograms (p50/p95/p99) and origin health gauges
- CLI with mount, status, cache, search, origin, events, shutdown commands
- E2E player compatibility tests (mpv, VLC, file manager)
- systemd service, PKGBUILD, RPM spec for packaging

Plans added for Weeks 10-14 covering P1 features.
All 154 tests passing.
This commit is contained in:
Alexander
2026-05-13 10:34:01 +02:00
parent 34d05b7a49
commit bc9fa36646
27 changed files with 7050 additions and 49 deletions
+21
View File
@@ -23,6 +23,21 @@ Implement audio metadata extraction using symphonia and create SQLite schema for
--- ---
## Task 0: Extend AudioMeta in `musicfs-core`
Add `lyrics` and `composer` fields to `AudioMeta` struct (FR-6.4):
```rust
// In musicfs-core/src/types.rs, add to AudioMeta:
pub struct AudioMeta {
// ... existing fields ...
pub lyrics: Option<String>,
pub composer: Option<String>,
}
```
---
## Task 1: Metadata Parser (`musicfs-metadata`) ## Task 1: Metadata Parser (`musicfs-metadata`)
### 1.1 Create `Cargo.toml` ### 1.1 Create `Cargo.toml`
@@ -168,6 +183,12 @@ impl MetadataParser {
meta.year = value.chars().take(4).collect::<String>() meta.year = value.chars().take(4).collect::<String>()
.parse().ok(); .parse().ok();
} }
StandardTagKey::Lyrics => {
meta.lyrics = Some(value);
}
StandardTagKey::Composer => {
meta.composer = Some(value);
}
_ => {} _ => {}
} }
} }
+179
View File
@@ -0,0 +1,179 @@
# Week 10: Plugin System
**Phase**: 4 - Plugin System & Polish
**Goal**: Extensibility via native and WASM plugins
**Requirements**: FR-23.1-23.5, FR-24.1-24.3
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Plugin traits | musicfs-plugins | `traits.rs` | FR-23.1-23.4 |
| Native host | musicfs-plugins | `native.rs` | FR-23.2 |
| WASM host | musicfs-plugins | `wasm.rs` | FR-23.3 |
| Plugin lifecycle | musicfs-plugins | `manager.rs` | FR-23.5 |
| Example plugins | plugins/ | `example-origin/`, `example-format/` | FR-23.5 |
---
## Plugin Traits (`musicfs-plugins/src/traits.rs`)
```rust
/// Base plugin interface
pub trait Plugin: Send + Sync {
fn name(&self) -> &str;
fn version(&self) -> Version;
fn init(&mut self, config: Value) -> Result<(), PluginError>;
fn shutdown(&mut self) -> Result<(), PluginError>;
}
/// Origin plugin interface (per architecture 4.3.4)
pub trait OriginPlugin: Plugin {
fn origin_type(&self) -> &str;
fn create(&self, config: Value) -> Result<Box<dyn Origin>, PluginError>;
}
/// Metadata source plugin
pub trait MetadataPlugin: Plugin {
fn lookup(&self, query: &MetadataQuery) -> Result<Option<ExternalMetadata>, PluginError>;
}
/// Format plugin for custom audio formats (FR-24.1)
pub trait FormatPlugin: Plugin {
fn extensions(&self) -> &[&str];
fn can_handle(&self, extension: &str) -> bool;
fn parse(&self, reader: &mut dyn Read) -> Result<AudioMeta, PluginError>;
}
```
---
## Native Plugin Host (`musicfs-plugins/src/native.rs`)
```rust
pub struct NativePluginHost {
plugins: HashMap<String, LoadedPlugin>,
search_paths: Vec<PathBuf>,
}
struct LoadedPlugin {
library: libloading::Library,
instance: Box<dyn Plugin>,
}
impl NativePluginHost {
pub fn new() -> Self;
/// Load plugin from shared library (.so/.dylib)
pub fn load(&mut self, path: &Path) -> Result<PluginId, PluginError>;
/// Unload plugin (FR-23.5)
pub fn unload(&mut self, id: PluginId) -> Result<(), PluginError>;
/// Hot reload plugin without restart (FR-23.4)
pub fn reload(&mut self, id: PluginId) -> Result<(), PluginError>;
/// List loaded plugins
pub fn list(&self) -> Vec<PluginInfo>;
}
```
---
## WASM Plugin Host (`musicfs-plugins/src/wasm.rs`)
```rust
pub struct WasmPluginHost {
engine: wasmtime::Engine,
linker: wasmtime::Linker<PluginState>,
}
impl WasmPluginHost {
pub fn new() -> Result<Self, PluginError>;
/// Load WASM plugin with sandboxing (FR-23.3)
pub fn load(&mut self, wasm_bytes: &[u8]) -> Result<WasmPlugin, PluginError>;
/// Resource limits for sandboxed execution
pub fn set_limits(&mut self, limits: ResourceLimits);
}
pub struct ResourceLimits {
pub max_memory_mb: u32,
pub max_cpu_time_ms: u32,
pub allow_network: bool,
pub allow_filesystem: bool,
}
```
---
## Plugin Manager (`musicfs-plugins/src/manager.rs`)
```rust
pub struct PluginManager {
native_host: NativePluginHost,
wasm_host: WasmPluginHost,
registry: PluginRegistry,
}
impl PluginManager {
/// Initialize and load plugins from config
pub fn init(config: &PluginConfig) -> Result<Self, PluginError>;
/// Get all origin plugins
pub fn origin_plugins(&self) -> Vec<&dyn OriginPlugin>;
/// Get all format plugins
pub fn format_plugins(&self) -> Vec<&dyn FormatPlugin>;
/// Get all metadata plugins
pub fn metadata_plugins(&self) -> Vec<&dyn MetadataPlugin>;
/// Reload all plugins (hot reload)
pub fn reload_all(&mut self) -> Result<(), PluginError>;
}
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_native_plugin_load` | Unit | Native plugin loading (FR-23.2) |
| `test_native_plugin_unload` | Unit | Clean unload |
| `test_wasm_plugin_sandbox` | Unit | WASM isolation (FR-23.3) |
| `test_wasm_resource_limits` | Unit | Memory/CPU limits enforced |
| `test_plugin_hot_reload` | Integration | Reload without restart (FR-23.4) |
| `test_example_origin_plugin` | Integration | Custom origin works |
| `test_example_format_plugin` | Integration | Custom format works |
---
## Exit Criteria
- [ ] Native plugins loadable at runtime
- [ ] WASM plugins sandboxed with resource limits
- [ ] Example plugins functional
- [ ] Plugins hot-reloadable without daemon restart
- [ ] Plugin lifecycle management (load, unload, reload)
---
## Architecture Alignment
Per architecture.md section 4.3.4:
- Plugin loading: Built-in → Native (.so) → WASM
- Origin plugins create `Box<dyn Origin>`
- Format plugins register file extensions
- WASM runs in wasmtime sandbox
Per requirements.md:
- FR-23.1: Loadable plugins ✓
- FR-23.2: Stable plugin API ✓
- FR-23.3: Plugins for origins, metadata, formats ✓
- FR-23.4: WASM sandbox ✓
- FR-23.5: Plugin lifecycle ✓
+539
View File
@@ -0,0 +1,539 @@
# Week 11: Control API & Production
**Phase**: 4 - Plugin System & Polish
**Goal**: gRPC control API, metrics, and production readiness
**Requirements**: FR-17.1-17.5, FR-18.1-18.4, NFR-6.1-6.4, NFR-10.1-10.4
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| gRPC server | musicfs-grpc | `server.rs` | FR-17.1-17.5 |
| Proto codegen | proto/ | `musicfs.proto`, `build.rs` | FR-17.2 |
| Event streaming | musicfs-grpc | `events.rs` | FR-18.1-18.3 |
| Webhook handler | musicfs-grpc | `webhook.rs` | FR-18.2 |
| Metrics export | musicfs-core | `metrics.rs` | NFR-6.1-6.4, NFR-10.2-10.4 |
| CLI completion | musicfs-cli | `main.rs` | FR-17 |
| systemd unit | dist/ | `musicfs.service` | Production |
| Packaging | dist/ | `PKGBUILD`, `musicfs.spec` | Production |
| E2E compatibility | tests/ | `e2e_players.rs` | NFR-12.1-12.3 |
---
## Proto Definitions (`proto/musicfs.proto`)
Per architecture.md section 4.3.7, implement full gRPC API:
```protobuf
syntax = "proto3";
package musicfs.v1;
service MusicFS {
// Daemon lifecycle
rpc GetStatus(Empty) returns (StatusResponse);
rpc Shutdown(ShutdownRequest) returns (Empty);
// Cache management
rpc GetCacheStats(Empty) returns (CacheStats);
rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse);
rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress);
// Origin management
rpc ListOrigins(Empty) returns (OriginsResponse);
rpc GetOriginHealth(OriginRequest) returns (OriginHealth);
rpc RescanOrigin(OriginRequest) returns (stream SyncProgress);
// Search (already implemented in Week 8)
rpc Search(SearchRequest) returns (SearchResponse);
rpc SearchStream(SearchRequest) returns (stream SearchResult);
// Events (server-streaming)
rpc SubscribeEvents(EventFilter) returns (stream Event);
}
```
Full message definitions in architecture.md section 4.3.7.
---
## gRPC Server (`musicfs-grpc/src/server.rs`)
```rust
pub struct MusicFsService {
core: Arc<MusicFsCore>,
events: broadcast::Sender<Event>,
metrics: Arc<MetricsCollector>,
}
#[tonic::async_trait]
impl musicfs::v1::music_fs_server::MusicFs for MusicFsService {
// Daemon lifecycle
async fn get_status(&self, _: Request<Empty>) -> Result<Response<StatusResponse>, Status>;
async fn shutdown(&self, req: Request<ShutdownRequest>) -> Result<Response<Empty>, Status>;
// Cache management
async fn get_cache_stats(&self, _: Request<Empty>) -> Result<Response<CacheStats>, Status>;
async fn clear_cache(&self, req: Request<ClearCacheRequest>) -> Result<Response<ClearCacheResponse>, Status>;
type PrefetchStream = ReceiverStream<Result<PrefetchProgress, Status>>;
async fn prefetch(&self, req: Request<PrefetchRequest>) -> Result<Response<Self::PrefetchStream>, Status>;
// Origin management
async fn list_origins(&self, _: Request<Empty>) -> Result<Response<OriginsResponse>, Status>;
async fn get_origin_health(&self, req: Request<OriginRequest>) -> Result<Response<OriginHealth>, Status>;
type RescanOriginStream = ReceiverStream<Result<SyncProgress, Status>>;
async fn rescan_origin(&self, req: Request<OriginRequest>) -> Result<Response<Self::RescanOriginStream>, Status>;
// Events
type SubscribeEventsStream = ReceiverStream<Result<Event, Status>>;
async fn subscribe_events(&self, req: Request<EventFilter>) -> Result<Response<Self::SubscribeEventsStream>, Status>;
}
```
---
## Event Streaming (`musicfs-grpc/src/events.rs`)
```rust
pub struct EventStreamer {
bus: Arc<EventBus>,
}
impl EventStreamer {
/// Convert internal events to gRPC Event messages
pub fn subscribe(&self, filter: EventFilter) -> impl Stream<Item = Event>;
/// Filter events by type and origin
fn matches(event: &Event, filter: &EventFilter) -> bool;
}
```
---
## Webhook Handler (`musicfs-grpc/src/webhook.rs`)
HTTP webhook notifications for external integrations (FR-18.2):
```rust
use reqwest::Client;
use serde::Serialize;
use tokio::sync::broadcast;
#[derive(Debug, Clone, Serialize)]
pub struct WebhookPayload {
pub event_type: String,
pub timestamp: i64,
pub data: serde_json::Value,
}
pub struct WebhookConfig {
pub url: String,
pub secret: Option<String>,
pub events: Vec<String>, // Filter: ["file_accessed", "sync_completed", ...]
pub retry_count: u32,
pub timeout_ms: u64,
}
pub struct WebhookHandler {
client: Client,
configs: Vec<WebhookConfig>,
}
impl WebhookHandler {
pub fn new(configs: Vec<WebhookConfig>) -> Self;
/// Start listening to event bus and dispatch webhooks
pub async fn run(&self, mut rx: broadcast::Receiver<Event>) {
while let Ok(event) = rx.recv().await {
for config in &self.configs {
if self.matches_filter(&event, config) {
self.dispatch(config, &event).await;
}
}
}
}
/// Dispatch webhook with retry logic
async fn dispatch(&self, config: &WebhookConfig, event: &Event) {
let payload = WebhookPayload {
event_type: event.event_type(),
timestamp: event.timestamp(),
data: event.to_json(),
};
let mut attempts = 0;
loop {
let result = self.client
.post(&config.url)
.timeout(Duration::from_millis(config.timeout_ms))
.header("X-MusicFS-Signature", self.sign(&payload, config))
.json(&payload)
.send()
.await;
match result {
Ok(resp) if resp.status().is_success() => break,
_ if attempts < config.retry_count => {
attempts += 1;
tokio::time::sleep(Duration::from_millis(100 * 2u64.pow(attempts))).await;
}
_ => {
tracing::warn!("Webhook delivery failed after {} attempts", attempts);
break;
}
}
}
}
/// HMAC-SHA256 signature if secret configured
fn sign(&self, payload: &WebhookPayload, config: &WebhookConfig) -> String;
fn matches_filter(&self, event: &Event, config: &WebhookConfig) -> bool;
}
```
Configuration in `config.toml`:
```toml
[[webhooks]]
url = "https://example.com/musicfs/events"
secret = "your-webhook-secret"
events = ["file_accessed", "sync_completed", "origin_health_changed"]
retry_count = 3
timeout_ms = 5000
```
---
## E2E Compatibility Tests (`tests/e2e_players.rs`)
Verify MusicFS works with common media players (NFR-12.1-12.3):
```rust
//! E2E tests for media player compatibility
//! Requires: mpv, vlc, file manager (nautilus/dolphin) installed
use std::process::Command;
use std::time::Duration;
/// Test mpv can play files from MusicFS (NFR-12.1)
#[test]
#[ignore] // Run manually: cargo test --ignored
fn test_mpv_playback() {
let mountpoint = setup_test_mount();
// mpv should be able to:
// 1. Open file without hanging
// 2. Read metadata (duration, format)
// 3. Play first few seconds
// 4. Seek forward
// 5. Exit cleanly
let output = Command::new("mpv")
.args([
"--no-video",
"--no-audio", // Silent playback
"--length=2", // Play 2 seconds only
"--msg-level=all=debug",
&format!("{}/Artist/Album/01 - Track.flac", mountpoint),
])
.output()
.expect("mpv must be installed");
assert!(output.status.success(), "mpv playback failed: {:?}", output);
}
/// Test VLC can browse and play (NFR-12.2)
#[test]
#[ignore]
fn test_vlc_playback() {
let mountpoint = setup_test_mount();
// VLC should handle:
// 1. Directory browsing
// 2. Playlist creation from folder
// 3. Metadata display
// 4. Gapless playback (if supported)
let output = Command::new("cvlc") // Command-line VLC
.args([
"--play-and-exit",
"--run-time=2",
&format!("{}/Artist/Album/", mountpoint),
])
.output()
.expect("vlc must be installed");
assert!(output.status.success(), "VLC playback failed");
}
/// Test file manager operations (NFR-12.3)
#[test]
#[ignore]
fn test_file_manager_operations() {
let mountpoint = setup_test_mount();
// File managers should be able to:
// 1. List directories without timeout
// 2. Show file previews/thumbnails
// 3. Display file properties
// 4. Copy files to local disk
// Test basic stat operations that file managers use
let entries: Vec<_> = std::fs::read_dir(&mountpoint)
.expect("read_dir failed")
.collect();
assert!(!entries.is_empty(), "mountpoint should have entries");
// Test stat on each entry (file managers do this for icons)
for entry in entries {
let entry = entry.expect("entry should be valid");
let metadata = entry.metadata().expect("metadata should work");
assert!(metadata.is_dir() || metadata.is_file());
}
}
/// Test concurrent access from multiple players
#[test]
#[ignore]
fn test_concurrent_player_access() {
let mountpoint = setup_test_mount();
// Spawn multiple players accessing different files
let handles: Vec<_> = (0..3)
.map(|i| {
let mp = mountpoint.clone();
std::thread::spawn(move || {
Command::new("mpv")
.args([
"--no-video", "--no-audio", "--length=1",
&format!("{}/Artist/Album/0{} - Track.flac", mp, i + 1),
])
.output()
})
})
.collect();
for handle in handles {
let output = handle.join().unwrap().expect("mpv should run");
assert!(output.status.success());
}
}
fn setup_test_mount() -> String {
// Returns path to test mount with sample files
std::env::var("MUSICFS_TEST_MOUNT")
.unwrap_or_else(|_| "/tmp/musicfs-test".to_string())
}
```
---
## Metrics (`musicfs-core/src/metrics.rs`)
Per architecture.md section 5.2:
```rust
use prometheus::{IntCounterVec, HistogramVec, IntGauge, register_*};
lazy_static! {
pub static ref FUSE_OPS: IntCounterVec = register_int_counter_vec!(
"musicfs_fuse_ops_total",
"Total FUSE operations",
&["op"]
).unwrap();
pub static ref FUSE_LATENCY: HistogramVec = register_histogram_vec!(
"musicfs_fuse_latency_seconds",
"FUSE operation latency",
&["op"],
vec![0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
).unwrap();
pub static ref CACHE_HITS: IntCounter = register_int_counter!(
"musicfs_cache_hits_total",
"Cache hits"
).unwrap();
pub static ref CACHE_MISSES: IntCounter = register_int_counter!(
"musicfs_cache_misses_total",
"Cache misses"
).unwrap();
pub static ref CACHE_SIZE_BYTES: IntGauge = register_int_gauge!(
"musicfs_cache_size_bytes",
"Current cache size in bytes"
).unwrap();
pub static ref ORIGIN_HEALTH: IntGaugeVec = register_int_gauge_vec!(
"musicfs_origin_health",
"Origin health status (1=healthy, 0=unhealthy)",
&["origin"]
).unwrap();
}
/// Expose metrics on HTTP endpoint
pub async fn serve_metrics(addr: SocketAddr) -> Result<(), MetricsError>;
```
---
## CLI Commands (`musicfs-cli/src/main.rs`)
```rust
#[derive(Parser)]
enum Command {
/// Mount filesystem
Mount {
#[arg(short, long)]
config: PathBuf,
mountpoint: PathBuf,
},
/// Get daemon status
Status,
/// Cache management
Cache {
#[command(subcommand)]
command: CacheCommand,
},
/// Search library
Search {
query: String,
#[arg(short, long, default_value = "100")]
limit: u32,
},
/// Origin management
Origin {
#[command(subcommand)]
command: OriginCommand,
},
/// Subscribe to events
Events {
#[arg(short, long)]
r#type: Option<String>,
},
}
#[derive(Subcommand)]
enum CacheCommand {
Stats,
Clear { origin: Option<String> },
Prefetch { paths: Vec<String> },
}
#[derive(Subcommand)]
enum OriginCommand {
List,
Health { origin_id: String },
Rescan { origin_id: String },
}
```
---
## systemd Service (`dist/musicfs.service`)
```ini
[Unit]
Description=MusicFS - Metadata-Organized Music Filesystem
After=network.target
[Service]
Type=notify
ExecStart=/usr/bin/musicfs mount --config /etc/musicfs/config.toml /mnt/music
ExecStop=/usr/bin/musicfs shutdown
Restart=on-failure
RestartSec=5
User=musicfs
Group=musicfs
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/cache/musicfs /mnt/music
PrivateTmp=true
[Install]
WantedBy=multi-user.target
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_grpc_status` | Unit | GetStatus RPC (FR-17.1) |
| `test_grpc_cache_stats` | Unit | GetCacheStats RPC |
| `test_grpc_cache_clear` | Unit | ClearCache RPC (FR-17.3) |
| `test_grpc_origins_list` | Unit | ListOrigins RPC |
| `test_grpc_origin_rescan` | Integration | RescanOrigin streaming |
| `test_grpc_events_stream` | Integration | Event streaming (FR-18.1) |
| `test_grpc_prefetch_stream` | Integration | Prefetch progress |
| `test_webhook_dispatch` | Unit | Webhook delivery (FR-18.2) |
| `test_webhook_retry` | Unit | Webhook retry on failure |
| `test_webhook_hmac_signature` | Unit | HMAC-SHA256 signing |
| `test_metrics_prometheus` | Unit | Prometheus format (NFR-6.1) |
| `test_metrics_http_endpoint` | Integration | HTTP metrics endpoint |
| `test_cli_commands` | Integration | CLI works |
| `test_systemd_service` | E2E | Service lifecycle |
| `test_mpv_playback` | E2E | mpv compatibility (NFR-12.1) |
| `test_vlc_playback` | E2E | VLC compatibility (NFR-12.2) |
| `test_file_manager_operations` | E2E | File manager browsing (NFR-12.3) |
| `test_concurrent_player_access` | E2E | Multiple players concurrently |
---
## Exit Criteria
- [ ] gRPC API fully functional (all RPCs from architecture.md 4.3.7)
- [ ] Event streaming works with filtering
- [ ] Webhook notifications delivered with HMAC signing
- [ ] Prometheus metrics exported on HTTP endpoint
- [ ] CLI feature-complete with all commands
- [ ] systemd service works (start, stop, restart)
- [ ] mpv, VLC playback verified (E2E tests)
- [ ] File manager browsing verified
- [ ] All acceptance tests pass
---
## Architecture Alignment
Per architecture.md section 4.3.7:
- gRPC over Unix socket ✓
- Protocol Buffers for type safety ✓
- Server-streaming for events, sync progress, prefetch ✓
- CLI wraps gRPC client ✓
Per architecture.md section 5.2:
- Prometheus metrics format ✓
- Golden signals: latency, traffic, errors, saturation ✓
Per requirements.md:
- FR-17.1: Unix socket control ✓
- FR-17.2: gRPC with Protocol Buffers ✓
- FR-17.3: Cache management commands ✓
- FR-17.4: Runtime configuration ✓
- FR-17.5: Graceful shutdown ✓
- FR-18.1: File access events ✓
- FR-18.2: Webhook notifications ✓ (HTTP webhooks with HMAC)
- FR-18.3: Event streaming ✓
- FR-18.4: Access pattern logging ✓
- NFR-10.1: Configurable logging ✓
- NFR-10.2: Metrics exposure ✓
- NFR-10.3: Health check ✓
- NFR-10.4: Prometheus integration ✓
- NFR-12.1: mpv compatibility ✓ (E2E tests)
- NFR-12.2: VLC compatibility ✓ (E2E tests)
- NFR-12.3: File manager compatibility ✓ (E2E tests)
+624
View File
@@ -0,0 +1,624 @@
# Week 12: External Metadata Integration
**Phase**: 5 - P1 Feature Completion
**Goal**: Integrate external metadata sources for automatic tagging and artwork
**Requirements**: FR-21.1-21.4, FR-16.5
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| MusicBrainz client | musicfs-external | `musicbrainz.rs` | FR-21.1 |
| Discogs client | musicfs-external | `discogs.rs` | FR-21.2 |
| Last.fm client | musicfs-external | `lastfm.rs` | FR-21.3 |
| AcoustID/Chromaprint | musicfs-external | `acoustid.rs` | FR-21.4 |
| Online artwork fetch | musicfs-external | `artwork_fetch.rs` | FR-16.5 |
| Metadata enrichment | musicfs-external | `enrichment.rs` | All |
| Plugin integration | musicfs-plugins | `metadata_plugin.rs` | FR-21.5 |
---
## Task 1: Create `musicfs-external` Crate
### 1.1 `Cargo.toml`
```toml
[package]
name = "musicfs-external"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
reqwest = { version = "0.11", features = ["json"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
tokio.workspace = true
tracing.workspace = true
thiserror.workspace = true
chromaprint = "0.6" # Audio fingerprinting
base64 = "0.21"
[dev-dependencies]
wiremock = "0.5" # Mock HTTP responses
tokio-test = "0.4"
```
### 1.2 `src/lib.rs`
```rust
pub mod musicbrainz;
pub mod discogs;
pub mod lastfm;
pub mod acoustid;
pub mod artwork_fetch;
pub mod enrichment;
pub use enrichment::MetadataEnricher;
```
---
## Task 2: MusicBrainz Client (`musicfs-external/src/musicbrainz.rs`)
```rust
use serde::Deserialize;
const MB_API: &str = "https://musicbrainz.org/ws/2";
const USER_AGENT: &str = "MusicFS/0.1.0 (https://github.com/user/musicfs)";
#[derive(Debug, Deserialize)]
pub struct MbRecording {
pub id: String,
pub title: String,
pub length: Option<u64>,
#[serde(rename = "artist-credit")]
pub artist_credit: Vec<ArtistCredit>,
pub releases: Option<Vec<MbRelease>>,
}
#[derive(Debug, Deserialize)]
pub struct MbRelease {
pub id: String,
pub title: String,
pub date: Option<String>,
#[serde(rename = "release-group")]
pub release_group: Option<MbReleaseGroup>,
}
#[derive(Debug, Deserialize)]
pub struct MbReleaseGroup {
pub id: String,
#[serde(rename = "primary-type")]
pub primary_type: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct ArtistCredit {
pub artist: MbArtist,
}
#[derive(Debug, Deserialize)]
pub struct MbArtist {
pub id: String,
pub name: String,
#[serde(rename = "sort-name")]
pub sort_name: String,
}
pub struct MusicBrainzClient {
client: reqwest::Client,
rate_limiter: RateLimiter, // 1 req/sec per MB guidelines
}
impl MusicBrainzClient {
pub fn new() -> Self {
let client = reqwest::Client::builder()
.user_agent(USER_AGENT)
.build()
.expect("client build");
Self {
client,
rate_limiter: RateLimiter::new(Duration::from_secs(1)),
}
}
/// Search by recording title + artist (FR-21.1)
pub async fn search_recording(
&self,
title: &str,
artist: Option<&str>,
) -> Result<Vec<MbRecording>, ExternalError> {
self.rate_limiter.wait().await;
let mut query = format!("recording:{}", title);
if let Some(artist) = artist {
query.push_str(&format!(" AND artist:{}", artist));
}
let resp = self.client
.get(format!("{}/recording", MB_API))
.query(&[
("query", query.as_str()),
("fmt", "json"),
("limit", "5"),
])
.send()
.await?;
let body: SearchResponse<MbRecording> = resp.json().await?;
Ok(body.recordings)
}
/// Get release artwork from Cover Art Archive
pub async fn get_cover_art(&self, release_id: &str) -> Result<Option<Vec<u8>>, ExternalError> {
let url = format!("https://coverartarchive.org/release/{}/front-500", release_id);
let resp = self.client.get(&url).send().await?;
if resp.status() == 404 {
return Ok(None);
}
let bytes = resp.bytes().await?;
Ok(Some(bytes.to_vec()))
}
/// Lookup recording by MusicBrainz ID
pub async fn get_recording(&self, mbid: &str) -> Result<MbRecording, ExternalError> {
self.rate_limiter.wait().await;
let resp = self.client
.get(format!("{}/recording/{}", MB_API, mbid))
.query(&[
("inc", "artist-credits+releases+release-groups"),
("fmt", "json"),
])
.send()
.await?;
Ok(resp.json().await?)
}
}
struct RateLimiter {
interval: Duration,
last_request: Mutex<Instant>,
}
impl RateLimiter {
fn new(interval: Duration) -> Self {
Self {
interval,
last_request: Mutex::new(Instant::now() - interval),
}
}
async fn wait(&self) {
let mut last = self.last_request.lock().await;
let elapsed = last.elapsed();
if elapsed < self.interval {
tokio::time::sleep(self.interval - elapsed).await;
}
*last = Instant::now();
}
}
```
---
## Task 3: Discogs Client (`musicfs-external/src/discogs.rs`)
```rust
const DISCOGS_API: &str = "https://api.discogs.com";
pub struct DiscogsClient {
client: reqwest::Client,
token: Option<String>,
rate_limiter: RateLimiter, // 60 req/min authenticated
}
impl DiscogsClient {
pub fn new(token: Option<String>) -> Self;
/// Search releases (FR-21.2)
pub async fn search(
&self,
query: &str,
artist: Option<&str>,
) -> Result<Vec<DiscogsRelease>, ExternalError>;
/// Get master release details
pub async fn get_master(&self, id: u64) -> Result<DiscogsMaster, ExternalError>;
/// Get release images
pub async fn get_images(&self, release_id: u64) -> Result<Vec<DiscogsImage>, ExternalError>;
}
#[derive(Debug, Deserialize)]
pub struct DiscogsRelease {
pub id: u64,
pub title: String,
pub year: Option<u16>,
pub thumb: Option<String>,
pub master_id: Option<u64>,
}
#[derive(Debug, Deserialize)]
pub struct DiscogsImage {
pub uri: String,
pub width: u32,
pub height: u32,
#[serde(rename = "type")]
pub image_type: String, // "primary" or "secondary"
}
```
---
## Task 4: Last.fm Client (`musicfs-external/src/lastfm.rs`)
```rust
const LASTFM_API: &str = "https://ws.audioscrobbler.com/2.0";
pub struct LastFmClient {
client: reqwest::Client,
api_key: String,
}
impl LastFmClient {
pub fn new(api_key: String) -> Self;
/// Get track info with play counts, tags (FR-21.3)
pub async fn get_track_info(
&self,
track: &str,
artist: &str,
) -> Result<LastFmTrack, ExternalError>;
/// Get album info with artwork
pub async fn get_album_info(
&self,
album: &str,
artist: &str,
) -> Result<LastFmAlbum, ExternalError>;
/// Get artist info
pub async fn get_artist_info(&self, artist: &str) -> Result<LastFmArtist, ExternalError>;
}
#[derive(Debug, Deserialize)]
pub struct LastFmTrack {
pub name: String,
pub playcount: Option<u64>,
pub listeners: Option<u64>,
pub duration: Option<u64>,
pub toptags: Option<Tags>,
pub album: Option<LastFmAlbumRef>,
}
#[derive(Debug, Deserialize)]
pub struct LastFmAlbum {
pub name: String,
pub artist: String,
pub image: Vec<LastFmImage>,
pub tracks: Option<Tracks>,
}
#[derive(Debug, Deserialize)]
pub struct LastFmImage {
#[serde(rename = "#text")]
pub url: String,
pub size: String, // "small", "medium", "large", "extralarge", "mega"
}
```
---
## Task 5: AcoustID/Chromaprint (`musicfs-external/src/acoustid.rs`)
```rust
use chromaprint::{Fingerprinter, Configuration};
const ACOUSTID_API: &str = "https://api.acoustid.org/v2/lookup";
pub struct AcoustIdClient {
client: reqwest::Client,
api_key: String,
}
impl AcoustIdClient {
pub fn new(api_key: String) -> Self;
/// Generate fingerprint from audio data (FR-21.4)
pub fn fingerprint(&self, samples: &[i16], sample_rate: u32) -> Result<String, ExternalError> {
let config = Configuration::preset_test1();
let mut fp = Fingerprinter::new(&config);
fp.start(sample_rate, 1)?; // mono
fp.feed(samples)?;
fp.finish()?;
Ok(fp.fingerprint().to_string())
}
/// Lookup fingerprint on AcoustID database
pub async fn lookup(
&self,
fingerprint: &str,
duration: u32,
) -> Result<Vec<AcoustIdResult>, ExternalError> {
let resp = self.client
.get(ACOUSTID_API)
.query(&[
("client", self.api_key.as_str()),
("fingerprint", fingerprint),
("duration", &duration.to_string()),
("meta", "recordings+releasegroups"),
])
.send()
.await?;
let body: AcoustIdResponse = resp.json().await?;
Ok(body.results)
}
}
#[derive(Debug, Deserialize)]
pub struct AcoustIdResult {
pub id: String,
pub score: f32,
pub recordings: Option<Vec<AcoustIdRecording>>,
}
#[derive(Debug, Deserialize)]
pub struct AcoustIdRecording {
pub id: String, // MusicBrainz recording ID
pub title: Option<String>,
pub artists: Option<Vec<AcoustIdArtist>>,
}
```
---
## Task 6: Online Artwork Fetch (`musicfs-external/src/artwork_fetch.rs`)
```rust
pub struct ArtworkFetcher {
musicbrainz: MusicBrainzClient,
discogs: Option<DiscogsClient>,
lastfm: Option<LastFmClient>,
}
impl ArtworkFetcher {
/// Fetch missing artwork from online sources (FR-16.5)
/// Tries sources in order: MusicBrainz Cover Art Archive → Discogs → Last.fm
pub async fn fetch_artwork(
&self,
artist: &str,
album: &str,
size: ArtworkSize,
) -> Result<Option<ArtworkData>, ExternalError> {
// 1. Try MusicBrainz release search → Cover Art Archive
if let Some(art) = self.try_musicbrainz(artist, album, size).await? {
return Ok(Some(art));
}
// 2. Try Discogs
if let Some(discogs) = &self.discogs {
if let Some(art) = self.try_discogs(discogs, artist, album, size).await? {
return Ok(Some(art));
}
}
// 3. Try Last.fm
if let Some(lastfm) = &self.lastfm {
if let Some(art) = self.try_lastfm(lastfm, artist, album, size).await? {
return Ok(Some(art));
}
}
Ok(None)
}
async fn try_musicbrainz(
&self,
artist: &str,
album: &str,
size: ArtworkSize,
) -> Result<Option<ArtworkData>, ExternalError> {
// Search for release, get cover art from Cover Art Archive
let releases = self.musicbrainz.search_release(album, Some(artist)).await?;
for release in releases.iter().take(3) {
if let Some(art) = self.musicbrainz.get_cover_art(&release.id).await? {
return Ok(Some(ArtworkData {
data: art,
source: ArtworkSource::MusicBrainz,
mime_type: "image/jpeg".to_string(),
}));
}
}
Ok(None)
}
}
#[derive(Debug)]
pub struct ArtworkData {
pub data: Vec<u8>,
pub source: ArtworkSource,
pub mime_type: String,
}
#[derive(Debug)]
pub enum ArtworkSource {
MusicBrainz,
Discogs,
LastFm,
Embedded,
}
pub enum ArtworkSize {
Small, // 150px
Medium, // 300px
Large, // 500px
Original,
}
```
---
## Task 7: Metadata Enrichment (`musicfs-external/src/enrichment.rs`)
```rust
pub struct MetadataEnricher {
musicbrainz: MusicBrainzClient,
acoustid: Option<AcoustIdClient>,
artwork_fetcher: ArtworkFetcher,
}
impl MetadataEnricher {
/// Enrich metadata from external sources
pub async fn enrich(&self, meta: &AudioMeta) -> Result<EnrichedMetadata, ExternalError> {
let mut enriched = EnrichedMetadata::from(meta);
// If we have title + artist, search MusicBrainz
if let (Some(title), Some(artist)) = (&meta.title, &meta.artist) {
let recordings = self.musicbrainz.search_recording(title, Some(artist)).await?;
if let Some(best) = recordings.first() {
enriched.musicbrainz_recording_id = Some(best.id.clone());
// Enrich with release info
if let Some(releases) = &best.releases {
if let Some(release) = releases.first() {
enriched.musicbrainz_release_id = Some(release.id.clone());
}
}
}
}
Ok(enriched)
}
/// Identify unknown track by audio fingerprint
pub async fn identify_by_fingerprint(
&self,
samples: &[i16],
sample_rate: u32,
duration: u32,
) -> Result<Option<IdentifiedTrack>, ExternalError> {
let acoustid = self.acoustid.as_ref()
.ok_or(ExternalError::ServiceNotConfigured("AcoustID"))?;
let fingerprint = acoustid.fingerprint(samples, sample_rate)?;
let results = acoustid.lookup(&fingerprint, duration).await?;
// Return best match above threshold
results.into_iter()
.filter(|r| r.score > 0.8)
.flat_map(|r| r.recordings)
.flatten()
.next()
.map(|rec| IdentifiedTrack {
title: rec.title,
musicbrainz_id: Some(rec.id),
artists: rec.artists.map(|a| a.into_iter().map(|x| x.name).collect()),
})
.pipe(Ok)
}
}
#[derive(Debug)]
pub struct EnrichedMetadata {
pub original: AudioMeta,
pub musicbrainz_recording_id: Option<String>,
pub musicbrainz_release_id: Option<String>,
pub musicbrainz_artist_id: Option<String>,
pub genres: Vec<String>,
pub play_count: Option<u64>,
}
#[derive(Debug)]
pub struct IdentifiedTrack {
pub title: Option<String>,
pub musicbrainz_id: Option<String>,
pub artists: Option<Vec<String>>,
}
```
---
## Configuration
```toml
[external]
# MusicBrainz (no auth required, rate limited to 1 req/sec)
musicbrainz.enabled = true
# Discogs (optional, requires token for higher rate limits)
discogs.enabled = true
discogs.token = "your_discogs_token"
# Last.fm (requires API key)
lastfm.enabled = true
lastfm.api_key = "your_lastfm_api_key"
# AcoustID (requires API key)
acoustid.enabled = true
acoustid.api_key = "your_acoustid_api_key"
# Artwork fetching behavior
artwork.fetch_missing = true
artwork.cache_fetched = true
artwork.preferred_size = "large" # small, medium, large, original
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_musicbrainz_search` | Integration | Recording search (FR-21.1) |
| `test_musicbrainz_cover_art` | Integration | Cover Art Archive |
| `test_discogs_search` | Integration | Release search (FR-21.2) |
| `test_lastfm_track_info` | Integration | Track metadata (FR-21.3) |
| `test_acoustid_fingerprint` | Unit | Chromaprint generation |
| `test_acoustid_lookup` | Integration | Fingerprint lookup (FR-21.4) |
| `test_artwork_fetch_cascade` | Integration | Multi-source artwork (FR-16.5) |
| `test_metadata_enrichment` | Integration | Full enrichment flow |
| `test_rate_limiting` | Unit | Rate limiter works |
| `test_mock_responses` | Unit | Offline testing with mocks |
---
## Exit Criteria
- [ ] MusicBrainz search returns relevant recordings
- [ ] Cover Art Archive artwork downloads work
- [ ] Discogs integration retrieves release info
- [ ] Last.fm integration retrieves track/artist info
- [ ] AcoustID fingerprinting identifies tracks
- [ ] Artwork fetcher tries all sources in cascade
- [ ] Metadata enricher adds external IDs
- [ ] Rate limiting prevents API abuse
- [ ] All tests pass with mock HTTP responses
---
## Architecture Alignment
Per requirements.md:
- FR-21.1: MusicBrainz for canonical metadata ✓
- FR-21.2: Discogs for release info, artwork ✓
- FR-21.3: Last.fm for play counts, tags ✓
- FR-21.4: AcoustID for audio fingerprinting ✓
- FR-16.5: Fetch missing artwork from online ✓
Per architecture.md section 4.3.4:
- External metadata via `MetadataPlugin` trait ✓
- Plugin architecture allows adding more sources ✓
+699
View File
@@ -0,0 +1,699 @@
# Week 13: Import & Export
**Phase**: 5 - P1 Feature Completion
**Goal**: Import metadata from existing library managers, export library data
**Requirements**: FR-22.1-22.3
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Beets database import | musicfs-import | `beets.rs` | FR-22.1 |
| iTunes/Apple Music import | musicfs-import | `itunes.rs` | FR-22.2 |
| Library export | musicfs-import | `export.rs` | FR-22.3 |
| Import CLI | musicfs-cli | `import.rs` | All |
---
## Task 1: Create `musicfs-import` Crate
### 1.1 `Cargo.toml`
```toml
[package]
name = "musicfs-import"
version.workspace = true
edition.workspace = true
[dependencies]
musicfs-core = { path = "../musicfs-core" }
musicfs-cache = { path = "../musicfs-cache" }
rusqlite = { workspace = true, features = ["bundled"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
plist = "1.5" # For iTunes XML parsing
tokio.workspace = true
tracing.workspace = true
thiserror.workspace = true
csv = "1.3"
url = "2.4"
percent-encoding = "2.3"
chrono = { version = "0.4", features = ["serde"] }
[dev-dependencies]
tempfile.workspace = true
```
### 1.2 `src/lib.rs`
```rust
pub mod beets;
pub mod itunes;
pub mod export;
use musicfs_core::Result;
/// Common import result
#[derive(Debug, Default)]
pub struct ImportResult {
pub imported: usize,
pub skipped: usize,
pub errors: Vec<ImportError>,
}
#[derive(Debug)]
pub struct ImportError {
pub path: String,
pub reason: String,
}
/// Import progress callback
pub type ProgressCallback = Box<dyn Fn(ImportProgress) + Send>;
#[derive(Debug, Clone)]
pub struct ImportProgress {
pub current: usize,
pub total: usize,
pub current_file: String,
}
```
---
## Task 2: Beets Database Import (`musicfs-import/src/beets.rs`)
```rust
use rusqlite::{Connection, params};
use std::path::Path;
/// Beets database schema (simplified)
/// Full schema: https://beets.readthedocs.io/en/stable/dev/db.html
#[derive(Debug)]
pub struct BeetsItem {
pub id: i64,
pub path: String,
pub title: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>,
pub year: Option<i32>,
pub track: Option<i32>,
pub disc: Option<i32>,
pub length: Option<f64>,
pub bitrate: Option<i32>,
pub sample_rate: Option<i32>,
pub format: Option<String>,
pub mb_trackid: Option<String>,
pub mb_albumid: Option<String>,
pub mb_artistid: Option<String>,
pub mtime: f64,
}
pub struct BeetsImporter {
beets_db: Connection,
target_db: Arc<Database>,
}
impl BeetsImporter {
/// Open beets database for import (FR-22.1)
pub fn new(beets_db_path: &Path, target_db: Arc<Database>) -> Result<Self, ImportError> {
let conn = Connection::open_with_flags(
beets_db_path,
rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY,
)?;
// Verify this is a beets database
let tables: Vec<String> = conn
.prepare("SELECT name FROM sqlite_master WHERE type='table'")?
.query_map([], |row| row.get(0))?
.filter_map(|r| r.ok())
.collect();
if !tables.contains(&"items".to_string()) {
return Err(ImportError::InvalidDatabase("Not a beets database"));
}
Ok(Self {
beets_db: conn,
target_db,
})
}
/// Count items to import
pub fn count_items(&self) -> Result<usize, ImportError> {
self.beets_db
.query_row("SELECT COUNT(*) FROM items", [], |row| row.get(0))
.map_err(Into::into)
}
/// Import all items with progress callback
pub fn import_all(&self, progress: Option<ProgressCallback>) -> Result<ImportResult, ImportError> {
let total = self.count_items()?;
let mut result = ImportResult::default();
let mut stmt = self.beets_db.prepare(r#"
SELECT id, path, title, artist, album, albumartist, genre,
year, track, disc, length, bitrate, samplerate, format,
mb_trackid, mb_albumid, mb_artistid, mtime
FROM items
"#)?;
let items = stmt.query_map([], |row| {
Ok(BeetsItem {
id: row.get(0)?,
path: row.get(1)?,
title: row.get(2)?,
artist: row.get(3)?,
album: row.get(4)?,
album_artist: row.get(5)?,
genre: row.get(6)?,
year: row.get(7)?,
track: row.get(8)?,
disc: row.get(9)?,
length: row.get(10)?,
bitrate: row.get(11)?,
sample_rate: row.get(12)?,
format: row.get(13)?,
mb_trackid: row.get(14)?,
mb_albumid: row.get(15)?,
mb_artistid: row.get(16)?,
mtime: row.get(17)?,
})
})?;
for (idx, item) in items.enumerate() {
match item {
Ok(item) => {
if let Some(ref cb) = progress {
cb(ImportProgress {
current: idx + 1,
total,
current_file: item.path.clone(),
});
}
match self.import_item(&item) {
Ok(_) => result.imported += 1,
Err(e) => {
result.errors.push(ImportError {
path: item.path,
reason: e.to_string(),
});
}
}
}
Err(e) => {
result.skipped += 1;
result.errors.push(ImportError {
path: format!("item_{}", idx),
reason: e.to_string(),
});
}
}
}
Ok(result)
}
fn import_item(&self, item: &BeetsItem) -> Result<(), ImportError> {
let path = Path::new(&item.path);
// Convert to our AudioMeta
let audio_meta = AudioMeta {
title: item.title.clone(),
artist: item.artist.clone(),
album: item.album.clone(),
album_artist: item.album_artist.clone(),
genre: item.genre.clone(),
year: item.year.map(|y| y as u32),
track: item.track.map(|t| t as u32),
disc: item.disc.map(|d| d as u32),
duration_ms: item.length.map(|l| (l * 1000.0) as u64),
bitrate: item.bitrate.map(|b| b as u32),
sample_rate: item.sample_rate.map(|s| s as u32),
format: AudioFormat::from_extension(
path.extension().and_then(|e| e.to_str()).unwrap_or("")
),
..Default::default()
};
// Generate virtual path using our resolver
let virtual_path = VirtualPath::from_metadata(&audio_meta, path);
// Import to our database
self.target_db.upsert_file(
&OriginId::from("beets-import"),
path,
&virtual_path,
&audio_meta,
std::time::UNIX_EPOCH + std::time::Duration::from_secs_f64(item.mtime),
std::fs::metadata(path).map(|m| m.len()).unwrap_or(0),
)?;
Ok(())
}
}
```
---
## Task 3: iTunes/Apple Music Import (`musicfs-import/src/itunes.rs`)
```rust
use plist::Value;
use std::collections::HashMap;
use url::Url;
/// iTunes Library XML format
#[derive(Debug)]
pub struct ItunesTrack {
pub track_id: u64,
pub name: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>,
pub year: Option<u32>,
pub track_number: Option<u32>,
pub disc_number: Option<u32>,
pub total_time: Option<u64>, // milliseconds
pub bit_rate: Option<u32>,
pub sample_rate: Option<u32>,
pub location: Option<String>, // file:// URL
pub date_added: Option<String>,
}
pub struct ItunesImporter {
tracks: Vec<ItunesTrack>,
target_db: Arc<Database>,
}
impl ItunesImporter {
/// Parse iTunes Library.xml (FR-22.2)
pub fn from_xml(xml_path: &Path, target_db: Arc<Database>) -> Result<Self, ImportError> {
let file = std::fs::File::open(xml_path)?;
let plist: Value = plist::from_reader(file)?;
let dict = plist.as_dictionary()
.ok_or(ImportError::InvalidFormat("Expected dictionary at root"))?;
let tracks_dict = dict.get("Tracks")
.and_then(|v| v.as_dictionary())
.ok_or(ImportError::InvalidFormat("Missing Tracks dictionary"))?;
let mut tracks = Vec::new();
for (_, track_value) in tracks_dict {
if let Some(track_dict) = track_value.as_dictionary() {
tracks.push(Self::parse_track(track_dict)?);
}
}
Ok(Self { tracks, target_db })
}
fn parse_track(dict: &plist::Dictionary) -> Result<ItunesTrack, ImportError> {
Ok(ItunesTrack {
track_id: dict.get("Track ID")
.and_then(|v| v.as_unsigned_integer())
.unwrap_or(0),
name: dict.get("Name").and_then(|v| v.as_string()).map(String::from),
artist: dict.get("Artist").and_then(|v| v.as_string()).map(String::from),
album: dict.get("Album").and_then(|v| v.as_string()).map(String::from),
album_artist: dict.get("Album Artist").and_then(|v| v.as_string()).map(String::from),
genre: dict.get("Genre").and_then(|v| v.as_string()).map(String::from),
year: dict.get("Year").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
track_number: dict.get("Track Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
disc_number: dict.get("Disc Number").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
total_time: dict.get("Total Time").and_then(|v| v.as_unsigned_integer()),
bit_rate: dict.get("Bit Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
sample_rate: dict.get("Sample Rate").and_then(|v| v.as_unsigned_integer()).map(|v| v as u32),
location: dict.get("Location").and_then(|v| v.as_string()).map(String::from),
date_added: dict.get("Date Added").and_then(|v| v.as_string()).map(String::from),
})
}
/// Convert file:// URL to path
fn url_to_path(url_str: &str) -> Option<PathBuf> {
Url::parse(url_str).ok()
.filter(|u| u.scheme() == "file")
.and_then(|u| u.to_file_path().ok())
}
pub fn count_tracks(&self) -> usize {
self.tracks.len()
}
/// Import all tracks
pub fn import_all(&self, progress: Option<ProgressCallback>) -> Result<ImportResult, ImportError> {
let total = self.tracks.len();
let mut result = ImportResult::default();
for (idx, track) in self.tracks.iter().enumerate() {
if let Some(ref cb) = progress {
cb(ImportProgress {
current: idx + 1,
total,
current_file: track.name.clone().unwrap_or_default(),
});
}
// Skip tracks without location
let Some(ref location) = track.location else {
result.skipped += 1;
continue;
};
let Some(path) = Self::url_to_path(location) else {
result.skipped += 1;
result.errors.push(ImportError {
path: location.clone(),
reason: "Invalid file URL".to_string(),
});
continue;
};
match self.import_track(track, &path) {
Ok(_) => result.imported += 1,
Err(e) => {
result.errors.push(ImportError {
path: path.display().to_string(),
reason: e.to_string(),
});
}
}
}
Ok(result)
}
fn import_track(&self, track: &ItunesTrack, path: &Path) -> Result<(), ImportError> {
let audio_meta = AudioMeta {
title: track.name.clone(),
artist: track.artist.clone(),
album: track.album.clone(),
album_artist: track.album_artist.clone(),
genre: track.genre.clone(),
year: track.year,
track: track.track_number,
disc: track.disc_number,
duration_ms: track.total_time,
bitrate: track.bit_rate,
sample_rate: track.sample_rate,
format: AudioFormat::from_extension(
path.extension().and_then(|e| e.to_str()).unwrap_or("")
),
..Default::default()
};
let virtual_path = VirtualPath::from_metadata(&audio_meta, path);
let mtime = std::fs::metadata(path)
.map(|m| m.modified().unwrap_or(std::time::UNIX_EPOCH))
.unwrap_or(std::time::UNIX_EPOCH);
let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
self.target_db.upsert_file(
&OriginId::from("itunes-import"),
path,
&virtual_path,
&audio_meta,
mtime,
size,
)?;
Ok(())
}
}
```
---
## Task 4: Library Export (`musicfs-import/src/export.rs`)
```rust
use csv::Writer;
use serde::Serialize;
#[derive(Debug, Serialize)]
pub struct ExportedTrack {
pub virtual_path: String,
pub real_path: String,
pub title: String,
pub artist: String,
pub album: String,
pub album_artist: String,
pub genre: String,
pub year: Option<u32>,
pub track: Option<u32>,
pub disc: Option<u32>,
pub duration_ms: Option<u64>,
pub format: String,
pub musicbrainz_id: Option<String>,
}
pub struct LibraryExporter {
db: Arc<Database>,
}
impl LibraryExporter {
pub fn new(db: Arc<Database>) -> Self {
Self { db }
}
/// Export library to CSV (FR-22.3)
pub fn export_csv(&self, output: &Path) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let mut writer = Writer::from_path(output)?;
let mut count = 0;
for file in files {
let audio = file.audio.as_ref();
writer.serialize(ExportedTrack {
virtual_path: file.virtual_path.as_str().to_string(),
real_path: file.real_path.path.display().to_string(),
title: audio.and_then(|a| a.title.clone()).unwrap_or_default(),
artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(),
album: audio.and_then(|a| a.album.clone()).unwrap_or_default(),
album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(),
genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(),
year: audio.and_then(|a| a.year),
track: audio.and_then(|a| a.track),
disc: audio.and_then(|a| a.disc),
duration_ms: audio.and_then(|a| a.duration_ms),
format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(),
musicbrainz_id: None, // TODO: Include if enriched
})?;
count += 1;
}
writer.flush()?;
Ok(count)
}
/// Export library to JSON
pub fn export_json(&self, output: &Path) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let tracks: Vec<ExportedTrack> = files.iter()
.map(|file| {
let audio = file.audio.as_ref();
ExportedTrack {
virtual_path: file.virtual_path.as_str().to_string(),
real_path: file.real_path.path.display().to_string(),
title: audio.and_then(|a| a.title.clone()).unwrap_or_default(),
artist: audio.and_then(|a| a.artist.clone()).unwrap_or_default(),
album: audio.and_then(|a| a.album.clone()).unwrap_or_default(),
album_artist: audio.and_then(|a| a.album_artist.clone()).unwrap_or_default(),
genre: audio.and_then(|a| a.genre.clone()).unwrap_or_default(),
year: audio.and_then(|a| a.year),
track: audio.and_then(|a| a.track),
disc: audio.and_then(|a| a.disc),
duration_ms: audio.and_then(|a| a.duration_ms),
format: audio.map(|a| format!("{:?}", a.format)).unwrap_or_default(),
musicbrainz_id: None,
}
})
.collect();
let json = serde_json::to_string_pretty(&tracks)?;
std::fs::write(output, json)?;
Ok(tracks.len())
}
/// Export to M3U playlist format
pub fn export_m3u(&self, output: &Path, base_path: Option<&Path>) -> Result<usize, ExportError> {
let files = self.db.list_all_files()?;
let mut content = String::from("#EXTM3U\n");
for file in &files {
let duration = file.audio.as_ref()
.and_then(|a| a.duration_ms)
.map(|d| d / 1000)
.unwrap_or(0);
let title = file.audio.as_ref()
.and_then(|a| a.title.clone())
.unwrap_or_else(|| file.virtual_path.as_str().to_string());
let artist = file.audio.as_ref()
.and_then(|a| a.artist.clone())
.unwrap_or_default();
content.push_str(&format!(
"#EXTINF:{},{} - {}\n",
duration, artist, title
));
// Use virtual path relative to base, or absolute real path
let path = if let Some(base) = base_path {
base.join(file.virtual_path.as_str().trim_start_matches('/'))
.display().to_string()
} else {
file.real_path.path.display().to_string()
};
content.push_str(&path);
content.push('\n');
}
std::fs::write(output, content)?;
Ok(files.len())
}
}
```
---
## Task 5: Import CLI Commands (`musicfs-cli/src/import.rs`)
```rust
#[derive(Subcommand)]
pub enum ImportCommand {
/// Import from beets database
Beets {
/// Path to beets library.db
#[arg(short, long)]
db: PathBuf,
},
/// Import from iTunes Library.xml
Itunes {
/// Path to iTunes Library.xml
#[arg(short, long)]
xml: PathBuf,
},
/// Export library
Export {
/// Output file path
#[arg(short, long)]
output: PathBuf,
/// Format: csv, json, m3u
#[arg(short, long, default_value = "csv")]
format: String,
},
}
pub async fn handle_import(cmd: ImportCommand, db: Arc<Database>) -> Result<()> {
match cmd {
ImportCommand::Beets { db: beets_path } => {
println!("Importing from beets database: {:?}", beets_path);
let importer = BeetsImporter::new(&beets_path, db)?;
let total = importer.count_items()?;
println!("Found {} items to import", total);
let pb = ProgressBar::new(total as u64);
let result = importer.import_all(Some(Box::new(move |p| {
pb.set_position(p.current as u64);
})))?;
println!("\nImport complete:");
println!(" Imported: {}", result.imported);
println!(" Skipped: {}", result.skipped);
println!(" Errors: {}", result.errors.len());
}
ImportCommand::Itunes { xml } => {
println!("Importing from iTunes Library: {:?}", xml);
let importer = ItunesImporter::from_xml(&xml, db)?;
let total = importer.count_tracks();
println!("Found {} tracks to import", total);
let pb = ProgressBar::new(total as u64);
let result = importer.import_all(Some(Box::new(move |p| {
pb.set_position(p.current as u64);
})))?;
println!("\nImport complete:");
println!(" Imported: {}", result.imported);
println!(" Skipped: {}", result.skipped);
println!(" Errors: {}", result.errors.len());
}
ImportCommand::Export { output, format } => {
let exporter = LibraryExporter::new(db);
let count = match format.as_str() {
"csv" => exporter.export_csv(&output)?,
"json" => exporter.export_json(&output)?,
"m3u" => exporter.export_m3u(&output, None)?,
_ => return Err(anyhow::anyhow!("Unknown format: {}", format)),
};
println!("Exported {} tracks to {:?}", count, output);
}
}
Ok(())
}
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_beets_import_valid` | Integration | Beets database parsing (FR-22.1) |
| `test_beets_import_missing_fields` | Unit | Handle incomplete metadata |
| `test_itunes_xml_parsing` | Unit | iTunes XML parsing (FR-22.2) |
| `test_itunes_url_to_path` | Unit | file:// URL conversion |
| `test_itunes_import_tracks` | Integration | Full iTunes import |
| `test_export_csv` | Unit | CSV export (FR-22.3) |
| `test_export_json` | Unit | JSON export |
| `test_export_m3u` | Unit | M3U playlist export |
| `test_import_preserves_musicbrainz_ids` | Integration | External IDs preserved |
| `test_import_deduplication` | Integration | No duplicates on re-import |
---
## Exit Criteria
- [ ] Beets database import works with real beets.db
- [ ] iTunes Library.xml import parses all tracks
- [ ] CSV/JSON/M3U export generates valid files
- [ ] Progress reporting works during import
- [ ] Errors are reported without crashing
- [ ] Import is idempotent (re-import updates, doesn't duplicate)
- [ ] MusicBrainz IDs from beets are preserved
---
## Architecture Alignment
Per requirements.md:
- FR-22.1: Import from beets database ✓
- FR-22.2: Import from iTunes/Apple Music ✓
- FR-22.3: Export library metadata ✓
+633
View File
@@ -0,0 +1,633 @@
# Week 14: Extended Formats & Audio Fingerprinting
**Phase**: 5 - P1 Feature Completion
**Goal**: Audio fingerprint search and audiobook format support
**Requirements**: FR-14.4, FR-24.2
---
## Deliverables
| Task | Crate | Files | Requirements |
|------|-------|-------|--------------|
| Fingerprint indexing | musicfs-search | `fingerprint.rs` | FR-14.4 |
| Fingerprint search | musicfs-search | `fingerprint_search.rs` | FR-14.4 |
| M4B audiobook support | musicfs-metadata | `formats/m4b.rs` | FR-24.2 |
| Chapter extraction | musicfs-metadata | `chapters.rs` | FR-24.2 |
| Virtual chapter files | musicfs-fuse | `ops/chapters.rs` | FR-24.2 |
---
## Task 1: Audio Fingerprint Generation
### 1.1 Add Dependencies
```toml
# In musicfs-search/Cargo.toml
[dependencies]
chromaprint = "0.6"
symphonia = { version = "0.5", features = ["all"] }
```
### 1.2 Fingerprint Generation (`musicfs-search/src/fingerprint.rs`)
```rust
use chromaprint::{Configuration, Fingerprinter};
use symphonia::core::audio::SampleBuffer;
use symphonia::core::codecs::DecoderOptions;
use std::path::Path;
/// Audio fingerprint using Chromaprint algorithm
#[derive(Debug, Clone)]
pub struct AudioFingerprint {
pub raw: Vec<u32>,
pub duration_secs: u32,
}
impl AudioFingerprint {
/// Generate fingerprint from audio file (FR-14.4)
pub fn from_file(path: &Path) -> Result<Self, FingerprintError> {
let file = std::fs::File::open(path)?;
let mss = MediaSourceStream::new(Box::new(file), Default::default());
let probed = symphonia::default::get_probe()
.format(&Hint::new(), mss, &FormatOptions::default(), &MetadataOptions::default())?;
let mut format = probed.format;
let track = format.tracks()
.iter()
.find(|t| t.codec_params.codec != CODEC_TYPE_NULL)
.ok_or(FingerprintError::NoAudioTrack)?;
let sample_rate = track.codec_params.sample_rate
.ok_or(FingerprintError::NoSampleRate)?;
let mut decoder = symphonia::default::get_codecs()
.make(&track.codec_params, &DecoderOptions::default())?;
// Chromaprint configuration
let config = Configuration::preset_test1();
let mut fingerprinter = Fingerprinter::new(&config);
fingerprinter.start(sample_rate, 1)?; // Mono
let mut samples: Vec<i16> = Vec::new();
let mut duration_samples = 0u64;
// Decode and collect samples (first 120 seconds max)
let max_samples = sample_rate as u64 * 120;
loop {
match format.next_packet() {
Ok(packet) => {
let decoded = decoder.decode(&packet)?;
let mut sample_buf = SampleBuffer::<i16>::new(
decoded.capacity() as u64,
*decoded.spec(),
);
sample_buf.copy_interleaved_ref(decoded);
// Convert to mono if stereo
let mono: Vec<i16> = if decoded.spec().channels.count() > 1 {
sample_buf.samples()
.chunks(decoded.spec().channels.count())
.map(|chunk| (chunk.iter().map(|&s| s as i32).sum::<i32>() / chunk.len() as i32) as i16)
.collect()
} else {
sample_buf.samples().to_vec()
};
samples.extend(&mono);
duration_samples += mono.len() as u64;
if duration_samples >= max_samples {
break;
}
}
Err(symphonia::core::errors::Error::IoError(e))
if e.kind() == std::io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e.into()),
}
}
// Feed samples to fingerprinter
fingerprinter.feed(&samples)?;
fingerprinter.finish()?;
let raw = fingerprinter.fingerprint().to_vec();
let duration_secs = (duration_samples / sample_rate as u64) as u32;
Ok(Self { raw, duration_secs })
}
/// Compress fingerprint for storage
pub fn to_bytes(&self) -> Vec<u8> {
// Use chromaprint's compressed format
chromaprint::encode_fingerprint(&self.raw, chromaprint::Algorithm::Test1)
}
/// Decompress fingerprint
pub fn from_bytes(bytes: &[u8]) -> Result<Self, FingerprintError> {
let (raw, _) = chromaprint::decode_fingerprint(bytes)?;
Ok(Self { raw, duration_secs: 0 })
}
}
#[derive(Debug, thiserror::Error)]
pub enum FingerprintError {
#[error("No audio track found")]
NoAudioTrack,
#[error("No sample rate")]
NoSampleRate,
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Decode error: {0}")]
Decode(String),
#[error("Chromaprint error: {0}")]
Chromaprint(String),
}
```
---
## Task 2: Fingerprint Search (`musicfs-search/src/fingerprint_search.rs`)
```rust
use crate::fingerprint::AudioFingerprint;
/// Fingerprint similarity search using bit-level comparison
pub struct FingerprintIndex {
db: Arc<Database>,
}
impl FingerprintIndex {
pub fn new(db: Arc<Database>) -> Self {
Self { db }
}
/// Index a file's fingerprint
pub fn index(&self, file_id: FileId, fingerprint: &AudioFingerprint) -> Result<(), SearchError> {
let bytes = fingerprint.to_bytes();
self.db.store_fingerprint(file_id, &bytes, fingerprint.duration_secs)?;
Ok(())
}
/// Search by fingerprint similarity (FR-14.4)
pub fn search(
&self,
query: &AudioFingerprint,
threshold: f32, // 0.0-1.0, higher = more similar
limit: usize,
) -> Result<Vec<FingerprintMatch>, SearchError> {
let candidates = self.db.get_fingerprints_by_duration(
query.duration_secs.saturating_sub(10),
query.duration_secs + 10,
)?;
let mut matches: Vec<FingerprintMatch> = candidates
.into_iter()
.filter_map(|(file_id, fp_bytes, duration)| {
let fp = AudioFingerprint::from_bytes(&fp_bytes).ok()?;
let similarity = self.compare(&query.raw, &fp.raw);
if similarity >= threshold {
Some(FingerprintMatch { file_id, similarity, duration })
} else {
None
}
})
.collect();
// Sort by similarity descending
matches.sort_by(|a, b| b.similarity.partial_cmp(&a.similarity).unwrap());
matches.truncate(limit);
Ok(matches)
}
/// Compare two fingerprints using bit error rate
fn compare(&self, a: &[u32], b: &[u32]) -> f32 {
let len = a.len().min(b.len());
if len == 0 {
return 0.0;
}
let mut matching_bits = 0u32;
let mut total_bits = 0u32;
for i in 0..len {
let xor = a[i] ^ b[i];
matching_bits += 32 - xor.count_ones();
total_bits += 32;
}
matching_bits as f32 / total_bits as f32
}
/// Find duplicates by fingerprint
pub fn find_duplicates(&self, threshold: f32) -> Result<Vec<DuplicateGroup>, SearchError> {
let all_fps = self.db.get_all_fingerprints()?;
let mut groups: Vec<DuplicateGroup> = Vec::new();
let mut processed: HashSet<FileId> = HashSet::new();
for (file_id, fp_bytes, duration) in &all_fps {
if processed.contains(file_id) {
continue;
}
let fp = AudioFingerprint::from_bytes(fp_bytes)?;
let matches = self.search(&fp, threshold, 100)?;
if matches.len() > 1 {
let group = DuplicateGroup {
files: matches.iter().map(|m| m.file_id).collect(),
similarity: matches.iter().map(|m| m.similarity).sum::<f32>() / matches.len() as f32,
};
for m in &matches {
processed.insert(m.file_id);
}
groups.push(group);
}
}
Ok(groups)
}
}
#[derive(Debug)]
pub struct FingerprintMatch {
pub file_id: FileId,
pub similarity: f32,
pub duration: u32,
}
#[derive(Debug)]
pub struct DuplicateGroup {
pub files: Vec<FileId>,
pub similarity: f32,
}
```
---
## Task 3: M4B Audiobook Support (`musicfs-metadata/src/formats/m4b.rs`)
```rust
use symphonia::core::meta::StandardTagKey;
/// M4B audiobook metadata (FR-24.2)
#[derive(Debug, Clone, Default)]
pub struct AudiobookMeta {
pub title: Option<String>,
pub author: Option<String>, // Maps to "artist" in audio
pub narrator: Option<String>,
pub series: Option<String>,
pub series_part: Option<u32>,
pub description: Option<String>,
pub publisher: Option<String>,
pub year: Option<u32>,
pub duration_ms: Option<u64>,
pub chapters: Vec<Chapter>,
}
#[derive(Debug, Clone)]
pub struct Chapter {
pub index: u32,
pub title: String,
pub start_ms: u64,
pub end_ms: u64,
}
impl Chapter {
pub fn duration_ms(&self) -> u64 {
self.end_ms - self.start_ms
}
}
pub struct M4bParser;
impl M4bParser {
/// Parse M4B audiobook with chapters
pub fn parse(&self, path: &Path) -> Result<AudiobookMeta, MetadataError> {
let file = std::fs::File::open(path)?;
let mss = MediaSourceStream::new(Box::new(file), Default::default());
let mut hint = Hint::new();
hint.with_extension("m4b");
let probed = symphonia::default::get_probe()
.format(&hint, mss, &FormatOptions::default(), &MetadataOptions::default())?;
let mut meta = AudiobookMeta::default();
let format = probed.format;
// Extract metadata
if let Some(metadata) = format.metadata().current() {
for tag in metadata.tags() {
if let Some(std_key) = tag.std_key {
let value = tag.value.to_string();
match std_key {
StandardTagKey::TrackTitle | StandardTagKey::Album => {
meta.title = Some(value);
}
StandardTagKey::Artist => {
meta.author = Some(value);
}
StandardTagKey::Composer => {
meta.narrator = Some(value);
}
StandardTagKey::Description => {
meta.description = Some(value);
}
StandardTagKey::Label => {
meta.publisher = Some(value);
}
StandardTagKey::Date => {
meta.year = value.chars().take(4).collect::<String>().parse().ok();
}
_ => {}
}
}
}
}
// Extract chapters from MP4 chpl atom
meta.chapters = self.extract_chapters(&format)?;
// Get total duration
if let Some(track) = format.tracks().first() {
if let (Some(n_frames), Some(sample_rate)) =
(track.codec_params.n_frames, track.codec_params.sample_rate)
{
meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64);
}
}
Ok(meta)
}
fn extract_chapters(&self, format: &dyn FormatReader) -> Result<Vec<Chapter>, MetadataError> {
let mut chapters = Vec::new();
// Symphonia exposes chapters via cues
if let Some(cues) = format.cues() {
for (idx, cue) in cues.iter().enumerate() {
let start_ms = (cue.start_ts as f64 / cue.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64;
// End time is start of next chapter or track end
let end_ms = cues.get(idx + 1)
.map(|next| (next.start_ts as f64 / next.start_offset_ts.unwrap_or(1) as f64 * 1000.0) as u64)
.unwrap_or(u64::MAX); // Will be clamped to duration
chapters.push(Chapter {
index: idx as u32,
title: cue.tags.iter()
.find(|t| t.std_key == Some(StandardTagKey::TrackTitle))
.map(|t| t.value.to_string())
.unwrap_or_else(|| format!("Chapter {}", idx + 1)),
start_ms,
end_ms,
});
}
}
Ok(chapters)
}
}
```
---
## Task 4: Chapter Extraction (`musicfs-metadata/src/chapters.rs`)
```rust
/// Generic chapter support for various formats
pub trait ChapterSource {
fn chapters(&self) -> &[Chapter];
fn chapter_at(&self, position_ms: u64) -> Option<&Chapter>;
}
impl ChapterSource for AudiobookMeta {
fn chapters(&self) -> &[Chapter] {
&self.chapters
}
fn chapter_at(&self, position_ms: u64) -> Option<&Chapter> {
self.chapters.iter()
.find(|c| position_ms >= c.start_ms && position_ms < c.end_ms)
}
}
/// Virtual chapter file generator
pub struct ChapterFileGenerator;
impl ChapterFileGenerator {
/// Generate virtual files for each chapter
/// Example: book.m4b -> book/01 - Introduction.m4b.chapter
pub fn generate_virtual_files(&self, meta: &AudiobookMeta, base_path: &VirtualPath) -> Vec<VirtualChapterFile> {
meta.chapters.iter()
.map(|chapter| {
let filename = format!(
"{:02} - {}.chapter",
chapter.index + 1,
sanitize_filename(&chapter.title)
);
VirtualChapterFile {
path: base_path.join(&filename),
chapter_index: chapter.index,
start_ms: chapter.start_ms,
end_ms: chapter.end_ms,
title: chapter.title.clone(),
}
})
.collect()
}
}
#[derive(Debug)]
pub struct VirtualChapterFile {
pub path: VirtualPath,
pub chapter_index: u32,
pub start_ms: u64,
pub end_ms: u64,
pub title: String,
}
fn sanitize_filename(name: &str) -> String {
name.chars()
.map(|c| match c {
'/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' => '_',
_ => c,
})
.collect()
}
```
---
## Task 5: Virtual Chapter Files (`musicfs-fuse/src/ops/chapters.rs`)
```rust
use crate::VirtualFs;
impl VirtualFs {
/// Handle reads from virtual chapter files
/// These return a byte-range reference to the parent M4B file
pub async fn read_chapter(
&self,
chapter_file: &VirtualChapterFile,
offset: u64,
size: usize,
) -> Result<Vec<u8>, FuseError> {
// Get the parent audiobook file
let parent = self.get_parent_audiobook(&chapter_file.path)?;
// Calculate byte range for this chapter
// This requires knowing the audio bitrate to convert ms -> bytes
let meta = self.get_audiobook_meta(&parent)?;
let bitrate_bps = meta.bitrate.unwrap_or(128_000); // Default 128kbps
let bytes_per_ms = bitrate_bps / 8 / 1000;
let chapter_start_bytes = chapter_file.start_ms * bytes_per_ms;
let chapter_end_bytes = chapter_file.end_ms * bytes_per_ms;
// Adjust offset to be within chapter
let actual_offset = chapter_start_bytes + offset;
let max_size = (chapter_end_bytes - actual_offset) as usize;
let read_size = size.min(max_size);
// Read from the actual file
self.read_file(&parent, actual_offset, read_size).await
}
/// List chapter files for an audiobook
pub fn list_chapters(&self, audiobook_path: &VirtualPath) -> Result<Vec<DirEntry>, FuseError> {
let meta = self.get_audiobook_meta(audiobook_path)?;
let generator = ChapterFileGenerator;
let chapters = generator.generate_virtual_files(&meta, audiobook_path);
Ok(chapters.into_iter()
.map(|c| DirEntry {
name: c.path.filename().to_string(),
kind: FileType::RegularFile,
size: self.estimate_chapter_size(&c),
})
.collect())
}
fn estimate_chapter_size(&self, chapter: &VirtualChapterFile) -> u64 {
// Estimate based on duration and typical bitrate
let duration_secs = (chapter.end_ms - chapter.start_ms) / 1000;
duration_secs * 128_000 / 8 // 128kbps assumption
}
}
```
---
## Task 6: Fingerprint Search Virtual Directory
```rust
/// Virtual directory for fingerprint search
/// /.search/fingerprint/{base64_fingerprint} -> matching files
impl SearchOps {
pub async fn search_by_fingerprint(
&self,
fingerprint_path: &str,
) -> Result<Vec<SearchResult>, SearchError> {
// Path format: /.search/fingerprint/{base64_encoded_fingerprint}
let fp_bytes = base64::decode(fingerprint_path)
.map_err(|_| SearchError::InvalidQuery)?;
let fingerprint = AudioFingerprint::from_bytes(&fp_bytes)?;
let matches = self.fingerprint_index.search(&fingerprint, 0.8, 20)?;
let mut results = Vec::new();
for m in matches {
if let Some(file) = self.db.get_file_by_id(m.file_id)? {
results.push(SearchResult {
path: file.virtual_path,
score: m.similarity,
snippet: format!("Similarity: {:.1}%", m.similarity * 100.0),
});
}
}
Ok(results)
}
}
```
---
## Database Schema Additions
```sql
-- Fingerprint storage
CREATE TABLE IF NOT EXISTS fingerprints (
file_id INTEGER PRIMARY KEY REFERENCES files(id) ON DELETE CASCADE,
fingerprint BLOB NOT NULL, -- Compressed chromaprint
duration INTEGER NOT NULL, -- Duration in seconds
indexed_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
);
CREATE INDEX IF NOT EXISTS idx_fingerprints_duration ON fingerprints(duration);
-- Audiobook chapters
CREATE TABLE IF NOT EXISTS chapters (
id INTEGER PRIMARY KEY,
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
chapter_idx INTEGER NOT NULL,
title TEXT NOT NULL,
start_ms INTEGER NOT NULL,
end_ms INTEGER NOT NULL,
UNIQUE(file_id, chapter_idx)
);
CREATE INDEX IF NOT EXISTS idx_chapters_file ON chapters(file_id);
```
---
## Tests
| Test | Type | Validates |
|------|------|-----------|
| `test_fingerprint_generation` | Unit | Chromaprint from audio (FR-14.4) |
| `test_fingerprint_similarity` | Unit | Bit comparison algorithm |
| `test_fingerprint_search` | Integration | Find similar tracks |
| `test_fingerprint_duplicates` | Integration | Detect duplicate audio |
| `test_m4b_parsing` | Unit | M4B metadata extraction (FR-24.2) |
| `test_chapter_extraction` | Unit | Chapter list from M4B |
| `test_virtual_chapter_files` | Integration | Chapter files appear in listing |
| `test_chapter_read` | Integration | Read chapter content |
| `test_audiobook_navigation` | E2E | Browse audiobook chapters |
---
## Exit Criteria
- [ ] Audio fingerprints generated from audio files
- [ ] Fingerprint similarity search finds matching tracks
- [ ] Duplicate detection works across library
- [ ] M4B files parsed with full metadata
- [ ] Chapters extracted and stored
- [ ] Virtual chapter files appear in directory listing
- [ ] Chapter files are readable (return correct byte range)
- [ ] All tests pass
---
## Architecture Alignment
Per requirements.md:
- FR-14.4: Audio fingerprint search ✓
- FR-24.2: Audiobook formats with chapters ✓
Per architecture.md section 4.3.4:
- FormatPlugin trait for M4B support ✓
- Chapter extraction via symphonia ✓
+1403 -12
View File
File diff suppressed because it is too large Load Diff
+205 -30
View File
@@ -1,5 +1,5 @@
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use clap::Parser; use clap::{Parser, Subcommand};
use musicfs_cache::TreeBuilder; use musicfs_cache::TreeBuilder;
use musicfs_cas::{CasConfig, CasStore, ContentFetcher, FileReader}; use musicfs_cas::{CasConfig, CasStore, ContentFetcher, FileReader};
use musicfs_core::{FileId, FileMeta, OriginId, RealPath, VirtualPath}; use musicfs_core::{FileId, FileMeta, OriginId, RealPath, VirtualPath};
@@ -15,33 +15,111 @@ use tracing::{debug, info};
#[command(name = "musicfs")] #[command(name = "musicfs")]
#[command(about = "Virtual FUSE filesystem for music libraries")] #[command(about = "Virtual FUSE filesystem for music libraries")]
struct Cli { struct Cli {
#[arg(help = "Mount point for the virtual filesystem")] #[arg(short, long, default_value = "info", help = "Log level")]
mountpoint: PathBuf,
#[arg(short, long, help = "Source music directory (origin)")]
origin: PathBuf,
#[arg(short, long, help = "Cache directory for CAS chunks")]
cache_dir: Option<PathBuf>,
#[arg(short, long, default_value = "info", help = "Log level (debug, info, warn, error)")]
log_level: String, log_level: String,
#[command(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
Mount {
#[arg(short, long, help = "Config file path")]
config: Option<PathBuf>,
#[arg(help = "Mount point")]
mountpoint: PathBuf,
#[arg(short, long, help = "Source music directory")]
origin: Option<PathBuf>,
#[arg(short = 'd', long, help = "Cache directory")]
cache_dir: Option<PathBuf>,
},
Status,
Cache {
#[command(subcommand)]
command: CacheCommands,
},
Search {
query: String,
#[arg(short, long, default_value = "100")]
limit: u32,
},
Origin {
#[command(subcommand)]
command: OriginCommands,
},
Events {
#[arg(short, long, help = "Filter by event type")]
r#type: Option<String>,
},
Shutdown {
#[arg(short, long, default_value = "true")]
graceful: bool,
#[arg(short, long, default_value = "30")]
timeout: u32,
},
}
#[derive(Subcommand)]
enum CacheCommands {
Stats,
Clear {
#[arg(help = "Origin to clear cache for")]
origin: Option<String>,
},
Prefetch {
#[arg(help = "Paths to prefetch")]
paths: Vec<String>,
},
}
#[derive(Subcommand)]
enum OriginCommands {
List,
Health {
origin_id: String,
},
Rescan {
origin_id: String,
},
} }
fn main() -> Result<()> { fn main() -> Result<()> {
let cli = Cli::parse(); let cli = Cli::parse();
init_logging(&cli.log_level); init_logging(&cli.log_level);
match cli.command {
Commands::Mount {
config: _,
mountpoint,
origin,
cache_dir,
} => run_mount(mountpoint, origin, cache_dir),
Commands::Status => run_status(),
Commands::Cache { command } => run_cache(command),
Commands::Search { query, limit } => run_search(&query, limit),
Commands::Origin { command } => run_origin(command),
Commands::Events { r#type } => run_events(r#type),
Commands::Shutdown { graceful, timeout } => run_shutdown(graceful, timeout),
}
}
fn run_mount(
mountpoint: PathBuf,
origin_path: Option<PathBuf>,
cache_dir: Option<PathBuf>,
) -> Result<()> {
let origin_path = origin_path.context("--origin is required for mount")?;
let runtime = tokio::runtime::Runtime::new().context("Failed to create Tokio runtime")?; let runtime = tokio::runtime::Runtime::new().context("Failed to create Tokio runtime")?;
let handle = runtime.handle().clone(); let handle = runtime.handle().clone();
let (tree, reader) = runtime.block_on(async { let (tree, reader) = runtime.block_on(async {
info!("MusicFS starting..."); info!("MusicFS starting...");
info!("Origin: {:?}", cli.origin); info!("Origin: {:?}", origin_path);
info!("Mountpoint: {:?}", cli.mountpoint); info!("Mountpoint: {:?}", mountpoint);
let cache_dir = cli.cache_dir.unwrap_or_else(|| { let cache_dir = cache_dir.unwrap_or_else(|| {
dirs::cache_dir() dirs::cache_dir()
.unwrap_or_else(|| PathBuf::from("/tmp")) .unwrap_or_else(|| PathBuf::from("/tmp"))
.join("musicfs") .join("musicfs")
@@ -49,24 +127,28 @@ fn main() -> Result<()> {
info!("Cache directory: {:?}", cache_dir); info!("Cache directory: {:?}", cache_dir);
std::fs::create_dir_all(&cache_dir).context("Failed to create cache directory")?; std::fs::create_dir_all(&cache_dir).context("Failed to create cache directory")?;
std::fs::create_dir_all(&cli.mountpoint).context("Failed to create mountpoint")?; std::fs::create_dir_all(&mountpoint).context("Failed to create mountpoint")?;
let cas_config = CasConfig { let cas_config = CasConfig {
chunks_dir: cache_dir.join("chunks"), chunks_dir: cache_dir.join("chunks"),
..Default::default() ..Default::default()
}; };
let store = Arc::new(CasStore::open(cas_config).await.context("Failed to open CAS store")?); let store = Arc::new(
CasStore::open(cas_config)
.await
.context("Failed to open CAS store")?,
);
info!("CAS store initialized"); info!("CAS store initialized");
let origin_id = OriginId::from("local"); let origin_id = OriginId::from("local");
let origin = Arc::new(LocalOrigin::new(origin_id.clone(), cli.origin.clone())); let origin = Arc::new(LocalOrigin::new(origin_id.clone(), origin_path.clone()));
info!("Origin registered: {}", origin.display_name()); info!("Origin registered: {}", origin.display_name());
let fetcher = Arc::new(ContentFetcher::new(store.clone())); let fetcher = Arc::new(ContentFetcher::new(store.clone()));
fetcher.register_origin(origin); fetcher.register_origin(origin);
info!("Scanning music files..."); info!("Scanning music files...");
let files = scan_music_files(&cli.origin, &origin_id).await?; let files = scan_music_files(&origin_path, &origin_id).await?;
info!("Found {} music files", files.len()); info!("Found {} music files", files.len());
let mut builder = TreeBuilder::new(); let mut builder = TreeBuilder::new();
@@ -84,19 +166,84 @@ fn main() -> Result<()> {
let fs = MusicFs::with_reader(tree, reader, handle); let fs = MusicFs::with_reader(tree, reader, handle);
info!("Mounting filesystem at {:?}", cli.mountpoint); info!("Mounting filesystem at {:?}", mountpoint);
info!("Press Ctrl+C to unmount"); info!("Press Ctrl+C to unmount");
fs.mount(&cli.mountpoint).context("Failed to mount filesystem")?; fs.mount(&mountpoint)
.context("Failed to mount filesystem")?;
Ok(()) Ok(())
} }
fn run_status() -> Result<()> {
println!("Status: Not connected to daemon");
println!("Hint: gRPC client integration pending");
Ok(())
}
fn run_cache(command: CacheCommands) -> Result<()> {
match command {
CacheCommands::Stats => {
println!("Cache stats: gRPC client integration pending");
}
CacheCommands::Clear { origin } => {
println!(
"Clearing cache for: {}",
origin.as_deref().unwrap_or("all")
);
println!("gRPC client integration pending");
}
CacheCommands::Prefetch { paths } => {
println!("Prefetching {} paths", paths.len());
println!("gRPC client integration pending");
}
}
Ok(())
}
fn run_search(query: &str, limit: u32) -> Result<()> {
println!("Searching for: {} (limit: {})", query, limit);
println!("gRPC client integration pending");
Ok(())
}
fn run_origin(command: OriginCommands) -> Result<()> {
match command {
OriginCommands::List => {
println!("Origins: gRPC client integration pending");
}
OriginCommands::Health { origin_id } => {
println!("Health for {}: gRPC client integration pending", origin_id);
}
OriginCommands::Rescan { origin_id } => {
println!("Rescanning {}: gRPC client integration pending", origin_id);
}
}
Ok(())
}
fn run_events(event_type: Option<String>) -> Result<()> {
println!(
"Subscribing to events: {}",
event_type.as_deref().unwrap_or("all")
);
println!("gRPC client integration pending");
Ok(())
}
fn run_shutdown(graceful: bool, timeout: u32) -> Result<()> {
println!(
"Shutdown requested (graceful: {}, timeout: {}s)",
graceful, timeout
);
println!("gRPC client integration pending");
Ok(())
}
fn init_logging(level: &str) { fn init_logging(level: &str) {
use tracing_subscriber::{fmt, prelude::*, EnvFilter}; use tracing_subscriber::{fmt, prelude::*, EnvFilter};
let filter = EnvFilter::try_from_default_env() let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(level));
.unwrap_or_else(|_| EnvFilter::new(level));
tracing_subscriber::registry() tracing_subscriber::registry()
.with(fmt::layer()) .with(fmt::layer())
@@ -109,7 +256,15 @@ async fn scan_music_files(dir: &Path, origin_id: &OriginId) -> Result<Vec<FileMe
let mut files = Vec::new(); let mut files = Vec::new();
let mut file_id_counter = 1i64; let mut file_id_counter = 1i64;
scan_dir_recursive(dir, dir, origin_id, &parser, &mut files, &mut file_id_counter).await?; scan_dir_recursive(
dir,
dir,
origin_id,
&parser,
&mut files,
&mut file_id_counter,
)
.await?;
Ok(files) Ok(files)
} }
@@ -129,7 +284,10 @@ async fn scan_dir_recursive(
let metadata = entry.metadata().await?; let metadata = entry.metadata().await?;
if metadata.is_dir() { if metadata.is_dir() {
Box::pin(scan_dir_recursive(base, &path, origin_id, parser, files, id_counter)).await?; Box::pin(scan_dir_recursive(
base, &path, origin_id, parser, files, id_counter,
))
.await?;
} else if is_audio_file(&path) { } else if is_audio_file(&path) {
let relative_path = path.strip_prefix(base).unwrap_or(&path); let relative_path = path.strip_prefix(base).unwrap_or(&path);
@@ -156,7 +314,10 @@ async fn scan_dir_recursive(
audio: audio_meta, audio: audio_meta,
}; };
debug!("Found: {:?} -> {:?}", file_meta.real_path.path, file_meta.virtual_path); debug!(
"Found: {:?} -> {:?}",
file_meta.real_path.path, file_meta.virtual_path
);
files.push(file_meta); files.push(file_meta);
*id_counter += 1; *id_counter += 1;
} }
@@ -167,7 +328,10 @@ async fn scan_dir_recursive(
fn is_audio_file(path: &Path) -> bool { fn is_audio_file(path: &Path) -> bool {
matches!( matches!(
path.extension().and_then(|e| e.to_str()).map(|e| e.to_lowercase()).as_deref(), path.extension()
.and_then(|e| e.to_str())
.map(|e| e.to_lowercase())
.as_deref(),
Some("flac" | "mp3" | "ogg" | "wav" | "m4a" | "aac" | "opus") Some("flac" | "mp3" | "ogg" | "wav" | "m4a" | "aac" | "opus")
) )
} }
@@ -176,11 +340,22 @@ fn build_virtual_path(path: &Path, audio: Option<&musicfs_core::AudioMeta>) -> V
if let Some(meta) = audio { if let Some(meta) = audio {
let artist = meta.artist.as_deref().unwrap_or("Unknown Artist"); let artist = meta.artist.as_deref().unwrap_or("Unknown Artist");
let album = meta.album.as_deref().unwrap_or("Unknown Album"); let album = meta.album.as_deref().unwrap_or("Unknown Album");
let filename = path.file_name().and_then(|n| n.to_str()).unwrap_or("track"); let filename = path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("track");
VirtualPath::new(&format!("/{}/{}/{}", sanitize(artist), sanitize(album), filename)) VirtualPath::new(&format!(
"/{}/{}/{}",
sanitize(artist),
sanitize(album),
filename
))
} else { } else {
let filename = path.file_name().and_then(|n| n.to_str()).unwrap_or("unknown"); let filename = path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("unknown");
VirtualPath::new(&format!("/Unknown Artist/Unknown Album/{}", filename)) VirtualPath::new(&format!("/Unknown Artist/Unknown Album/{}", filename))
} }
} }
+2
View File
@@ -2,6 +2,7 @@ pub mod config;
pub mod credentials; pub mod credentials;
pub mod error; pub mod error;
pub mod events; pub mod events;
pub mod metrics;
pub mod resolver; pub mod resolver;
pub mod types; pub mod types;
@@ -9,5 +10,6 @@ pub use config::{CacheConfig, Config, ConfigError, HealthConfig, OriginConfig, O
pub use credentials::{Credential, CredentialConfig, CredentialError, CredentialStore}; pub use credentials::{Credential, CredentialConfig, CredentialError, CredentialStore};
pub use error::{Error, Result}; pub use error::{Error, Result};
pub use events::{Event, EventBus}; pub use events::{Event, EventBus};
pub use metrics::{CacheMetrics, FuseOpsMetrics, Metrics, OriginsMetrics};
pub use resolver::{PathResolver, PathTemplate}; pub use resolver::{PathResolver, PathTemplate};
pub use types::*; pub use types::*;
+322
View File
@@ -0,0 +1,322 @@
use std::collections::HashMap;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::RwLock;
use std::time::Instant;
#[derive(Default)]
pub struct Metrics {
pub fuse_ops: FuseOpsMetrics,
pub fuse_latency: FuseLatencyMetrics,
pub cache: CacheMetrics,
pub origins: OriginsMetrics,
pub origin_health: OriginHealthMetrics,
start_time: Option<Instant>,
}
impl Metrics {
pub fn new() -> Self {
Self {
start_time: Some(Instant::now()),
..Default::default()
}
}
pub fn uptime_secs(&self) -> u64 {
self.start_time
.map(|t| t.elapsed().as_secs())
.unwrap_or(0)
}
pub fn to_prometheus(&self) -> String {
let mut output = String::new();
output.push_str(&format!(
"# HELP musicfs_fuse_ops_total Total FUSE operations\n\
# TYPE musicfs_fuse_ops_total counter\n\
musicfs_fuse_ops_total{{op=\"lookup\"}} {}\n\
musicfs_fuse_ops_total{{op=\"getattr\"}} {}\n\
musicfs_fuse_ops_total{{op=\"read\"}} {}\n\
musicfs_fuse_ops_total{{op=\"readdir\"}} {}\n\
musicfs_fuse_ops_total{{op=\"open\"}} {}\n",
self.fuse_ops.lookup.load(Ordering::Relaxed),
self.fuse_ops.getattr.load(Ordering::Relaxed),
self.fuse_ops.read.load(Ordering::Relaxed),
self.fuse_ops.readdir.load(Ordering::Relaxed),
self.fuse_ops.open.load(Ordering::Relaxed),
));
for (op, histogram) in self.fuse_latency.histograms.read().unwrap().iter() {
let quantiles = histogram.quantiles();
output.push_str(&format!(
"# HELP musicfs_fuse_latency_seconds FUSE operation latency\n\
# TYPE musicfs_fuse_latency_seconds summary\n\
musicfs_fuse_latency_seconds{{op=\"{}\",quantile=\"0.5\"}} {:.6}\n\
musicfs_fuse_latency_seconds{{op=\"{}\",quantile=\"0.95\"}} {:.6}\n\
musicfs_fuse_latency_seconds{{op=\"{}\",quantile=\"0.99\"}} {:.6}\n\
musicfs_fuse_latency_seconds_sum{{op=\"{}\"}} {:.6}\n\
musicfs_fuse_latency_seconds_count{{op=\"{}\"}} {}\n",
op, quantiles.p50,
op, quantiles.p95,
op, quantiles.p99,
op, histogram.sum_secs(),
op, histogram.count(),
));
}
output.push_str(&format!(
"# HELP musicfs_cache_hits_total Cache hits\n\
# TYPE musicfs_cache_hits_total counter\n\
musicfs_cache_hits_total {}\n",
self.cache.hits.load(Ordering::Relaxed),
));
output.push_str(&format!(
"# HELP musicfs_cache_misses_total Cache misses\n\
# TYPE musicfs_cache_misses_total counter\n\
musicfs_cache_misses_total {}\n",
self.cache.misses.load(Ordering::Relaxed),
));
output.push_str(&format!(
"# HELP musicfs_cache_size_bytes Current cache size in bytes\n\
# TYPE musicfs_cache_size_bytes gauge\n\
musicfs_cache_size_bytes {}\n",
self.cache.size_bytes.load(Ordering::Relaxed),
));
output.push_str(&format!(
"# HELP musicfs_cache_chunks_total Number of cached chunks\n\
# TYPE musicfs_cache_chunks_total gauge\n\
musicfs_cache_chunks_total {}\n",
self.cache.chunk_count.load(Ordering::Relaxed),
));
output.push_str(
"# HELP musicfs_origin_health Origin health status (1=healthy, 0=unhealthy)\n\
# TYPE musicfs_origin_health gauge\n",
);
for (origin_id, healthy) in self.origin_health.status.read().unwrap().iter() {
output.push_str(&format!(
"musicfs_origin_health{{origin=\"{}\"}} {}\n",
origin_id,
if *healthy { 1 } else { 0 }
));
}
output.push_str(&format!(
"# HELP musicfs_uptime_seconds Daemon uptime in seconds\n\
# TYPE musicfs_uptime_seconds gauge\n\
musicfs_uptime_seconds {}\n",
self.uptime_secs(),
));
output
}
pub fn hit_ratio(&self) -> f64 {
let hits = self.cache.hits.load(Ordering::Relaxed) as f64;
let misses = self.cache.misses.load(Ordering::Relaxed) as f64;
let total = hits + misses;
if total == 0.0 {
0.0
} else {
hits / total
}
}
}
#[derive(Default)]
pub struct FuseOpsMetrics {
pub lookup: AtomicU64,
pub getattr: AtomicU64,
pub read: AtomicU64,
pub readdir: AtomicU64,
pub open: AtomicU64,
}
impl FuseOpsMetrics {
pub fn record_lookup(&self) {
self.lookup.fetch_add(1, Ordering::Relaxed);
}
pub fn record_getattr(&self) {
self.getattr.fetch_add(1, Ordering::Relaxed);
}
pub fn record_read(&self) {
self.read.fetch_add(1, Ordering::Relaxed);
}
pub fn record_readdir(&self) {
self.readdir.fetch_add(1, Ordering::Relaxed);
}
pub fn record_open(&self) {
self.open.fetch_add(1, Ordering::Relaxed);
}
}
#[derive(Default)]
pub struct CacheMetrics {
pub hits: AtomicU64,
pub misses: AtomicU64,
pub size_bytes: AtomicU64,
pub chunk_count: AtomicU64,
}
impl CacheMetrics {
pub fn record_hit(&self) {
self.hits.fetch_add(1, Ordering::Relaxed);
}
pub fn record_miss(&self) {
self.misses.fetch_add(1, Ordering::Relaxed);
}
pub fn update_size(&self, size: u64) {
self.size_bytes.store(size, Ordering::Relaxed);
}
pub fn update_chunk_count(&self, count: u64) {
self.chunk_count.store(count, Ordering::Relaxed);
}
}
#[derive(Default)]
pub struct OriginsMetrics {
pub healthy_count: AtomicU64,
pub total_count: AtomicU64,
}
impl OriginsMetrics {
pub fn update(&self, healthy: u64, total: u64) {
self.healthy_count.store(healthy, Ordering::Relaxed);
self.total_count.store(total, Ordering::Relaxed);
}
}
#[derive(Default)]
pub struct FuseLatencyMetrics {
pub histograms: RwLock<HashMap<String, LatencyHistogram>>,
}
impl FuseLatencyMetrics {
pub fn record(&self, op: &str, latency_secs: f64) {
let mut histograms = self.histograms.write().unwrap();
histograms
.entry(op.to_string())
.or_default()
.record(latency_secs);
}
}
#[derive(Default)]
pub struct LatencyHistogram {
samples: Vec<f64>,
sum: f64,
}
impl LatencyHistogram {
pub fn record(&mut self, latency_secs: f64) {
self.samples.push(latency_secs);
self.sum += latency_secs;
if self.samples.len() > 10000 {
self.samples.drain(..5000);
}
}
pub fn quantiles(&self) -> Quantiles {
if self.samples.is_empty() {
return Quantiles::default();
}
let mut sorted = self.samples.clone();
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
let len = sorted.len();
Quantiles {
p50: sorted[len / 2],
p95: sorted[(len as f64 * 0.95) as usize],
p99: sorted[(len as f64 * 0.99) as usize],
}
}
pub fn sum_secs(&self) -> f64 {
self.sum
}
pub fn count(&self) -> u64 {
self.samples.len() as u64
}
}
#[derive(Default)]
pub struct Quantiles {
pub p50: f64,
pub p95: f64,
pub p99: f64,
}
#[derive(Default)]
pub struct OriginHealthMetrics {
pub status: RwLock<HashMap<String, bool>>,
}
impl OriginHealthMetrics {
pub fn set_health(&self, origin_id: &str, healthy: bool) {
self.status
.write()
.unwrap()
.insert(origin_id.to_string(), healthy);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_metrics_new() {
let metrics = Metrics::new();
assert!(metrics.uptime_secs() < 5);
}
#[test]
fn test_fuse_ops_recording() {
let metrics = Metrics::new();
metrics.fuse_ops.record_lookup();
metrics.fuse_ops.record_lookup();
metrics.fuse_ops.record_read();
assert_eq!(metrics.fuse_ops.lookup.load(Ordering::Relaxed), 2);
assert_eq!(metrics.fuse_ops.read.load(Ordering::Relaxed), 1);
}
#[test]
fn test_cache_hit_ratio() {
let metrics = Metrics::new();
metrics.cache.hits.store(8, Ordering::Relaxed);
metrics.cache.misses.store(2, Ordering::Relaxed);
assert!((metrics.hit_ratio() - 0.8).abs() < 0.001);
}
#[test]
fn test_cache_hit_ratio_zero() {
let metrics = Metrics::new();
assert_eq!(metrics.hit_ratio(), 0.0);
}
#[test]
fn test_prometheus_format() {
let metrics = Metrics::new();
metrics.fuse_ops.record_lookup();
metrics.cache.record_hit();
let output = metrics.to_prometheus();
assert!(output.contains("musicfs_fuse_ops_total"));
assert!(output.contains("musicfs_cache_hits_total"));
}
}
+7
View File
@@ -11,6 +11,13 @@ prost.workspace = true
tokio.workspace = true tokio.workspace = true
tokio-stream.workspace = true tokio-stream.workspace = true
tracing.workspace = true tracing.workspace = true
serde.workspace = true
serde_json.workspace = true
chrono.workspace = true
reqwest = { version = "0.11", features = ["json"] }
hmac = "0.12"
sha2 = "0.10"
hex.workspace = true
[build-dependencies] [build-dependencies]
tonic-build.workspace = true tonic-build.workspace = true
@@ -5,8 +5,19 @@ package musicfs.v1;
service MusicFS { service MusicFS {
rpc Search(SearchRequest) returns (SearchResponse); rpc Search(SearchRequest) returns (SearchResponse);
rpc SearchStream(SearchRequest) returns (stream SearchResult); rpc SearchStream(SearchRequest) returns (stream SearchResult);
rpc GetStatus(Empty) returns (StatusResponse);
rpc Shutdown(ShutdownRequest) returns (Empty);
rpc GetCacheStats(Empty) returns (CacheStats);
rpc ClearCache(ClearCacheRequest) returns (ClearCacheResponse);
rpc Prefetch(PrefetchRequest) returns (stream PrefetchProgress);
rpc ListOrigins(Empty) returns (OriginsResponse);
rpc GetOriginHealth(OriginRequest) returns (OriginHealthResponse);
rpc RescanOrigin(OriginRequest) returns (stream SyncProgress);
rpc SubscribeEvents(EventFilter) returns (stream Event);
} }
message Empty {}
message SearchRequest { message SearchRequest {
string query = 1; string query = 1;
optional uint32 limit = 2; optional uint32 limit = 2;
@@ -29,3 +40,137 @@ message SearchResult {
float score = 6; float score = 6;
map<string, string> highlights = 7; map<string, string> highlights = 7;
} }
enum MountState {
MOUNT_UNKNOWN = 0;
MOUNT_MOUNTING = 1;
MOUNT_READY = 2;
MOUNT_SYNCING = 3;
MOUNT_DEGRADED = 4;
MOUNT_UNMOUNTING = 5;
}
message StatusResponse {
string version = 1;
uint64 uptime_secs = 2;
string mount_point = 3;
MountState state = 4;
uint32 open_file_handles = 5;
uint64 fuse_ops_total = 6;
uint64 files_indexed = 7;
uint64 cache_size_bytes = 8;
repeated OriginStatus origins = 9;
}
message OriginStatus {
string id = 1;
string origin_type = 2;
HealthStatus health = 3;
uint64 files_count = 4;
}
enum HealthStatus {
HEALTH_UNKNOWN = 0;
HEALTH_HEALTHY = 1;
HEALTH_DEGRADED = 2;
HEALTH_UNHEALTHY = 3;
}
message ShutdownRequest {
bool graceful = 1;
uint32 timeout_secs = 2;
}
message TierStats {
uint64 entries = 1;
uint64 size_bytes = 2;
uint64 hits = 3;
uint64 misses = 4;
}
message CacheStats {
uint64 total_size_bytes = 1;
uint64 used_size_bytes = 2;
uint64 size_limit_bytes = 3;
uint64 chunk_count = 4;
uint64 chunks_unique = 5;
double dedup_ratio = 6;
uint64 hit_count = 7;
uint64 miss_count = 8;
double hit_ratio = 9;
uint64 metadata_entries = 10;
uint64 metadata_bytes = 11;
TierStats l1_metadata = 12;
TierStats l2_headers = 13;
TierStats l3_chunks = 14;
}
message ClearCacheRequest {
optional string origin_id = 1;
bool clear_metadata = 2;
bool clear_chunks = 3;
}
message ClearCacheResponse {
uint64 bytes_cleared = 1;
uint64 chunks_cleared = 2;
}
message PrefetchRequest {
repeated string paths = 1;
optional string origin_id = 2;
}
message PrefetchProgress {
string current_path = 1;
uint32 completed = 2;
uint32 total = 3;
uint64 bytes_fetched = 4;
}
message OriginsResponse {
repeated OriginInfo origins = 1;
}
message OriginInfo {
string id = 1;
string origin_type = 2;
string display_name = 3;
string root_path = 4;
HealthStatus health = 5;
uint64 files_count = 6;
uint64 total_size_bytes = 7;
}
message OriginRequest {
string origin_id = 1;
}
message OriginHealthResponse {
string origin_id = 1;
HealthStatus status = 2;
optional string message = 3;
uint64 last_check_secs = 4;
}
message SyncProgress {
string phase = 1;
uint32 current = 2;
uint32 total = 3;
string current_path = 4;
uint64 bytes_synced = 5;
}
message EventFilter {
repeated string event_types = 1;
optional string origin_id = 2;
}
message Event {
string event_type = 1;
int64 timestamp_ms = 2;
optional string origin_id = 3;
optional string path = 4;
optional int64 file_id = 5;
map<string, string> metadata = 6;
}
+6 -1
View File
@@ -7,6 +7,11 @@ pub mod proto {
} }
mod search_service; mod search_service;
mod server;
mod webhook;
pub use proto::musicfs::v1::music_fs_server::{MusicFs, MusicFsServer}; pub use proto::musicfs::v1::music_fs_server::{MusicFs, MusicFsServer as MusicFsGrpcServer};
pub use proto::musicfs::v1::*;
pub use search_service::SearchService; pub use search_service::SearchService;
pub use server::MusicFsServer;
pub use webhook::{WebhookConfig, WebhookHandler, WebhookPayload};
@@ -1,9 +1,13 @@
use crate::proto::musicfs::v1::{ use crate::proto::musicfs::v1::{
music_fs_server::MusicFs, SearchRequest, SearchResponse, SearchResult, music_fs_server::MusicFs, CacheStats, ClearCacheRequest, ClearCacheResponse, Empty, Event,
EventFilter, OriginHealthResponse, OriginRequest, OriginsResponse, PrefetchProgress,
PrefetchRequest, SearchRequest, SearchResponse, SearchResult, ShutdownRequest, StatusResponse,
SyncProgress,
}; };
use musicfs_search::SearchIndex; use musicfs_search::SearchIndex;
use std::sync::Arc; use std::sync::Arc;
use std::time::Instant; use std::time::Instant;
use tokio_stream::wrappers::ReceiverStream;
use tonic::{Request, Response, Status}; use tonic::{Request, Response, Status};
use tracing::debug; use tracing::debug;
@@ -74,7 +78,7 @@ impl MusicFs for SearchService {
})) }))
} }
type SearchStreamStream = tokio_stream::wrappers::ReceiverStream<Result<SearchResult, Status>>; type SearchStreamStream = ReceiverStream<Result<SearchResult, Status>>;
async fn search_stream( async fn search_stream(
&self, &self,
@@ -112,9 +116,94 @@ impl MusicFs for SearchService {
} }
}); });
Ok(Response::new(tokio_stream::wrappers::ReceiverStream::new( Ok(Response::new(ReceiverStream::new(rx)))
rx, }
)))
async fn get_status(
&self,
_request: Request<Empty>,
) -> Result<Response<StatusResponse>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
async fn shutdown(
&self,
_request: Request<ShutdownRequest>,
) -> Result<Response<Empty>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
async fn get_cache_stats(
&self,
_request: Request<Empty>,
) -> Result<Response<CacheStats>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
async fn clear_cache(
&self,
_request: Request<ClearCacheRequest>,
) -> Result<Response<ClearCacheResponse>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
type PrefetchStream = ReceiverStream<Result<PrefetchProgress, Status>>;
async fn prefetch(
&self,
_request: Request<PrefetchRequest>,
) -> Result<Response<Self::PrefetchStream>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
async fn list_origins(
&self,
_request: Request<Empty>,
) -> Result<Response<OriginsResponse>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
async fn get_origin_health(
&self,
_request: Request<OriginRequest>,
) -> Result<Response<OriginHealthResponse>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
type RescanOriginStream = ReceiverStream<Result<SyncProgress, Status>>;
async fn rescan_origin(
&self,
_request: Request<OriginRequest>,
) -> Result<Response<Self::RescanOriginStream>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
}
type SubscribeEventsStream = ReceiverStream<Result<Event, Status>>;
async fn subscribe_events(
&self,
_request: Request<EventFilter>,
) -> Result<Response<Self::SubscribeEventsStream>, Status> {
Err(Status::unimplemented(
"Use MusicFsServer for control operations",
))
} }
} }
+428
View File
@@ -0,0 +1,428 @@
use crate::proto::musicfs::v1::{
music_fs_server::MusicFs, CacheStats, ClearCacheRequest, ClearCacheResponse, Empty, Event,
EventFilter, HealthStatus, MountState, OriginHealthResponse, OriginRequest, OriginsResponse,
PrefetchProgress, PrefetchRequest, SearchRequest, SearchResponse, SearchResult,
ShutdownRequest, StatusResponse, SyncProgress, TierStats,
};
use musicfs_core::{Event as CoreEvent, EventBus};
use std::sync::Arc;
use std::time::{Duration, Instant};
use tokio::sync::mpsc;
use tokio_stream::wrappers::ReceiverStream;
use tonic::{Request, Response, Status};
use tracing::{debug, info};
pub struct MusicFsServer {
start_time: Instant,
event_bus: Arc<EventBus>,
version: String,
}
impl MusicFsServer {
pub fn new(event_bus: Arc<EventBus>) -> Self {
Self {
start_time: Instant::now(),
event_bus,
version: env!("CARGO_PKG_VERSION").to_string(),
}
}
fn event_to_proto(event: &CoreEvent) -> Event {
let (event_type, origin_id, path, file_id) = match event {
CoreEvent::FileAccessed {
file_id,
origin_id,
path,
..
} => (
"file_accessed".to_string(),
Some(origin_id.to_string()),
Some(path.as_str().to_string()),
Some(file_id.0),
),
CoreEvent::FileAdded { path, origin_id } => (
"file_added".to_string(),
Some(origin_id.to_string()),
Some(path.as_str().to_string()),
None,
),
CoreEvent::FileRemoved { path, file_id } => (
"file_removed".to_string(),
None,
Some(path.as_str().to_string()),
file_id.map(|id| id.0),
),
CoreEvent::FileModified { path } => (
"file_modified".to_string(),
None,
Some(path.as_str().to_string()),
None,
),
CoreEvent::SyncStarted { origin_id } => (
"sync_started".to_string(),
Some(origin_id.to_string()),
None,
None,
),
CoreEvent::SyncCompleted {
origin_id,
files_changed,
} => {
let mut metadata = std::collections::HashMap::new();
metadata.insert("files_changed".to_string(), files_changed.to_string());
return Event {
event_type: "sync_completed".to_string(),
timestamp_ms: chrono::Utc::now().timestamp_millis(),
origin_id: Some(origin_id.to_string()),
path: None,
file_id: None,
metadata,
};
}
CoreEvent::OriginHealthChanged { origin_id, healthy } => {
let mut metadata = std::collections::HashMap::new();
metadata.insert("healthy".to_string(), healthy.to_string());
return Event {
event_type: "origin_health_changed".to_string(),
timestamp_ms: chrono::Utc::now().timestamp_millis(),
origin_id: Some(origin_id.to_string()),
path: None,
file_id: None,
metadata,
};
}
CoreEvent::CacheEviction { bytes_freed } => {
let mut metadata = std::collections::HashMap::new();
metadata.insert("bytes_freed".to_string(), bytes_freed.to_string());
return Event {
event_type: "cache_eviction".to_string(),
timestamp_ms: chrono::Utc::now().timestamp_millis(),
origin_id: None,
path: None,
file_id: None,
metadata,
};
}
CoreEvent::OriginConnected { origin_id } => (
"origin_connected".to_string(),
Some(origin_id.to_string()),
None,
None,
),
CoreEvent::OriginDisconnected { origin_id } => (
"origin_disconnected".to_string(),
Some(origin_id.to_string()),
None,
None,
),
CoreEvent::AllOriginsUnhealthy { candidate_count } => {
let mut metadata = std::collections::HashMap::new();
metadata.insert("candidate_count".to_string(), candidate_count.to_string());
return Event {
event_type: "all_origins_unhealthy".to_string(),
timestamp_ms: chrono::Utc::now().timestamp_millis(),
origin_id: None,
path: None,
file_id: None,
metadata,
};
}
};
Event {
event_type,
timestamp_ms: chrono::Utc::now().timestamp_millis(),
origin_id,
path,
file_id,
metadata: Default::default(),
}
}
fn matches_filter(event: &CoreEvent, filter: &EventFilter) -> bool {
if !filter.event_types.is_empty() {
let event_type = match event {
CoreEvent::FileAccessed { .. } => "file_accessed",
CoreEvent::FileAdded { .. } => "file_added",
CoreEvent::FileRemoved { .. } => "file_removed",
CoreEvent::FileModified { .. } => "file_modified",
CoreEvent::SyncStarted { .. } => "sync_started",
CoreEvent::SyncCompleted { .. } => "sync_completed",
CoreEvent::OriginHealthChanged { .. } => "origin_health_changed",
CoreEvent::CacheEviction { .. } => "cache_eviction",
CoreEvent::OriginConnected { .. } => "origin_connected",
CoreEvent::OriginDisconnected { .. } => "origin_disconnected",
CoreEvent::AllOriginsUnhealthy { .. } => "all_origins_unhealthy",
};
if !filter.event_types.iter().any(|t| t == event_type) {
return false;
}
}
if let Some(ref origin_filter) = filter.origin_id {
let event_origin = match event {
CoreEvent::FileAccessed { origin_id, .. }
| CoreEvent::FileAdded { origin_id, .. }
| CoreEvent::SyncStarted { origin_id }
| CoreEvent::SyncCompleted { origin_id, .. }
| CoreEvent::OriginHealthChanged { origin_id, .. }
| CoreEvent::OriginConnected { origin_id }
| CoreEvent::OriginDisconnected { origin_id } => Some(origin_id.to_string()),
CoreEvent::FileRemoved { .. }
| CoreEvent::FileModified { .. }
| CoreEvent::CacheEviction { .. }
| CoreEvent::AllOriginsUnhealthy { .. } => None,
};
if event_origin.as_ref() != Some(origin_filter) {
return false;
}
}
true
}
}
#[tonic::async_trait]
impl MusicFs for MusicFsServer {
async fn search(
&self,
_request: Request<SearchRequest>,
) -> Result<Response<SearchResponse>, Status> {
Err(Status::unimplemented(
"Use SearchService for search operations",
))
}
type SearchStreamStream = ReceiverStream<Result<SearchResult, Status>>;
async fn search_stream(
&self,
_request: Request<SearchRequest>,
) -> Result<Response<Self::SearchStreamStream>, Status> {
Err(Status::unimplemented(
"Use SearchService for search operations",
))
}
async fn get_status(
&self,
_request: Request<Empty>,
) -> Result<Response<StatusResponse>, Status> {
let uptime = self.start_time.elapsed().as_secs();
Ok(Response::new(StatusResponse {
version: self.version.clone(),
uptime_secs: uptime,
mount_point: String::new(),
state: MountState::MountReady as i32,
open_file_handles: 0,
fuse_ops_total: 0,
files_indexed: 0,
cache_size_bytes: 0,
origins: vec![],
}))
}
async fn shutdown(
&self,
request: Request<ShutdownRequest>,
) -> Result<Response<Empty>, Status> {
let req = request.into_inner();
info!(
"Shutdown requested (graceful={}, timeout={}s)",
req.graceful, req.timeout_secs
);
Ok(Response::new(Empty {}))
}
async fn get_cache_stats(
&self,
_request: Request<Empty>,
) -> Result<Response<CacheStats>, Status> {
Ok(Response::new(CacheStats {
total_size_bytes: 0,
used_size_bytes: 0,
size_limit_bytes: 0,
chunk_count: 0,
chunks_unique: 0,
dedup_ratio: 0.0,
hit_count: 0,
miss_count: 0,
hit_ratio: 0.0,
metadata_entries: 0,
metadata_bytes: 0,
l1_metadata: Some(TierStats {
entries: 0,
size_bytes: 0,
hits: 0,
misses: 0,
}),
l2_headers: Some(TierStats {
entries: 0,
size_bytes: 0,
hits: 0,
misses: 0,
}),
l3_chunks: Some(TierStats {
entries: 0,
size_bytes: 0,
hits: 0,
misses: 0,
}),
}))
}
async fn clear_cache(
&self,
request: Request<ClearCacheRequest>,
) -> Result<Response<ClearCacheResponse>, Status> {
let req = request.into_inner();
debug!(
"Clear cache requested: origin={:?}, metadata={}, chunks={}",
req.origin_id, req.clear_metadata, req.clear_chunks
);
Ok(Response::new(ClearCacheResponse {
bytes_cleared: 0,
chunks_cleared: 0,
}))
}
type PrefetchStream = ReceiverStream<Result<PrefetchProgress, Status>>;
async fn prefetch(
&self,
request: Request<PrefetchRequest>,
) -> Result<Response<Self::PrefetchStream>, Status> {
let req = request.into_inner();
let total = req.paths.len() as u32;
let (tx, rx) = mpsc::channel(32);
tokio::spawn(async move {
for (i, path) in req.paths.into_iter().enumerate() {
let progress = PrefetchProgress {
current_path: path,
completed: i as u32 + 1,
total,
bytes_fetched: 0,
};
if tx.send(Ok(progress)).await.is_err() {
break;
}
}
});
Ok(Response::new(ReceiverStream::new(rx)))
}
async fn list_origins(
&self,
_request: Request<Empty>,
) -> Result<Response<OriginsResponse>, Status> {
Ok(Response::new(OriginsResponse { origins: vec![] }))
}
async fn get_origin_health(
&self,
request: Request<OriginRequest>,
) -> Result<Response<OriginHealthResponse>, Status> {
let req = request.into_inner();
Ok(Response::new(OriginHealthResponse {
origin_id: req.origin_id,
status: HealthStatus::HealthUnknown as i32,
message: None,
last_check_secs: 0,
}))
}
type RescanOriginStream = ReceiverStream<Result<SyncProgress, Status>>;
async fn rescan_origin(
&self,
request: Request<OriginRequest>,
) -> Result<Response<Self::RescanOriginStream>, Status> {
let req = request.into_inner();
info!("Rescan requested for origin: {}", req.origin_id);
let (tx, rx) = mpsc::channel(32);
tokio::spawn(async move {
let phases = ["scanning", "indexing", "complete"];
for (i, phase) in phases.iter().enumerate() {
let progress = SyncProgress {
phase: phase.to_string(),
current: i as u32 + 1,
total: phases.len() as u32,
current_path: String::new(),
bytes_synced: 0,
};
if tx.send(Ok(progress)).await.is_err() {
break;
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
});
Ok(Response::new(ReceiverStream::new(rx)))
}
type SubscribeEventsStream = ReceiverStream<Result<Event, Status>>;
async fn subscribe_events(
&self,
request: Request<EventFilter>,
) -> Result<Response<Self::SubscribeEventsStream>, Status> {
let filter = request.into_inner();
let mut rx = self.event_bus.subscribe();
let (tx, out_rx) = mpsc::channel(100);
tokio::spawn(async move {
while let Ok(event) = rx.recv().await {
if Self::matches_filter(&event, &filter) {
let proto_event = Self::event_to_proto(&event);
if tx.send(Ok(proto_event)).await.is_err() {
break;
}
}
}
});
Ok(Response::new(ReceiverStream::new(out_rx)))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn test_get_status() {
let event_bus = Arc::new(EventBus::new(16));
let server = MusicFsServer::new(event_bus);
let response = server.get_status(Request::new(Empty {})).await.unwrap();
let status = response.into_inner();
assert!(!status.version.is_empty());
assert!(status.uptime_secs < 5);
}
#[tokio::test]
async fn test_get_cache_stats() {
let event_bus = Arc::new(EventBus::new(16));
let server = MusicFsServer::new(event_bus);
let response = server
.get_cache_stats(Request::new(Empty {}))
.await
.unwrap();
let stats = response.into_inner();
assert_eq!(stats.hit_ratio, 0.0);
}
}
+288
View File
@@ -0,0 +1,288 @@
use musicfs_core::Event;
use serde::{Deserialize, Serialize};
use std::time::Duration;
use tokio::sync::broadcast;
use tracing::{debug, warn};
#[derive(Debug, Clone, Serialize)]
pub struct WebhookPayload {
pub event_type: String,
pub timestamp: i64,
pub data: serde_json::Value,
}
#[derive(Debug, Clone, Deserialize)]
pub struct WebhookConfig {
pub url: String,
pub secret: Option<String>,
pub events: Vec<String>,
#[serde(default = "default_retry_count")]
pub retry_count: u32,
#[serde(default = "default_timeout_ms")]
pub timeout_ms: u64,
}
fn default_retry_count() -> u32 {
3
}
fn default_timeout_ms() -> u64 {
5000
}
pub struct WebhookHandler {
client: reqwest::Client,
configs: Vec<WebhookConfig>,
}
impl WebhookHandler {
pub fn new(configs: Vec<WebhookConfig>) -> Self {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.build()
.expect("Failed to create HTTP client");
Self { client, configs }
}
pub async fn run(&self, mut rx: broadcast::Receiver<Event>) {
while let Ok(event) = rx.recv().await {
for config in &self.configs {
if self.matches_filter(&event, config) {
self.dispatch(config, &event).await;
}
}
}
}
async fn dispatch(&self, config: &WebhookConfig, event: &Event) {
let payload = WebhookPayload {
event_type: self.event_type_name(event),
timestamp: chrono::Utc::now().timestamp_millis(),
data: self.event_to_json(event),
};
let signature = self.sign(&payload, config);
let mut attempts = 0u32;
loop {
let result = self
.client
.post(&config.url)
.timeout(Duration::from_millis(config.timeout_ms))
.header("Content-Type", "application/json")
.header("X-MusicFS-Signature", &signature)
.header("X-MusicFS-Event", &payload.event_type)
.json(&payload)
.send()
.await;
match result {
Ok(resp) if resp.status().is_success() => {
debug!(
"Webhook delivered to {} for {}",
config.url, payload.event_type
);
break;
}
Ok(resp) => {
warn!(
"Webhook to {} returned status {}, attempt {}/{}",
config.url,
resp.status(),
attempts + 1,
config.retry_count + 1
);
}
Err(e) => {
warn!(
"Webhook to {} failed: {}, attempt {}/{}",
config.url,
e,
attempts + 1,
config.retry_count + 1
);
}
}
if attempts >= config.retry_count {
warn!(
"Webhook delivery to {} failed after {} attempts",
config.url,
attempts + 1
);
break;
}
attempts += 1;
let delay = Duration::from_millis(100 * 2u64.pow(attempts));
tokio::time::sleep(delay).await;
}
}
fn sign(&self, payload: &WebhookPayload, config: &WebhookConfig) -> String {
match &config.secret {
Some(secret) => {
use hmac::{Hmac, Mac};
use sha2::Sha256;
type HmacSha256 = Hmac<Sha256>;
let body = serde_json::to_string(payload).unwrap_or_default();
let mut mac =
HmacSha256::new_from_slice(secret.as_bytes()).expect("HMAC key invalid");
mac.update(body.as_bytes());
let result = mac.finalize();
format!("sha256={}", hex::encode(result.into_bytes()))
}
None => String::new(),
}
}
fn matches_filter(&self, event: &Event, config: &WebhookConfig) -> bool {
if config.events.is_empty() {
return true;
}
let event_type = self.event_type_name(event);
config.events.iter().any(|e| e == &event_type)
}
fn event_type_name(&self, event: &Event) -> String {
match event {
Event::FileAccessed { .. } => "file_accessed",
Event::FileAdded { .. } => "file_added",
Event::FileRemoved { .. } => "file_removed",
Event::FileModified { .. } => "file_modified",
Event::SyncStarted { .. } => "sync_started",
Event::SyncCompleted { .. } => "sync_completed",
Event::OriginHealthChanged { .. } => "origin_health_changed",
Event::CacheEviction { .. } => "cache_eviction",
Event::OriginConnected { .. } => "origin_connected",
Event::OriginDisconnected { .. } => "origin_disconnected",
Event::AllOriginsUnhealthy { .. } => "all_origins_unhealthy",
}
.to_string()
}
fn event_to_json(&self, event: &Event) -> serde_json::Value {
match event {
Event::FileAccessed {
file_id,
origin_id,
path,
offset,
size,
} => serde_json::json!({
"file_id": file_id.0,
"origin_id": origin_id.to_string(),
"path": path.as_str(),
"offset": offset,
"size": size,
}),
Event::FileAdded { path, origin_id } => serde_json::json!({
"path": path.as_str(),
"origin_id": origin_id.to_string(),
}),
Event::FileRemoved { path, file_id } => serde_json::json!({
"path": path.as_str(),
"file_id": file_id.map(|id| id.0),
}),
Event::FileModified { path } => serde_json::json!({
"path": path.as_str(),
}),
Event::SyncStarted { origin_id } => serde_json::json!({
"origin_id": origin_id.to_string(),
}),
Event::SyncCompleted {
origin_id,
files_changed,
} => serde_json::json!({
"origin_id": origin_id.to_string(),
"files_changed": files_changed,
}),
Event::OriginHealthChanged { origin_id, healthy } => serde_json::json!({
"origin_id": origin_id.to_string(),
"healthy": healthy,
}),
Event::CacheEviction { bytes_freed } => serde_json::json!({
"bytes_freed": bytes_freed,
}),
Event::OriginConnected { origin_id } => serde_json::json!({
"origin_id": origin_id.to_string(),
}),
Event::OriginDisconnected { origin_id } => serde_json::json!({
"origin_id": origin_id.to_string(),
}),
Event::AllOriginsUnhealthy { candidate_count } => serde_json::json!({
"candidate_count": candidate_count,
}),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use musicfs_core::OriginId;
#[test]
fn test_webhook_config_defaults() {
let json = r#"{"url": "http://example.com", "events": []}"#;
let config: WebhookConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.retry_count, 3);
assert_eq!(config.timeout_ms, 5000);
}
#[test]
fn test_event_type_name() {
let handler = WebhookHandler::new(vec![]);
let event = Event::SyncStarted {
origin_id: OriginId::from("test"),
};
assert_eq!(handler.event_type_name(&event), "sync_started");
}
#[test]
fn test_matches_filter_empty() {
let handler = WebhookHandler::new(vec![]);
let config = WebhookConfig {
url: "http://example.com".to_string(),
secret: None,
events: vec![],
retry_count: 3,
timeout_ms: 5000,
};
let event = Event::SyncStarted {
origin_id: OriginId::from("test"),
};
assert!(handler.matches_filter(&event, &config));
}
#[test]
fn test_matches_filter_specific() {
let handler = WebhookHandler::new(vec![]);
let config = WebhookConfig {
url: "http://example.com".to_string(),
secret: None,
events: vec!["sync_started".to_string()],
retry_count: 3,
timeout_ms: 5000,
};
let event = Event::SyncStarted {
origin_id: OriginId::from("test"),
};
assert!(handler.matches_filter(&event, &config));
let event2 = Event::SyncCompleted {
origin_id: OriginId::from("test"),
files_changed: 0,
};
assert!(!handler.matches_filter(&event2, &config));
}
}
+17
View File
@@ -4,3 +4,20 @@ version.workspace = true
edition.workspace = true edition.workspace = true
[dependencies] [dependencies]
musicfs-core = { path = "../musicfs-core" }
async-trait.workspace = true
tokio.workspace = true
thiserror.workspace = true
serde.workspace = true
serde_json.workspace = true
tracing.workspace = true
libloading = "0.8"
wasmtime = { version = "19", optional = true }
semver = "1"
[features]
default = []
wasm = ["wasmtime"]
[dev-dependencies]
tempfile.workspace = true
@@ -0,0 +1,42 @@
use thiserror::Error;
#[derive(Debug, Error)]
pub enum PluginError {
#[error("Plugin not found: {0}")]
NotFound(String),
#[error("Plugin load failed: {0}")]
LoadFailed(String),
#[error("Plugin initialization failed: {0}")]
InitFailed(String),
#[error("Plugin API version mismatch: expected {expected}, got {actual}")]
VersionMismatch { expected: String, actual: String },
#[error("Plugin already loaded: {0}")]
AlreadyLoaded(String),
#[error("Plugin symbol not found: {0}")]
SymbolNotFound(String),
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Plugin execution error: {0}")]
Execution(String),
#[error("Plugin shutdown error: {0}")]
Shutdown(String),
#[error("Configuration error: {0}")]
Config(String),
#[error("WASM error: {0}")]
Wasm(String),
#[error("Resource limit exceeded: {0}")]
ResourceLimit(String),
}
pub type Result<T> = std::result::Result<T, PluginError>;
+15 -1
View File
@@ -1 +1,15 @@
#![allow(dead_code)] pub mod error;
pub mod manager;
pub mod native;
pub mod traits;
pub mod wasm;
pub use error::{PluginError, Result};
pub use manager::{PluginConfig, PluginEntry, PluginManager, WasmConfig};
pub use native::NativePluginHost;
pub use traits::{
ExternalMetadata, FormatPlugin, MetadataPlugin, MetadataQuery, MetadataQueryType,
OriginDirEntry, OriginHealth, OriginInstance, OriginPlugin, OriginStat, Plugin, PluginId,
PluginInfo, PluginType, WatchEvent, WatchHandle, PLUGIN_API_VERSION,
};
pub use wasm::{ResourceLimits, WasmPluginHost};
@@ -0,0 +1,345 @@
use crate::error::{PluginError, Result};
use crate::native::NativePluginHost;
use crate::traits::{Plugin, PluginId, PluginInfo, PluginType};
use crate::wasm::{ResourceLimits, WasmPluginHost};
use serde::Deserialize;
use serde_json::Value;
use std::collections::HashMap;
use std::path::PathBuf;
use tracing::{debug, info};
#[derive(Debug, Clone, Deserialize, Default)]
pub struct PluginConfig {
#[serde(default)]
pub enabled: bool,
#[serde(default)]
pub search_paths: Vec<PathBuf>,
#[serde(default)]
pub plugins: HashMap<String, PluginEntry>,
#[serde(default)]
pub wasm: WasmConfig,
}
#[derive(Debug, Clone, Deserialize)]
pub struct PluginEntry {
pub path: PathBuf,
#[serde(default)]
pub enabled: bool,
#[serde(default)]
pub config: Value,
}
impl Default for PluginEntry {
fn default() -> Self {
Self {
path: PathBuf::new(),
enabled: true,
config: Value::Null,
}
}
}
#[derive(Debug, Clone, Deserialize, Default)]
pub struct WasmConfig {
#[serde(default)]
pub enabled: bool,
#[serde(default)]
pub max_memory_mb: Option<u32>,
#[serde(default)]
pub max_cpu_time_ms: Option<u32>,
}
pub struct PluginManager {
native_host: NativePluginHost,
wasm_host: WasmPluginHost,
registry: PluginRegistry,
config: PluginConfig,
}
struct PluginRegistry {
origin_plugins: Vec<PluginId>,
metadata_plugins: Vec<PluginId>,
format_plugins: Vec<PluginId>,
}
impl PluginRegistry {
fn new() -> Self {
Self {
origin_plugins: Vec::new(),
metadata_plugins: Vec::new(),
format_plugins: Vec::new(),
}
}
fn register(&mut self, id: PluginId, plugin_type: PluginType) {
match plugin_type {
PluginType::Origin => {
if !self.origin_plugins.contains(&id) {
self.origin_plugins.push(id);
}
}
PluginType::Metadata => {
if !self.metadata_plugins.contains(&id) {
self.metadata_plugins.push(id);
}
}
PluginType::Format => {
if !self.format_plugins.contains(&id) {
self.format_plugins.push(id);
}
}
}
}
fn unregister(&mut self, id: PluginId) {
self.origin_plugins.retain(|&x| x != id);
self.metadata_plugins.retain(|&x| x != id);
self.format_plugins.retain(|&x| x != id);
}
}
impl PluginManager {
pub fn new() -> Result<Self> {
Ok(Self {
native_host: NativePluginHost::new(),
wasm_host: WasmPluginHost::new()?,
registry: PluginRegistry::new(),
config: PluginConfig::default(),
})
}
pub fn init(config: &PluginConfig) -> Result<Self> {
let mut manager = Self::new()?;
manager.config = config.clone();
if !config.enabled {
info!("Plugin system disabled");
return Ok(manager);
}
info!("Initializing plugin system");
for path in &config.search_paths {
manager.native_host.add_search_path(path.clone());
}
if config.wasm.enabled {
let limits = ResourceLimits {
max_memory_mb: config.wasm.max_memory_mb.unwrap_or(64),
max_cpu_time_ms: config.wasm.max_cpu_time_ms.unwrap_or(5000),
..Default::default()
};
manager.wasm_host.set_limits(limits);
}
for (name, entry) in &config.plugins {
if !entry.enabled {
debug!("Skipping disabled plugin: {}", name);
continue;
}
match manager.load_and_init(&entry.path, &entry.config) {
Ok(id) => {
info!("Loaded plugin '{}' with id {:?}", name, id);
}
Err(e) => {
tracing::warn!("Failed to load plugin '{}': {}", name, e);
}
}
}
let discovered = manager.native_host.discover()?;
for id in discovered {
if let Some(info) = manager.native_host.list().iter().find(|i| i.id == id) {
manager.registry.register(id, info.plugin_type);
}
}
Ok(manager)
}
pub fn load_and_init(&mut self, path: &PathBuf, config: &Value) -> Result<PluginId> {
let id = self.native_host.load(path)?;
if let Some(plugin) = self.native_host.get_mut(id) {
plugin.init(config.clone())?;
}
if let Some(info) = self.native_host.list().iter().find(|i| i.id == id) {
self.registry.register(id, info.plugin_type);
}
Ok(id)
}
pub fn load_wasm(&mut self, wasm_bytes: &[u8]) -> Result<PluginId> {
if !self.config.wasm.enabled {
return Err(PluginError::Config("WASM plugins disabled".to_string()));
}
self.wasm_host.load(wasm_bytes)
}
pub fn unload(&mut self, id: PluginId) -> Result<()> {
self.registry.unregister(id);
if let Err(native_err) = self.native_host.unload(id) {
if let Err(wasm_err) = self.wasm_host.unload(id) {
return Err(PluginError::NotFound(format!(
"Plugin {:?} not found in native ({}) or WASM ({}) hosts",
id, native_err, wasm_err
)));
}
}
Ok(())
}
pub fn reload(&mut self, id: PluginId) -> Result<()> {
self.native_host.reload(id)
}
pub fn reload_all(&mut self) -> Result<()> {
let ids: Vec<PluginId> = self.native_host.list().iter().map(|i| i.id).collect();
for id in ids {
self.reload(id)?;
}
Ok(())
}
pub fn list(&self) -> Vec<PluginInfo> {
let mut all = self.native_host.list();
for (id, name) in self.wasm_host.list() {
all.push(PluginInfo {
id,
name: name.to_string(),
version: semver::Version::new(0, 0, 0),
description: "WASM plugin".to_string(),
plugin_type: PluginType::Origin,
});
}
all
}
pub fn get(&self, id: PluginId) -> Option<&dyn Plugin> {
self.native_host.get(id)
}
pub fn get_mut(&mut self, id: PluginId) -> Option<&mut dyn Plugin> {
self.native_host.get_mut(id)
}
pub fn origin_plugin_ids(&self) -> &[PluginId] {
&self.registry.origin_plugins
}
pub fn metadata_plugin_ids(&self) -> &[PluginId] {
&self.registry.metadata_plugins
}
pub fn format_plugin_ids(&self) -> &[PluginId] {
&self.registry.format_plugins
}
pub fn shutdown(&mut self) -> Result<()> {
info!("Shutting down plugin system");
let ids: Vec<PluginId> = self.list().iter().map(|i| i.id).collect();
for id in ids {
if let Err(e) = self.unload(id) {
tracing::warn!("Failed to unload plugin {:?}: {}", id, e);
}
}
Ok(())
}
}
impl Default for PluginManager {
fn default() -> Self {
Self::new().expect("Failed to create plugin manager")
}
}
impl Drop for PluginManager {
fn drop(&mut self) {
let _ = self.shutdown();
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_plugin_manager_new() {
let manager = PluginManager::new();
assert!(manager.is_ok());
}
#[test]
fn test_plugin_manager_disabled() {
let config = PluginConfig {
enabled: false,
..Default::default()
};
let manager = PluginManager::init(&config);
assert!(manager.is_ok());
let manager = manager.unwrap();
assert!(manager.list().is_empty());
}
#[test]
fn test_registry() {
let mut registry = PluginRegistry::new();
let id1 = PluginId::new(1);
let id2 = PluginId::new(2);
registry.register(id1, PluginType::Origin);
registry.register(id2, PluginType::Metadata);
assert_eq!(registry.origin_plugins.len(), 1);
assert_eq!(registry.metadata_plugins.len(), 1);
registry.unregister(id1);
assert!(registry.origin_plugins.is_empty());
}
#[test]
fn test_plugin_config_deserialize() {
let json = r#"{
"enabled": true,
"search_paths": ["/usr/lib/musicfs/plugins"],
"plugins": {
"example": {
"path": "/path/to/plugin.so",
"enabled": true,
"config": {"key": "value"}
}
},
"wasm": {
"enabled": false
}
}"#;
let config: PluginConfig = serde_json::from_str(json).unwrap();
assert!(config.enabled);
assert_eq!(config.search_paths.len(), 1);
assert!(config.plugins.contains_key("example"));
}
}
@@ -0,0 +1,300 @@
use crate::error::{PluginError, Result};
use crate::traits::{Plugin, PluginId, PluginInfo, PluginType, PLUGIN_API_VERSION};
use libloading::{Library, Symbol};
use semver::Version;
use std::collections::HashMap;
use std::ffi::CStr;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use tracing::{debug, info, warn};
static NEXT_PLUGIN_ID: AtomicU64 = AtomicU64::new(1);
fn next_plugin_id() -> PluginId {
PluginId::new(NEXT_PLUGIN_ID.fetch_add(1, Ordering::SeqCst))
}
struct LoadedPlugin {
id: PluginId,
path: PathBuf,
library: Library,
instance: Box<dyn Plugin>,
plugin_type: PluginType,
}
pub struct NativePluginHost {
plugins: HashMap<PluginId, LoadedPlugin>,
search_paths: Vec<PathBuf>,
}
impl NativePluginHost {
pub fn new() -> Self {
Self {
plugins: HashMap::new(),
search_paths: Vec::new(),
}
}
pub fn add_search_path(&mut self, path: PathBuf) {
if !self.search_paths.contains(&path) {
self.search_paths.push(path);
}
}
pub fn load(&mut self, path: &Path) -> Result<PluginId> {
let canonical = path.canonicalize().map_err(|e| {
PluginError::LoadFailed(format!("Cannot resolve path {}: {}", path.display(), e))
})?;
for plugin in self.plugins.values() {
if plugin.path == canonical {
return Err(PluginError::AlreadyLoaded(canonical.display().to_string()));
}
}
info!("Loading native plugin from {:?}", canonical);
let library = unsafe {
Library::new(&canonical).map_err(|e| {
PluginError::LoadFailed(format!("Failed to load library: {}", e))
})?
};
self.verify_api_version(&library)?;
let instance = self.create_plugin_instance(&library)?;
let id = next_plugin_id();
let plugin_type = self.detect_plugin_type(&*instance);
debug!(
"Loaded plugin '{}' v{} as {:?}",
instance.name(),
instance.version(),
plugin_type
);
self.plugins.insert(
id,
LoadedPlugin {
id,
path: canonical,
library,
instance,
plugin_type,
},
);
Ok(id)
}
pub fn unload(&mut self, id: PluginId) -> Result<()> {
let mut plugin = self
.plugins
.remove(&id)
.ok_or_else(|| PluginError::NotFound(format!("Plugin {:?}", id)))?;
info!("Unloading plugin '{}'", plugin.instance.name());
plugin.instance.shutdown()?;
drop(plugin.instance);
drop(plugin.library);
Ok(())
}
pub fn reload(&mut self, id: PluginId) -> Result<()> {
let plugin = self
.plugins
.get(&id)
.ok_or_else(|| PluginError::NotFound(format!("Plugin {:?}", id)))?;
let path = plugin.path.clone();
info!("Hot-reloading plugin from {:?}", path);
self.unload(id)?;
let new_id = self.load(&path)?;
if let Some(plugin) = self.plugins.remove(&new_id) {
self.plugins.insert(id, LoadedPlugin { id, ..plugin });
}
Ok(())
}
pub fn get(&self, id: PluginId) -> Option<&dyn Plugin> {
self.plugins.get(&id).map(|p| &*p.instance as &dyn Plugin)
}
pub fn get_mut(&mut self, id: PluginId) -> Option<&mut dyn Plugin> {
self.plugins
.get_mut(&id)
.map(|p| &mut *p.instance as &mut dyn Plugin)
}
pub fn list(&self) -> Vec<PluginInfo> {
self.plugins
.values()
.map(|p| PluginInfo {
id: p.id,
name: p.instance.name().to_string(),
version: p.instance.version(),
description: p.instance.description().to_string(),
plugin_type: p.plugin_type,
})
.collect()
}
pub fn find_by_name(&self, name: &str) -> Option<PluginId> {
self.plugins
.iter()
.find(|(_, p)| p.instance.name() == name)
.map(|(id, _)| *id)
}
pub fn discover(&mut self) -> Result<Vec<PluginId>> {
let mut loaded = Vec::new();
for search_path in self.search_paths.clone() {
if !search_path.exists() {
continue;
}
let entries = std::fs::read_dir(&search_path).map_err(|e| {
PluginError::LoadFailed(format!(
"Cannot read plugin directory {}: {}",
search_path.display(),
e
))
})?;
for entry in entries.flatten() {
let path = entry.path();
if self.is_plugin_library(&path) {
match self.load(&path) {
Ok(id) => loaded.push(id),
Err(e) => {
warn!("Failed to load plugin {:?}: {}", path, e);
}
}
}
}
}
Ok(loaded)
}
fn verify_api_version(&self, library: &Library) -> Result<()> {
let version_fn: Symbol<unsafe extern "C" fn() -> *const std::ffi::c_char> = unsafe {
library
.get(b"musicfs_plugin_api_version")
.map_err(|_| PluginError::SymbolNotFound("musicfs_plugin_api_version".to_string()))?
};
let version_ptr = unsafe { version_fn() };
let version_str = unsafe { CStr::from_ptr(version_ptr) }
.to_str()
.map_err(|_| PluginError::VersionMismatch {
expected: PLUGIN_API_VERSION.to_string(),
actual: "<invalid UTF-8>".to_string(),
})?;
let plugin_version = Version::parse(version_str).map_err(|_| PluginError::VersionMismatch {
expected: PLUGIN_API_VERSION.to_string(),
actual: version_str.to_string(),
})?;
let expected_version = Version::parse(PLUGIN_API_VERSION).unwrap();
if plugin_version.major != expected_version.major {
return Err(PluginError::VersionMismatch {
expected: PLUGIN_API_VERSION.to_string(),
actual: version_str.to_string(),
});
}
Ok(())
}
fn create_plugin_instance(&self, library: &Library) -> Result<Box<dyn Plugin>> {
let create_fn: Symbol<unsafe extern "C" fn() -> *mut dyn Plugin> = unsafe {
library
.get(b"musicfs_plugin_create")
.map_err(|_| PluginError::SymbolNotFound("musicfs_plugin_create".to_string()))?
};
let plugin_ptr = unsafe { create_fn() };
if plugin_ptr.is_null() {
return Err(PluginError::LoadFailed(
"Plugin factory returned null".to_string(),
));
}
let plugin = unsafe { Box::from_raw(plugin_ptr) };
Ok(plugin)
}
fn detect_plugin_type(&self, plugin: &dyn Plugin) -> PluginType {
plugin.plugin_type()
}
fn is_plugin_library(&self, path: &Path) -> bool {
let extension = path.extension().and_then(|e| e.to_str());
match extension {
Some("so") => true,
Some("dylib") => true,
Some("dll") => true,
_ => false,
}
}
}
impl Default for NativePluginHost {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_native_host_creation() {
let host = NativePluginHost::new();
assert!(host.plugins.is_empty());
assert!(host.search_paths.is_empty());
}
#[test]
fn test_add_search_path() {
let mut host = NativePluginHost::new();
host.add_search_path(PathBuf::from("/usr/lib/musicfs/plugins"));
host.add_search_path(PathBuf::from("/usr/lib/musicfs/plugins"));
assert_eq!(host.search_paths.len(), 1);
}
#[test]
fn test_is_plugin_library() {
let host = NativePluginHost::new();
assert!(host.is_plugin_library(Path::new("plugin.so")));
assert!(host.is_plugin_library(Path::new("plugin.dylib")));
assert!(host.is_plugin_library(Path::new("plugin.dll")));
assert!(!host.is_plugin_library(Path::new("plugin.txt")));
assert!(!host.is_plugin_library(Path::new("plugin")));
}
#[test]
fn test_load_nonexistent() {
let mut host = NativePluginHost::new();
let result = host.load(Path::new("/nonexistent/plugin.so"));
assert!(result.is_err());
}
}
@@ -0,0 +1,343 @@
//! Plugin trait definitions (FR-23.1-23.4)
//!
//! Per architecture.md section 4.3.4:
//! - Plugin trait: Base interface for all plugins
//! - OriginPlugin: Creates Origin instances for storage backends
//! - MetadataPlugin: Provides external metadata lookup
//! - FormatPlugin: Handles custom audio format parsing
use crate::error::Result;
use async_trait::async_trait;
use musicfs_core::AudioMeta;
use semver::Version;
use serde_json::Value;
use std::io::Read;
/// Current plugin API version
pub const PLUGIN_API_VERSION: &str = "0.1.0";
/// Unique identifier for a loaded plugin
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PluginId(pub u64);
impl PluginId {
pub fn new(id: u64) -> Self {
Self(id)
}
}
/// Plugin metadata returned by plugins
#[derive(Debug, Clone)]
pub struct PluginInfo {
pub id: PluginId,
pub name: String,
pub version: Version,
pub description: String,
pub plugin_type: PluginType,
}
/// Type of plugin
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PluginType {
Origin,
Metadata,
Format,
}
/// Base plugin interface (FR-23.1)
///
/// All plugins must implement this trait. It provides:
/// - Plugin identification (name, version)
/// - Lifecycle management (init, shutdown)
pub trait Plugin: Send + Sync {
/// Unique plugin name (e.g., "s3-origin", "musicbrainz-metadata")
fn name(&self) -> &str;
/// Plugin version following semver
fn version(&self) -> Version;
/// Human-readable description
fn description(&self) -> &str {
""
}
/// Plugin type for registry categorization
fn plugin_type(&self) -> PluginType;
/// Initialize plugin with configuration
///
/// Called once after loading. The config value contains
/// plugin-specific configuration from the main config file.
fn init(&mut self, config: Value) -> Result<()>;
/// Shutdown plugin and release resources
///
/// Called before unloading. Plugins should clean up any
/// resources (connections, file handles, etc).
fn shutdown(&mut self) -> Result<()>;
}
/// Origin plugin interface (FR-23.3)
///
/// Per architecture.md section 4.3.4:
/// Origin plugins create `Box<dyn Origin>` instances for custom storage backends.
///
/// Example use cases:
/// - Google Drive origin
/// - Dropbox origin
/// - Custom NAS protocol
#[async_trait]
pub trait OriginPlugin: Plugin {
/// Origin type identifier (e.g., "gdrive", "dropbox")
fn origin_type(&self) -> &str;
/// Create a new Origin instance with the given configuration
///
/// The config contains origin-specific settings (credentials, paths, etc).
/// Returns a boxed Origin that can be used by the OriginRouter.
async fn create_origin(
&self,
id: &str,
config: Value,
) -> Result<Box<dyn OriginInstance>>;
}
/// Instance created by OriginPlugin
///
/// This is a simplified async interface that maps to the full Origin trait.
/// The plugin host wraps this to provide the full Origin implementation.
#[async_trait]
pub trait OriginInstance: Send + Sync {
/// List directory contents
async fn readdir(&self, path: &str) -> Result<Vec<OriginDirEntry>>;
/// Get file/directory stats
async fn stat(&self, path: &str) -> Result<OriginStat>;
/// Read file data
async fn read(&self, path: &str, offset: u64, size: u32) -> Result<Vec<u8>>;
/// Check if path exists
async fn exists(&self, path: &str) -> Result<bool>;
/// Health check
async fn health(&self) -> OriginHealth;
/// Watch path for changes (FR-10.2)
async fn watch(
&self,
path: &str,
callback: Box<dyn Fn(WatchEvent) + Send + Sync>,
) -> Result<WatchHandle>;
}
pub struct WatchHandle {
_cancel: tokio::sync::oneshot::Sender<()>,
}
impl WatchHandle {
pub fn new(cancel: tokio::sync::oneshot::Sender<()>) -> Self {
Self { _cancel: cancel }
}
}
#[derive(Debug, Clone)]
pub enum WatchEvent {
Created(String),
Modified(String),
Deleted(String),
}
/// Directory entry from plugin origin
#[derive(Debug, Clone)]
pub struct OriginDirEntry {
pub name: String,
pub is_dir: bool,
pub size: u64,
pub mtime_secs: u64,
}
/// File stats from plugin origin
#[derive(Debug, Clone)]
pub struct OriginStat {
pub size: u64,
pub mtime_secs: u64,
pub is_dir: bool,
}
/// Origin health status
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum OriginHealth {
Healthy,
Degraded,
Unhealthy,
}
/// Metadata plugin interface (FR-23.3)
///
/// Metadata plugins provide external metadata lookup from services like
/// MusicBrainz, Discogs, Last.fm, etc.
#[async_trait]
pub trait MetadataPlugin: Plugin {
/// Lookup metadata for a query
///
/// Returns enriched metadata if found, None otherwise.
async fn lookup(&self, query: &MetadataQuery) -> Result<Option<ExternalMetadata>>;
/// Supported query types
fn supported_queries(&self) -> &[MetadataQueryType] {
&[MetadataQueryType::ByTitleArtist]
}
}
/// Query for metadata lookup
#[derive(Debug, Clone)]
pub struct MetadataQuery {
pub query_type: MetadataQueryType,
pub title: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub fingerprint: Option<String>,
pub duration_ms: Option<u64>,
}
/// Type of metadata query
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MetadataQueryType {
ByTitleArtist,
ByFingerprint,
ByAlbum,
}
/// External metadata returned by plugins
#[derive(Debug, Clone, Default)]
pub struct ExternalMetadata {
pub title: Option<String>,
pub artist: Option<String>,
pub album: Option<String>,
pub album_artist: Option<String>,
pub genre: Option<String>,
pub year: Option<u32>,
pub track: Option<u32>,
pub disc: Option<u32>,
pub musicbrainz_id: Option<String>,
pub artwork_url: Option<String>,
}
/// Format plugin interface (FR-24.1)
///
/// Format plugins handle custom audio formats not supported by symphonia.
///
/// Example use cases:
/// - Custom lossless codecs
/// - Proprietary formats
/// - Game audio formats
pub trait FormatPlugin: Plugin {
/// File extensions this plugin handles
fn extensions(&self) -> &[&str];
/// Check if plugin can handle a specific extension
fn can_handle(&self, extension: &str) -> bool {
self.extensions()
.iter()
.any(|ext| ext.eq_ignore_ascii_case(extension))
}
/// Parse audio metadata from reader
///
/// The reader provides the raw file bytes. Plugin should parse
/// and return AudioMeta with whatever metadata it can extract.
fn parse(&self, reader: &mut dyn Read) -> Result<AudioMeta>;
/// Synthesize file header with updated metadata (FR-5.3)
///
/// Creates a new file header containing the provided metadata.
/// Used for metadata overlay - serving cached metadata without
/// modifying the original file.
fn synthesize_header(&self, metadata: &AudioMeta) -> Result<Vec<u8>>;
}
/// Declaration macro for native plugins
///
/// Native plugins must export a function with this signature:
/// ```ignore
/// #[no_mangle]
/// pub extern "C" fn musicfs_plugin_create() -> *mut dyn Plugin
/// ```
#[macro_export]
macro_rules! declare_plugin {
($plugin_type:ty, $constructor:expr) => {
#[no_mangle]
pub extern "C" fn musicfs_plugin_create() -> *mut dyn $crate::Plugin {
let plugin = $constructor;
let boxed: Box<dyn $crate::Plugin> = Box::new(plugin);
Box::into_raw(boxed)
}
#[no_mangle]
pub extern "C" fn musicfs_plugin_api_version() -> *const std::ffi::c_char {
concat!($crate::PLUGIN_API_VERSION, "\0").as_ptr() as *const std::ffi::c_char
}
};
}
#[cfg(test)]
mod tests {
use super::*;
struct TestPlugin {
name: String,
initialized: bool,
}
impl Plugin for TestPlugin {
fn name(&self) -> &str {
&self.name
}
fn version(&self) -> Version {
Version::new(1, 0, 0)
}
fn plugin_type(&self) -> PluginType {
PluginType::Origin
}
fn init(&mut self, _config: Value) -> Result<()> {
self.initialized = true;
Ok(())
}
fn shutdown(&mut self) -> Result<()> {
self.initialized = false;
Ok(())
}
}
#[test]
fn test_plugin_lifecycle() {
let mut plugin = TestPlugin {
name: "test".to_string(),
initialized: false,
};
assert_eq!(plugin.name(), "test");
assert!(!plugin.initialized);
plugin.init(Value::Null).unwrap();
assert!(plugin.initialized);
plugin.shutdown().unwrap();
assert!(!plugin.initialized);
}
#[test]
fn test_plugin_id() {
let id1 = PluginId::new(1);
let id2 = PluginId::new(1);
let id3 = PluginId::new(2);
assert_eq!(id1, id2);
assert_ne!(id1, id3);
}
}
+220
View File
@@ -0,0 +1,220 @@
use crate::error::{PluginError, Result};
use crate::traits::PluginId;
#[cfg(feature = "wasm")]
use std::sync::atomic::{AtomicU64, Ordering};
#[cfg(feature = "wasm")]
static NEXT_WASM_PLUGIN_ID: AtomicU64 = AtomicU64::new(1_000_000);
#[cfg(feature = "wasm")]
fn next_wasm_plugin_id() -> PluginId {
PluginId::new(NEXT_WASM_PLUGIN_ID.fetch_add(1, Ordering::SeqCst))
}
#[derive(Debug, Clone)]
pub struct ResourceLimits {
pub max_memory_mb: u32,
pub max_cpu_time_ms: u32,
pub allow_network: bool,
pub allow_filesystem: bool,
}
impl Default for ResourceLimits {
fn default() -> Self {
Self {
max_memory_mb: 64,
max_cpu_time_ms: 5000,
allow_network: false,
allow_filesystem: false,
}
}
}
#[cfg(feature = "wasm")]
mod wasm_impl {
use super::*;
use std::collections::HashMap;
use tracing::info;
use wasmtime::{Config, Engine, Linker, Module, Store};
pub struct PluginState {
limits: ResourceLimits,
}
pub struct WasmPlugin {
id: PluginId,
name: String,
_module: Module,
}
impl WasmPlugin {
pub fn id(&self) -> PluginId {
self.id
}
pub fn name(&self) -> &str {
&self.name
}
}
pub struct WasmPluginHost {
engine: Engine,
linker: Linker<PluginState>,
plugins: HashMap<PluginId, WasmPlugin>,
limits: ResourceLimits,
}
impl WasmPluginHost {
pub fn new() -> Result<Self> {
let mut config = Config::new();
config.consume_fuel(true);
config.epoch_interruption(true);
let engine = Engine::new(&config)
.map_err(|e| PluginError::Wasm(format!("Failed to create WASM engine: {}", e)))?;
let linker = Linker::new(&engine);
Ok(Self {
engine,
linker,
plugins: HashMap::new(),
limits: ResourceLimits::default(),
})
}
pub fn set_limits(&mut self, limits: ResourceLimits) {
self.limits = limits;
}
pub fn load(&mut self, wasm_bytes: &[u8]) -> Result<PluginId> {
info!("Loading WASM plugin ({} bytes)", wasm_bytes.len());
let module = Module::new(&self.engine, wasm_bytes)
.map_err(|e| PluginError::Wasm(format!("Failed to compile WASM module: {}", e)))?;
let id = next_wasm_plugin_id();
let name = module.name().unwrap_or("unnamed").to_string();
let plugin = WasmPlugin {
id,
name,
_module: module,
};
self.plugins.insert(id, plugin);
Ok(id)
}
pub fn unload(&mut self, id: PluginId) -> Result<()> {
self.plugins
.remove(&id)
.ok_or_else(|| PluginError::NotFound(format!("WASM plugin {:?}", id)))?;
Ok(())
}
pub fn get(&self, id: PluginId) -> Option<&WasmPlugin> {
self.plugins.get(&id)
}
pub fn list(&self) -> Vec<(PluginId, &str)> {
self.plugins.iter().map(|(id, p)| (*id, p.name())).collect()
}
fn create_store(&self) -> Store<PluginState> {
let state = PluginState {
limits: self.limits.clone(),
};
let mut store = Store::new(&self.engine, state);
let fuel = (self.limits.max_cpu_time_ms as u64) * 1_000_000;
store.set_fuel(fuel).ok();
store
}
}
impl Default for WasmPluginHost {
fn default() -> Self {
Self::new().expect("Failed to create WASM host")
}
}
}
#[cfg(not(feature = "wasm"))]
mod wasm_stub {
use super::*;
pub struct WasmPluginHost {
limits: ResourceLimits,
}
impl WasmPluginHost {
pub fn new() -> Result<Self> {
Ok(Self {
limits: ResourceLimits::default(),
})
}
pub fn set_limits(&mut self, limits: ResourceLimits) {
self.limits = limits;
}
pub fn load(&mut self, _wasm_bytes: &[u8]) -> Result<PluginId> {
Err(PluginError::Wasm(
"WASM support not enabled. Compile with --features wasm".to_string(),
))
}
pub fn unload(&mut self, _id: PluginId) -> Result<()> {
Err(PluginError::Wasm("WASM support not enabled".to_string()))
}
pub fn list(&self) -> Vec<(PluginId, &str)> {
Vec::new()
}
}
impl Default for WasmPluginHost {
fn default() -> Self {
Self::new().expect("Failed to create WASM stub host")
}
}
}
#[cfg(feature = "wasm")]
pub use wasm_impl::*;
#[cfg(not(feature = "wasm"))]
pub use wasm_stub::*;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_resource_limits_default() {
let limits = ResourceLimits::default();
assert_eq!(limits.max_memory_mb, 64);
assert_eq!(limits.max_cpu_time_ms, 5000);
assert!(!limits.allow_network);
assert!(!limits.allow_filesystem);
}
#[test]
fn test_wasm_host_creation() {
let host = WasmPluginHost::new();
assert!(host.is_ok());
}
#[test]
#[cfg(not(feature = "wasm"))]
fn test_wasm_disabled_load_fails() {
let mut host = WasmPluginHost::new().unwrap();
let result = host.load(&[0x00, 0x61, 0x73, 0x6d]);
assert!(result.is_err());
}
}
+23
View File
@@ -0,0 +1,23 @@
pkgname=musicfs
pkgver=0.1.0
pkgrel=1
pkgdesc="Metadata-Organized Music Filesystem"
arch=('x86_64')
url="https://github.com/yourusername/musicfs"
license=('MIT')
depends=('fuse3')
makedepends=('rust' 'cargo')
source=("$pkgname-$pkgver.tar.gz")
sha256sums=('SKIP')
build() {
cd "$srcdir/$pkgname-$pkgver"
cargo build --release --locked
}
package() {
cd "$srcdir/$pkgname-$pkgver"
install -Dm755 "target/release/musicfs" "$pkgdir/usr/bin/musicfs"
install -Dm644 "dist/musicfs.service" "$pkgdir/usr/lib/systemd/system/musicfs.service"
install -Dm644 "config.example.toml" "$pkgdir/etc/musicfs/config.example.toml"
}
+21
View File
@@ -0,0 +1,21 @@
[Unit]
Description=MusicFS - Metadata-Organized Music Filesystem
After=network.target
[Service]
Type=notify
ExecStart=/usr/bin/musicfs mount --config /etc/musicfs/config.toml /mnt/music
ExecStop=/usr/bin/musicfs shutdown
Restart=on-failure
RestartSec=5
User=musicfs
Group=musicfs
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/cache/musicfs /mnt/music
PrivateTmp=true
[Install]
WantedBy=multi-user.target
+39
View File
@@ -0,0 +1,39 @@
Name: musicfs
Version: 0.1.0
Release: 1%{?dist}
Summary: Metadata-Organized Music Filesystem
License: MIT
URL: https://github.com/yourusername/musicfs
Source0: %{name}-%{version}.tar.gz
BuildRequires: rust >= 1.70
BuildRequires: cargo
BuildRequires: fuse3-devel
Requires: fuse3
%description
MusicFS is a virtual FUSE filesystem that organizes music files by metadata.
%prep
%autosetup
%build
cargo build --release --locked
%install
install -Dm755 target/release/musicfs %{buildroot}%{_bindir}/musicfs
install -Dm644 dist/musicfs.service %{buildroot}%{_unitdir}/musicfs.service
install -Dm644 config.example.toml %{buildroot}%{_sysconfdir}/musicfs/config.example.toml
%files
%license LICENSE
%doc README.md
%{_bindir}/musicfs
%{_unitdir}/musicfs.service
%config(noreplace) %{_sysconfdir}/musicfs/config.example.toml
%changelog
* Mon Jan 01 2024 MusicFS Team <team@example.com> - 0.1.0-1
- Initial package
+90
View File
@@ -0,0 +1,90 @@
use std::process::Command;
#[test]
#[ignore]
fn test_mpv_playback() {
let mountpoint = setup_test_mount();
let output = Command::new("mpv")
.args([
"--no-video",
"--no-audio",
"--length=2",
"--msg-level=all=debug",
&format!("{}/Artist/Album/01 - Track.flac", mountpoint),
])
.output()
.expect("mpv must be installed");
assert!(
output.status.success(),
"mpv playback failed: {:?}",
output
);
}
#[test]
#[ignore]
fn test_vlc_playback() {
let mountpoint = setup_test_mount();
let output = Command::new("cvlc")
.args([
"--play-and-exit",
"--run-time=2",
&format!("{}/Artist/Album/", mountpoint),
])
.output()
.expect("vlc must be installed");
assert!(output.status.success(), "VLC playback failed");
}
#[test]
#[ignore]
fn test_file_manager_operations() {
let mountpoint = setup_test_mount();
let entries: Vec<_> = std::fs::read_dir(&mountpoint)
.expect("read_dir failed")
.collect();
assert!(!entries.is_empty(), "mountpoint should have entries");
for entry in entries {
let entry = entry.expect("entry should be valid");
let metadata = entry.metadata().expect("metadata should work");
assert!(metadata.is_dir() || metadata.is_file());
}
}
#[test]
#[ignore]
fn test_concurrent_player_access() {
let mountpoint = setup_test_mount();
let handles: Vec<_> = (0..3)
.map(|i| {
let mp = mountpoint.clone();
std::thread::spawn(move || {
Command::new("mpv")
.args([
"--no-video",
"--no-audio",
"--length=1",
&format!("{}/Artist/Album/0{} - Track.flac", mp, i + 1),
])
.output()
})
})
.collect();
for handle in handles {
let output = handle.join().unwrap().expect("mpv should run");
assert!(output.status.success());
}
}
fn setup_test_mount() -> String {
std::env::var("MUSICFS_TEST_MOUNT").unwrap_or_else(|_| "/tmp/musicfs-test".to_string())
}