Compare commits
35 Commits
872580602d
...
rewrite
| Author | SHA1 | Date | |
|---|---|---|---|
| 1dcfa7a803 | |||
| 78b4338f95 | |||
| 82be15b960 | |||
| 5f79b3c2b0 | |||
| f28091257e | |||
| 8f76dd5b39 | |||
| 8dba0c097d | |||
| d35aa2aecb | |||
| 0e50d8a2a6 | |||
| 9699da385b | |||
| d001e81128 | |||
| 601c466ce4 | |||
| 9eb537dfa4 | |||
| d1e1ac97c4 | |||
| ebf1c5b5e1 | |||
| 80dfe50aaa | |||
| 13f007ebb5 | |||
| 1a98d62357 | |||
| 4e946963e1 | |||
| 0779c0d076 | |||
| 24fd81ff9e | |||
| 53e6dbaef0 | |||
| a8262ac8f8 | |||
| 056d9b8c82 | |||
| d14710c073 | |||
| d49018bfc7 | |||
| 9737e26bc9 | |||
| 4f52549e46 | |||
| 68aba52362 | |||
| a5023a6441 | |||
| 3d1f9a206f | |||
| 536e117576 | |||
| b51c680af4 | |||
| f84061a30d | |||
| 9349d29200 |
@@ -1,4 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
export GIT_CONFIG_GLOBAL=/dev/null
|
||||
|
||||
eval "$(devenv direnvrc)"
|
||||
|
||||
|
||||
Generated
+3517
-41
File diff suppressed because it is too large
Load Diff
+46
-6
@@ -1,11 +1,51 @@
|
||||
[package]
|
||||
name = "musicfs"
|
||||
[workspace]
|
||||
members = [
|
||||
"crates/musicfs-proto",
|
||||
"crates/musicfs-core",
|
||||
"crates/musicfs-server",
|
||||
"crates/musicfs-client",
|
||||
]
|
||||
resolver = "2"
|
||||
|
||||
[workspace.package]
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
ctrlc = "3.5.2"
|
||||
[workspace.dependencies]
|
||||
musicfs-proto = { path = "crates/musicfs-proto" }
|
||||
musicfs-core = { path = "crates/musicfs-core" }
|
||||
|
||||
prost = "0.14"
|
||||
tonic = "0.14"
|
||||
tonic-prost = "0.14"
|
||||
tonic-health = "0.14"
|
||||
tonic-reflection = "0.14"
|
||||
|
||||
symphonia = { version = "0.5", default-features = false, features = [
|
||||
"aac", "alac", "flac", "mp3", "ogg", "vorbis", "wav",
|
||||
] }
|
||||
twox-hash = "2.1.2"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
|
||||
tracing-appender = "0.2.5"
|
||||
|
||||
tokio = { version = "1", features = [
|
||||
"macros", "rt-multi-thread", "signal", "fs", "io-util", "sync",
|
||||
] }
|
||||
anyhow = "1"
|
||||
async-trait = "0.1"
|
||||
clap = { version = "4.6.1", features = ["derive"] }
|
||||
notify = "8.2.0"
|
||||
tokio-stream = "0.1"
|
||||
|
||||
fuser = "0.17.0"
|
||||
libc = "0.2.186"
|
||||
serde_json = "1.0.150"
|
||||
time = "0.3.49"
|
||||
sea-orm = { version = "2.0.0-rc", features = [
|
||||
"sqlx-postgres", "runtime-tokio", "macros",
|
||||
] }
|
||||
log = "0.4"
|
||||
bytes = "1"
|
||||
http = "1"
|
||||
chrono = { version = "0.4", default-features = false, features = ["clock"] }
|
||||
|
||||
tempfile = "3"
|
||||
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
#+title: musicfs
|
||||
|
||||
A read-only FUSE filesystem that presents a music library reorganised by
|
||||
*metadata* rather than by however the files happen to sit on disk. Point it at a
|
||||
messy ~~/Music~ tree (or a remote ~musicfs-server~) and it mounts a clean
|
||||
=Artist/Album/Track= view, driven entirely by the tags parsed out of the audio
|
||||
files themselves.
|
||||
|
||||
#+begin_example
|
||||
mountpoint/
|
||||
├── ДДТ/
|
||||
│ └── Творчество в пустоте/
|
||||
│ ├── 01 Intro.flac
|
||||
│ └── 02 ...
|
||||
└── Some Artist/
|
||||
└── Some Album/
|
||||
└── 01 Track.flac
|
||||
#+end_example
|
||||
|
||||
The source files are never moved or modified. The directory hierarchy is
|
||||
*virtual*: synthesised from ~album_artist~/~album~ tags, with stable synthetic
|
||||
inodes so the layout survives remounts.
|
||||
|
||||
* How it works
|
||||
|
||||
Two *origins* feed the same FUSE frontend:
|
||||
|
||||
- *LocalOrigin* — walks a directory on the host, parses tags, builds the
|
||||
snapshot. A =notify= watcher keeps it live as files change.
|
||||
- *NetworkOrigin* — talks gRPC to a remote ~musicfs-server~. Metadata and file
|
||||
bytes arrive over the wire; a Postgres table caches both. Reconciliation is
|
||||
hash-based: the client sends the ~(inode, hash)~ pairs it has, the server
|
||||
replies with only what changed or was deleted, so almost nothing crosses the
|
||||
network on a steady-state refresh.
|
||||
|
||||
Which one is used is chosen automatically from the ~--source~ argument: an
|
||||
=http(s)://= URL means network, anything else is treated as a local path.
|
||||
|
||||
State lives in Postgres — parsed metadata, the virtual-path layout, and (for the
|
||||
network origin) a lazy byte cache populated only for files that were actually
|
||||
read.
|
||||
|
||||
* Layout
|
||||
|
||||
| Crate | Responsibility |
|
||||
|------------------+-----------------------------------------------------------------------|
|
||||
| =musicfs-proto= | Protobuf/gRPC definitions (=proto/musicfs.proto=), generated bindings |
|
||||
| =musicfs-core= | Shared logic: tag parsing (FLAC/MP3 via symphonia), hashing, logging |
|
||||
| =musicfs-client= | FUSE mount, origins, Postgres cache/sync — the =musicfs= binary |
|
||||
| =musicfs-server= | gRPC server sharing a library — the =musicfs-server= binary |
|
||||
|
||||
The gRPC surface (see =proto/musicfs.proto=): ~Reconcile~, ~GetMetadata~,
|
||||
~GetManifest~, ~GetFile~ (whole-file or byte-range streaming), and
|
||||
~SubscribeEvents~ (change wake-ups). The client also serves a ~ClientStatus~ RPC
|
||||
for introspection.
|
||||
|
||||
* Running the client
|
||||
|
||||
Everything runs inside the =devenv= shell, which provides the Rust toolchain,
|
||||
Postgres, FUSE tooling, and gRPC utilities.
|
||||
|
||||
#+begin_src bash
|
||||
devenv shell
|
||||
devenv up --profile local # starts Postgres + the FUSE mount
|
||||
#+end_src
|
||||
|
||||
Predefined profiles in =devenv.nix=:
|
||||
|
||||
| Profile | Source | Purpose |
|
||||
|-----------+---------------------------------+-------------------------------|
|
||||
| =local= | a local directory | mount a host music folder |
|
||||
| =remote= | =http://…:50051= | mount a remote musicfs-server |
|
||||
| =e2e= | =http://127.0.0.1:50061= | end-to-end test harness |
|
||||
|
||||
Or run the binary directly:
|
||||
|
||||
#+begin_src bash
|
||||
cargo run -p musicfs-client -- \
|
||||
--source /home/you/Music \
|
||||
--mountpoint /tmp/musicfs \
|
||||
--database "postgresql://you@localhost/musicfs?host=$PGHOST"
|
||||
#+end_src
|
||||
|
||||
Key flags: =--source= (local path or ~http(s)://~ server URL), =--mountpoint=,
|
||||
=--database= (Postgres URL), =--log-dir= (daily-rotated logs, default =./logs=),
|
||||
=--listen= (address for the client's status/health RPCs, default
|
||||
=127.0.0.1:50052=).
|
||||
|
||||
* Running the server in an Incus VM
|
||||
|
||||
~musicfs-server~ (gRPC) runs inside an Incus VM, live-sharing the host's
|
||||
~~/Music~ as a read-only virtiofs mount. =scripts/vm.sh= builds the binary on
|
||||
the host, bundles its nix ~glibc~ so it runs unmodified in the VM, ships it in,
|
||||
and runs it under systemd.
|
||||
|
||||
#+begin_src bash
|
||||
devenv shell # cargo, incus, patchelf, grpcurl
|
||||
scripts/vm.sh up # create VM, share ~/Music, build, ship, start server
|
||||
#+end_src
|
||||
|
||||
Options (env): =VM_NAME=, =IMAGE=, =MUSIC_SOURCE=, =LISTEN_PORT=, =RUST_LOG=.
|
||||
|
||||
| Command | Action |
|
||||
|------------------------+-------------------------------|
|
||||
| =scripts/vm.sh up= | create + build + ship + start |
|
||||
| =scripts/vm.sh redeploy= | rebuild after a code change |
|
||||
| =scripts/vm.sh logs= | tail server logs |
|
||||
| =scripts/vm.sh status= | VM + service status |
|
||||
| =scripts/vm.sh shell= | shell inside the VM |
|
||||
| =scripts/vm.sh down= | stop the VM |
|
||||
| =scripts/vm.sh destroy= | delete the VM |
|
||||
|
||||
Reach the server at the VM's bridge IP (=scripts/vm.sh status= prints it).
|
||||
|
||||
** Poke at it (Nushell)
|
||||
|
||||
#+begin_src nushell
|
||||
let VMIP = (incus list musicfs -f csv -c 4 | lines | first | split row " " | first)
|
||||
let ADDR = $"($VMIP):50051"
|
||||
|
||||
# liveness
|
||||
^nc -z -w 3 $VMIP 50051
|
||||
if $env.LAST_EXIT_CODE == 0 { print "alive" } else { print "dead" }
|
||||
|
||||
# gRPC: stream the manifest, count entries
|
||||
grpcurl -plaintext -import-path proto -proto musicfs.proto -d "{}" $ADDR musicfs.MusicFs/GetManifest | lines | find relPath | length
|
||||
|
||||
# files the server scans
|
||||
incus exec musicfs -- find /music -type f | lines | length
|
||||
|
||||
# play a track straight out of the VM's shared /music
|
||||
incus exec musicfs -- cat "/music/DDT/ДДТ - Творчество в пустоте – 2 - 01 Intro.flac" | ^mpv -
|
||||
#+end_src
|
||||
|
||||
Install mpv with =nix profile install nixpkgs#mpv=, or use =^ffplay -= instead.
|
||||
|
||||
In Nushell, always put =| lines= between an external command's stdout and a
|
||||
builtin (=find=, =first=, =length=, …); external-to-external pipes (=cat | mpv=)
|
||||
are byte-stable and don't need it.
|
||||
|
||||
* Development
|
||||
|
||||
#+begin_src bash
|
||||
just build # cargo build
|
||||
cargo test # unit tests
|
||||
just e2e # end-to-end suite (scripts/e2e/run.sh)
|
||||
just e2e-resilience # resilience suite
|
||||
#+end_src
|
||||
|
||||
Formatting and lint (clippy, treefmt with rustfmt + nixfmt) run as git hooks via
|
||||
=devenv=. Requires Rust edition 2024.
|
||||
@@ -0,0 +1,39 @@
|
||||
[package]
|
||||
name = "musicfs-client"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
default-run = "musicfs"
|
||||
|
||||
[lib]
|
||||
name = "musicfs"
|
||||
path = "src/lib.rs"
|
||||
|
||||
[[bin]]
|
||||
name = "musicfs"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
musicfs-core.workspace = true
|
||||
musicfs-proto.workspace = true
|
||||
fuser.workspace = true
|
||||
libc.workspace = true
|
||||
sea-orm.workspace = true
|
||||
tokio.workspace = true
|
||||
anyhow.workspace = true
|
||||
async-trait.workspace = true
|
||||
clap.workspace = true
|
||||
notify.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
tonic.workspace = true
|
||||
tonic-health.workspace = true
|
||||
tonic-reflection.workspace = true
|
||||
bytes.workspace = true
|
||||
http.workspace = true
|
||||
tracing.workspace = true
|
||||
log.workspace = true
|
||||
twox-hash.workspace = true
|
||||
chrono.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile.workspace = true
|
||||
sea-orm = { workspace = true, features = ["mock"] }
|
||||
@@ -0,0 +1,248 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
path::PathBuf,
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use musicfs_core::music::encoder::MusicMetadataEncoderFactory;
|
||||
use musicfs_proto::{
|
||||
ClientControl, FileEntry, FileMetadata, GetMusicMetadataRequest, GetMusicMetadataResponse,
|
||||
ListFilesRequest, ListFilesResponse, MusicMetadata as ProtoMusicMetadata, PictureDataRange,
|
||||
UpdateMusicMetadataRequest, UpdateMusicMetadataResponse,
|
||||
};
|
||||
use sea_orm::{ActiveModelTrait, DatabaseConnection, EntityTrait};
|
||||
use tonic::{Request, Response, Status};
|
||||
|
||||
use crate::db::entities;
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::music::db::save_music_metadata;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::virtual_dirs::{
|
||||
compute_new_layout, ensure_virtual_dirs, find_orphaned_dirs, parent_inode_from_path,
|
||||
};
|
||||
|
||||
pub struct ClientControlServiceImpl {
|
||||
files: Arc<Mutex<BTreeMap<INodeNo, Item>>>,
|
||||
db: DatabaseConnection,
|
||||
}
|
||||
|
||||
impl ClientControlServiceImpl {
|
||||
pub fn new(files: Arc<Mutex<BTreeMap<INodeNo, Item>>>, db: DatabaseConnection) -> Self {
|
||||
ClientControlServiceImpl { files, db }
|
||||
}
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ClientControl for ClientControlServiceImpl {
|
||||
async fn get_music_metadata(
|
||||
&self,
|
||||
request: Request<GetMusicMetadataRequest>,
|
||||
) -> Result<Response<GetMusicMetadataResponse>, Status> {
|
||||
let inode = INodeNo(request.into_inner().inode);
|
||||
let metadata = {
|
||||
let files = self.files.lock().unwrap();
|
||||
let item = files
|
||||
.get(&inode)
|
||||
.ok_or_else(|| Status::not_found(format!("inode {} not found", inode.0)))?;
|
||||
item.music_metadata.clone().ok_or_else(|| {
|
||||
Status::not_found(format!("inode {} has no music metadata", inode.0))
|
||||
})?
|
||||
};
|
||||
Ok(Response::new(GetMusicMetadataResponse {
|
||||
metadata: Some(music_metadata_to_proto(metadata)),
|
||||
}))
|
||||
}
|
||||
|
||||
async fn update_music_metadata(
|
||||
&self,
|
||||
request: Request<UpdateMusicMetadataRequest>,
|
||||
) -> Result<Response<UpdateMusicMetadataResponse>, Status> {
|
||||
let req = request.into_inner();
|
||||
let inode = INodeNo(req.inode);
|
||||
|
||||
let outcome = {
|
||||
let mut files = self.files.lock().unwrap();
|
||||
|
||||
let (old_local_path, current_name, source_for_virtual_dirs) = {
|
||||
let root = files.get(&INodeNo::ROOT);
|
||||
let item = files
|
||||
.get(&inode)
|
||||
.ok_or_else(|| Status::not_found(format!("inode {} not found", inode.0)))?;
|
||||
let source = root
|
||||
.map(|r| r.original_path.clone())
|
||||
.unwrap_or_else(|| PathBuf::from("/"));
|
||||
(item.local_path.clone(), item.name.clone(), source)
|
||||
};
|
||||
|
||||
// Phase 1: mutate the music metadata and re-encode the header.
|
||||
// Done in its own scope so the mutable borrow on `files` is
|
||||
// released before we touch `files` again for layout changes.
|
||||
let (updated_mm, _encoder_extension_source) = {
|
||||
let item = files
|
||||
.get_mut(&inode)
|
||||
.ok_or_else(|| Status::not_found(format!("inode {} not found", inode.0)))?;
|
||||
let mm = item.music_metadata.as_mut().ok_or_else(|| {
|
||||
Status::not_found(format!("inode {} has no music metadata", inode.0))
|
||||
})?;
|
||||
apply_update(mm, req);
|
||||
let encoder =
|
||||
MusicMetadataEncoderFactory::for_path(&item.local_path).ok_or_else(|| {
|
||||
Status::failed_precondition(format!(
|
||||
"no encoder for file type: {}",
|
||||
item.local_path.display()
|
||||
))
|
||||
})?;
|
||||
encoder.encode(mm);
|
||||
(mm.clone(), item.local_path.clone())
|
||||
};
|
||||
|
||||
// Phase 2: compute new layout, write it onto the item, then
|
||||
// ensure new virtual dirs and prune orphans. A fresh mutable
|
||||
// borrow is fine because phase 1's borrow has been released.
|
||||
let (new_name, new_local_path) = compute_new_layout(¤t_name, &updated_mm);
|
||||
{
|
||||
let item = files
|
||||
.get_mut(&inode)
|
||||
.ok_or_else(|| Status::not_found(format!("inode {} not found", inode.0)))?;
|
||||
item.name = new_name.clone();
|
||||
item.local_path = new_local_path.clone();
|
||||
item.parent_inode = parent_inode_from_path(&new_local_path);
|
||||
}
|
||||
ensure_virtual_dirs(&new_local_path, &source_for_virtual_dirs, &mut files);
|
||||
|
||||
let orphaned = find_orphaned_dirs(&files, inode, &old_local_path);
|
||||
for dir_inode in &orphaned {
|
||||
files.remove(dir_inode);
|
||||
}
|
||||
|
||||
UpdateOutcome {
|
||||
metadata: updated_mm,
|
||||
new_name,
|
||||
new_local_path,
|
||||
orphaned_dirs: orphaned,
|
||||
}
|
||||
};
|
||||
|
||||
save_music_metadata(inode.0 as i64, &outcome.metadata, &self.db)
|
||||
.await
|
||||
.map_err(|e| Status::internal(format!("DB save failed: {e}")))?;
|
||||
|
||||
persist_layout_changes(
|
||||
&self.db,
|
||||
inode.0 as i64,
|
||||
&outcome.new_name,
|
||||
&outcome.new_local_path,
|
||||
&outcome.orphaned_dirs,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| Status::internal(format!("DB layout save failed: {e}")))?;
|
||||
|
||||
tracing::debug!(
|
||||
ino = inode.0,
|
||||
orphaned_dirs = outcome.orphaned_dirs.len(),
|
||||
"control: music metadata updated, layout recomputed"
|
||||
);
|
||||
|
||||
Ok(Response::new(UpdateMusicMetadataResponse {
|
||||
metadata: Some(music_metadata_to_proto(outcome.metadata)),
|
||||
}))
|
||||
}
|
||||
|
||||
async fn list_files(
|
||||
&self,
|
||||
_request: Request<ListFilesRequest>,
|
||||
) -> Result<Response<ListFilesResponse>, Status> {
|
||||
let entries: Vec<FileEntry> = {
|
||||
let files = self.files.lock().unwrap();
|
||||
files
|
||||
.values()
|
||||
.filter(|item| item.file_type == FileType::File)
|
||||
.map(|item| FileEntry {
|
||||
inode: item.inode.0,
|
||||
original_path: item.original_path.to_string_lossy().into_owned(),
|
||||
local_path: item.local_path.to_string_lossy().into_owned(),
|
||||
name: item.name.clone(),
|
||||
metadata: item
|
||||
.music_metadata
|
||||
.as_ref()
|
||||
.map(|mm| music_metadata_to_file_metadata(mm)),
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
Ok(Response::new(ListFilesResponse { files: entries }))
|
||||
}
|
||||
}
|
||||
|
||||
fn music_metadata_to_file_metadata(mm: &MusicMetadata) -> FileMetadata {
|
||||
FileMetadata {
|
||||
artist: mm.artist.clone(),
|
||||
album_artist: mm.album_artist.clone(),
|
||||
album: mm.album.clone(),
|
||||
track_number: mm.track_number,
|
||||
track_title: mm.track_title.clone(),
|
||||
other_tags: mm.other_tags.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
fn music_metadata_to_proto(mm: crate::music::metadata::MusicMetadata) -> ProtoMusicMetadata {
|
||||
ProtoMusicMetadata {
|
||||
artist: mm.artist,
|
||||
album_artist: mm.album_artist,
|
||||
album: mm.album,
|
||||
track_number: mm.track_number,
|
||||
track_title: mm.track_title,
|
||||
other_tags: mm.other_tags,
|
||||
header: mm.header,
|
||||
picture_block_headers: mm.picture_block_headers,
|
||||
picture_data_ranges: mm
|
||||
.picture_data_ranges
|
||||
.into_iter()
|
||||
.map(|(offset, length)| PictureDataRange { offset, length })
|
||||
.collect(),
|
||||
real_audio_start: mm.real_audio_start,
|
||||
vorbis_comment_offset: mm.vorbis_comment_offset,
|
||||
vorbis_comment_length: mm.vorbis_comment_length,
|
||||
}
|
||||
}
|
||||
|
||||
struct UpdateOutcome {
|
||||
metadata: MusicMetadata,
|
||||
new_name: String,
|
||||
new_local_path: PathBuf,
|
||||
orphaned_dirs: Vec<INodeNo>,
|
||||
}
|
||||
|
||||
fn apply_update(mm: &mut MusicMetadata, req: UpdateMusicMetadataRequest) {
|
||||
mm.artist = req.artist;
|
||||
mm.album_artist = req.album_artist;
|
||||
mm.album = req.album;
|
||||
mm.track_number = req.track_number;
|
||||
mm.track_title = req.track_title;
|
||||
mm.other_tags = req.other_tags;
|
||||
}
|
||||
|
||||
async fn persist_layout_changes(
|
||||
db: &DatabaseConnection,
|
||||
inode: i64,
|
||||
new_name: &str,
|
||||
new_local_path: &std::path::Path,
|
||||
orphaned_dirs: &[INodeNo],
|
||||
) -> Result<(), sea_orm::DbErr> {
|
||||
use sea_orm::ActiveValue::Set;
|
||||
let new_local_path_str = new_local_path.to_string_lossy().into_owned();
|
||||
let update = entities::ActiveModel {
|
||||
inode: Set(inode),
|
||||
name: Set(new_name.to_string()),
|
||||
local_path: Set(new_local_path_str),
|
||||
..Default::default()
|
||||
};
|
||||
update.update(db).await?;
|
||||
|
||||
for dir_inode in orphaned_dirs {
|
||||
let _ = entities::Entity::delete_by_id(dir_inode.0 as i64)
|
||||
.exec(db)
|
||||
.await;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
use sea_orm::{
|
||||
ActiveValue::Set, ColumnTrait, DatabaseConnection, EntityTrait, QueryFilter,
|
||||
sea_query::OnConflict,
|
||||
};
|
||||
use tracing::{debug, error};
|
||||
|
||||
use crate::db::entities::cached_file_bytes::{ActiveModel, Column, Entity, Model};
|
||||
|
||||
/// Return the cached bytes for `inode` if present. `None` means "not cached";
|
||||
/// the caller should fetch from the server and call [`put_cached_bytes`].
|
||||
pub async fn get_cached_bytes(inode: i64, db: &DatabaseConnection) -> Option<Vec<u8>> {
|
||||
let row: Option<Model> = match Entity::find_by_id(inode).one(db).await {
|
||||
Ok(m) => m,
|
||||
Err(e) => {
|
||||
error!(%inode, error = %e, "get_cached_bytes: DB read failed");
|
||||
None
|
||||
}
|
||||
};
|
||||
return row.map(|m| m.data);
|
||||
}
|
||||
|
||||
/// Insert or replace the cached bytes for `inode`. Called after a successful
|
||||
/// `GetFile` round-trip so subsequent reads of the same range hit the cache.
|
||||
pub async fn put_cached_bytes(inode: i64, data: Vec<u8>, db: &DatabaseConnection) {
|
||||
let active = ActiveModel {
|
||||
inode: Set(inode),
|
||||
data: Set(data),
|
||||
fetched_at: Set(chrono::Utc::now()),
|
||||
};
|
||||
if let Err(e) = Entity::insert(active)
|
||||
.on_conflict(
|
||||
OnConflict::column(Column::Inode)
|
||||
.update_columns([Column::Data, Column::FetchedAt])
|
||||
.to_owned(),
|
||||
)
|
||||
.exec(db)
|
||||
.await
|
||||
{
|
||||
error!(%inode, error = %e, "put_cached_bytes: DB upsert failed");
|
||||
}
|
||||
}
|
||||
|
||||
/// Remove cache rows for the given inodes. Called by reconcile when a file's
|
||||
/// hash has changed (the cached bytes are now stale).
|
||||
pub async fn delete_cached_bytes_for(inodes: &[i64], db: &DatabaseConnection) {
|
||||
if inodes.is_empty() {
|
||||
return;
|
||||
}
|
||||
debug!(count = inodes.len(), "deleting cached bytes");
|
||||
if let Err(e) = Entity::delete_many()
|
||||
.filter(Column::Inode.is_in(inodes.to_vec()))
|
||||
.exec(db)
|
||||
.await
|
||||
{
|
||||
error!(error = %e, "delete_cached_bytes_for: DB delete failed");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use sea_orm::ConnectionTrait;
|
||||
|
||||
async fn connect() -> Option<DatabaseConnection> {
|
||||
let pg_host = std::env::var("PGHOST").ok()?;
|
||||
let url = format!("postgresql://fujin@localhost/musicfs?host={pg_host}");
|
||||
let db = sea_orm::Database::connect(&url).await.ok()?;
|
||||
db.execute_unprepared("SELECT 1 FROM cached_file_bytes LIMIT 0")
|
||||
.await
|
||||
.ok()?;
|
||||
Some(db)
|
||||
}
|
||||
|
||||
async fn setup_parent_item(db: &DatabaseConnection, inode: i64) {
|
||||
db.execute_unprepared(&format!(
|
||||
"INSERT INTO items (inode, name, original_path, local_path, file_type, hash) \
|
||||
VALUES ({inode}, 'test', '/test', '/test', 'file', 0) \
|
||||
ON CONFLICT (inode) DO NOTHING"
|
||||
))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn teardown(db: &DatabaseConnection, inode: i64) {
|
||||
let _ = db
|
||||
.execute_unprepared(&format!("DELETE FROM items WHERE inode = {inode}"))
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cache_round_trip_with_timestamptz() {
|
||||
let db = match connect().await {
|
||||
Some(db) => db,
|
||||
None => {
|
||||
eprintln!("skip: no database (PGHOST unset or unreachable)");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let inode = -999_999_i64;
|
||||
let test_data = b"regression-test-payload".to_vec();
|
||||
|
||||
setup_parent_item(&db, inode).await;
|
||||
put_cached_bytes(inode, test_data.clone(), &db).await;
|
||||
|
||||
let cached = get_cached_bytes(inode, &db).await;
|
||||
assert_eq!(
|
||||
cached,
|
||||
Some(test_data),
|
||||
"get_cached_bytes must decode fetched_at (TIMESTAMPTZ), not fail with type mismatch"
|
||||
);
|
||||
|
||||
teardown(&db, inode).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
use crate::item::{FileType, Item};
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "items")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
pub name: String,
|
||||
pub original_path: String,
|
||||
pub local_path: String,
|
||||
pub file_type: String,
|
||||
pub hash: i64,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
|
||||
impl From<&Item> for ActiveModel {
|
||||
fn from(item: &Item) -> Self {
|
||||
use sea_orm::ActiveValue::Set;
|
||||
ActiveModel {
|
||||
inode: Set(item.inode.0 as i64),
|
||||
name: Set(item.name.clone()),
|
||||
original_path: Set(item.original_path.to_string_lossy().into_owned()),
|
||||
local_path: Set(item.local_path.to_string_lossy().into_owned()),
|
||||
file_type: Set(match item.file_type {
|
||||
FileType::Directory => "directory".to_string(),
|
||||
FileType::File => "file".to_string(),
|
||||
}),
|
||||
hash: Set(item.hash as i64),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub mod cached_file_bytes {
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "cached_file_bytes")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
#[sea_orm(column_type = "Blob")]
|
||||
pub data: Vec<u8>,
|
||||
pub fetched_at: DateTimeUtc,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
use fuser::INodeNo;
|
||||
use std::path::PathBuf;
|
||||
|
||||
fn attrs_for_tempdir() -> FileAttrs {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = std::fs::metadata(tmp.path()).unwrap();
|
||||
return FileAttrs::from(&metadata);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn item_to_active_model_fields_match() {
|
||||
let item = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
PathBuf::from("/original/path"),
|
||||
PathBuf::from("local/path"),
|
||||
FileType::Directory,
|
||||
attrs_for_tempdir(),
|
||||
None,
|
||||
);
|
||||
let am = ActiveModel::from(&item);
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am.inode {
|
||||
assert_eq!(*v, 42i64);
|
||||
} else {
|
||||
panic!("inode field not set");
|
||||
}
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am.name {
|
||||
assert_eq!(v, "test");
|
||||
} else {
|
||||
panic!("name field not set");
|
||||
}
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am.original_path {
|
||||
assert_eq!(v, "/original/path");
|
||||
} else {
|
||||
panic!("original_path field not set");
|
||||
}
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am.local_path {
|
||||
assert_eq!(v, "local/path");
|
||||
} else {
|
||||
panic!("local_path field not set");
|
||||
}
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am.hash {
|
||||
assert_eq!(*v, item.hash as i64);
|
||||
} else {
|
||||
panic!("hash field not set");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_type_string_mapping() {
|
||||
let item_dir = Item::new(
|
||||
INodeNo(1),
|
||||
INodeNo::ROOT,
|
||||
"dir".to_string(),
|
||||
PathBuf::from("/original/dir"),
|
||||
PathBuf::from("local/dir"),
|
||||
FileType::Directory,
|
||||
attrs_for_tempdir(),
|
||||
None,
|
||||
);
|
||||
let am_dir = ActiveModel::from(&item_dir);
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am_dir.file_type {
|
||||
assert_eq!(v, "directory");
|
||||
} else {
|
||||
panic!("file_type field not set for directory");
|
||||
}
|
||||
|
||||
let item_file = Item::new(
|
||||
INodeNo(2),
|
||||
INodeNo::ROOT,
|
||||
"file".to_string(),
|
||||
PathBuf::from("/original/file"),
|
||||
PathBuf::from("local/file"),
|
||||
FileType::File,
|
||||
attrs_for_tempdir(),
|
||||
None,
|
||||
);
|
||||
let am_file = ActiveModel::from(&item_file);
|
||||
|
||||
if let sea_orm::ActiveValue::Set(v) = &am_file.file_type {
|
||||
assert_eq!(v, "file");
|
||||
} else {
|
||||
panic!("file_type field not set for file");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
pub mod cache;
|
||||
pub mod entities;
|
||||
pub mod sync;
|
||||
@@ -0,0 +1,92 @@
|
||||
use std::collections::{BTreeMap, HashMap, HashSet};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use sea_orm::entity::prelude::*;
|
||||
use tracing::{debug, error};
|
||||
|
||||
use crate::db::entities::{ActiveModel, Entity, Model};
|
||||
use crate::item::Item;
|
||||
use crate::music::db::save_music_metadata;
|
||||
|
||||
pub async fn sync_items_to_db(
|
||||
snapshot: &BTreeMap<INodeNo, Item>,
|
||||
db_items: &HashMap<i64, Model>,
|
||||
client: &sea_orm::DatabaseConnection,
|
||||
) {
|
||||
let mut to_insert: Vec<ActiveModel> = vec![];
|
||||
let mut to_update: Vec<ActiveModel> = vec![];
|
||||
let mut to_delete: Vec<i64> = vec![];
|
||||
let mut to_save_music: Vec<i64> = vec![];
|
||||
|
||||
for (ino, item) in snapshot {
|
||||
let ino_i64 = ino.0 as i64;
|
||||
match db_items.get(&ino_i64) {
|
||||
None => {
|
||||
to_insert.push(ActiveModel::from(item));
|
||||
if item.music_metadata.is_some() {
|
||||
to_save_music.push(ino_i64);
|
||||
}
|
||||
}
|
||||
Some(db_item) if db_item.hash != item.hash as i64 => {
|
||||
to_update.push(ActiveModel::from(item));
|
||||
if item.music_metadata.is_some() {
|
||||
to_save_music.push(ino_i64);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
let fresh_inodes: HashSet<i64> = snapshot.keys().map(|i| i.0 as i64).collect();
|
||||
for ino in db_items.keys().filter(|i| !fresh_inodes.contains(i)) {
|
||||
to_delete.push(*ino);
|
||||
}
|
||||
|
||||
debug!(
|
||||
insert = to_insert.len(),
|
||||
update = to_update.len(),
|
||||
delete = to_delete.len(),
|
||||
save_music = to_save_music.len(),
|
||||
"sync_items_to_db"
|
||||
);
|
||||
|
||||
if !to_insert.is_empty() {
|
||||
if let Err(e) = Entity::insert_many(to_insert).exec(client).await {
|
||||
error!(error = %e, "sync: insert_many failed");
|
||||
}
|
||||
}
|
||||
for model in to_update {
|
||||
if let Err(e) = model.update(client).await {
|
||||
error!(error = %e, "sync: update failed");
|
||||
}
|
||||
}
|
||||
for ino in to_delete {
|
||||
if let Err(e) = Entity::delete_by_id(ino).exec(client).await {
|
||||
error!(%ino, error = %e, "sync: delete failed");
|
||||
}
|
||||
}
|
||||
|
||||
for ino_i64 in &to_save_music {
|
||||
let ino = INodeNo(*ino_i64 as u64);
|
||||
if let Some(music_metadata) = snapshot
|
||||
.get(&ino)
|
||||
.and_then(|item| item.music_metadata.as_ref())
|
||||
{
|
||||
if let Err(e) = save_music_metadata(*ino_i64, music_metadata, client).await {
|
||||
error!(ino = *ino_i64, error = %e, "sync: save_music_metadata failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn run_db_blocking<F: std::future::Future>(future: F) -> F::Output {
|
||||
return tokio::runtime::Builder::new_current_thread()
|
||||
.enable_io()
|
||||
.enable_time()
|
||||
.build()
|
||||
.unwrap_or_else(|e| {
|
||||
error!(error = %e, "run_db_blocking: failed to build runtime");
|
||||
panic!("run_db_blocking: failed to build runtime: {e}");
|
||||
})
|
||||
.block_on(future);
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
net::SocketAddr,
|
||||
sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
},
|
||||
time::{SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use musicfs_proto::{
|
||||
ClientControlServer, ClientStatus, ClientStatusResponse, ClientStatusServer,
|
||||
GetClientStatusRequest,
|
||||
};
|
||||
use tokio::time::Duration;
|
||||
use tonic::{Request, Response, transport::Server};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct ServerStatus {
|
||||
connected: Arc<AtomicBool>,
|
||||
last_reconcile: Arc<Mutex<Option<i64>>>,
|
||||
origin_type: String,
|
||||
mountpoint: String,
|
||||
server_endpoint: String,
|
||||
}
|
||||
|
||||
impl ServerStatus {
|
||||
pub fn new_network(mountpoint: String, server_endpoint: String) -> Self {
|
||||
ServerStatus {
|
||||
connected: Arc::new(AtomicBool::new(false)),
|
||||
last_reconcile: Arc::new(Mutex::new(None)),
|
||||
origin_type: "network".to_string(),
|
||||
mountpoint,
|
||||
server_endpoint,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn new_local(mountpoint: String) -> Self {
|
||||
ServerStatus {
|
||||
connected: Arc::new(AtomicBool::new(true)),
|
||||
last_reconcile: Arc::new(Mutex::new(None)),
|
||||
origin_type: "local".to_string(),
|
||||
mountpoint,
|
||||
server_endpoint: String::new(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn set_connected(&self, connected: bool) {
|
||||
self.connected.store(connected, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub fn mark_reconciled(&self) {
|
||||
let now = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs() as i64)
|
||||
.unwrap_or(0);
|
||||
*self.last_reconcile.lock().unwrap() = Some(now);
|
||||
}
|
||||
|
||||
pub fn is_server_connected(&self) -> bool {
|
||||
self.connected.load(Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn spawn_grpc_server(
|
||||
addr: SocketAddr,
|
||||
status: ServerStatus,
|
||||
files: Arc<Mutex<BTreeMap<INodeNo, crate::item::Item>>>,
|
||||
db: sea_orm::DatabaseConnection,
|
||||
) {
|
||||
let (reporter, health_service) = tonic_health::server::health_reporter();
|
||||
reporter
|
||||
.set_serving::<ClientStatusServer<ClientStatusServiceImpl>>()
|
||||
.await;
|
||||
|
||||
let reporter_clone = reporter.clone();
|
||||
let status_clone = status.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut was_connected = status_clone.is_server_connected();
|
||||
loop {
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
let now_connected = status_clone.is_server_connected();
|
||||
if now_connected != was_connected {
|
||||
if now_connected {
|
||||
reporter_clone
|
||||
.set_serving::<ClientStatusServer<ClientStatusServiceImpl>>()
|
||||
.await;
|
||||
} else {
|
||||
reporter_clone
|
||||
.set_not_serving::<ClientStatusServer<ClientStatusServiceImpl>>()
|
||||
.await;
|
||||
}
|
||||
was_connected = now_connected;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
let reflection_v1 = tonic_reflection::server::Builder::configure()
|
||||
.register_encoded_file_descriptor_set(musicfs_proto::musicfs::FILE_DESCRIPTOR_SET)
|
||||
.register_encoded_file_descriptor_set(tonic_health::pb::FILE_DESCRIPTOR_SET)
|
||||
.build_v1()
|
||||
.expect("build reflection v1");
|
||||
|
||||
let reflection_v1alpha = tonic_reflection::server::Builder::configure()
|
||||
.register_encoded_file_descriptor_set(musicfs_proto::musicfs::FILE_DESCRIPTOR_SET)
|
||||
.register_encoded_file_descriptor_set(tonic_health::pb::FILE_DESCRIPTOR_SET)
|
||||
.build_v1alpha()
|
||||
.expect("build reflection v1alpha");
|
||||
|
||||
let status_svc = ClientStatusServer::new(ClientStatusServiceImpl {
|
||||
status,
|
||||
files: files.clone(),
|
||||
});
|
||||
let control_svc =
|
||||
ClientControlServer::new(crate::control::ClientControlServiceImpl::new(files, db));
|
||||
|
||||
tracing::info!(%addr, "health server listening");
|
||||
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = Server::builder()
|
||||
.add_service(health_service)
|
||||
.add_service(status_svc)
|
||||
.add_service(reflection_v1)
|
||||
.add_service(reflection_v1alpha)
|
||||
.add_service(control_svc)
|
||||
.serve(addr)
|
||||
.await
|
||||
{
|
||||
tracing::error!(error = %e, "health server ended with error");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
struct ClientStatusServiceImpl {
|
||||
status: ServerStatus,
|
||||
files: Arc<Mutex<BTreeMap<INodeNo, crate::item::Item>>>,
|
||||
}
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl ClientStatus for ClientStatusServiceImpl {
|
||||
async fn get_client_status(
|
||||
&self,
|
||||
_request: Request<GetClientStatusRequest>,
|
||||
) -> Result<Response<ClientStatusResponse>, tonic::Status> {
|
||||
let file_count = self
|
||||
.files
|
||||
.lock()
|
||||
.unwrap()
|
||||
.values()
|
||||
.filter(|i| i.file_type == crate::item::FileType::File)
|
||||
.count() as u64;
|
||||
|
||||
let last_reconcile_unix = self.status.last_reconcile.lock().unwrap().unwrap_or(0);
|
||||
|
||||
Ok(Response::new(ClientStatusResponse {
|
||||
origin_type: self.status.origin_type.clone(),
|
||||
mountpoint: self.status.mountpoint.clone(),
|
||||
file_count,
|
||||
server_connected: self.status.is_server_connected(),
|
||||
server_endpoint: self.status.server_endpoint.clone(),
|
||||
last_reconcile_unix,
|
||||
}))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
use std::{
|
||||
path::PathBuf,
|
||||
time::{SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
use std::os::unix::ffi::OsStrExt;
|
||||
|
||||
use fuser::INodeNo;
|
||||
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
|
||||
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
|
||||
pub enum FileType {
|
||||
Directory,
|
||||
File,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Item {
|
||||
pub inode: INodeNo,
|
||||
pub parent_inode: INodeNo,
|
||||
pub name: String,
|
||||
pub original_path: PathBuf,
|
||||
pub local_path: PathBuf,
|
||||
pub file_type: FileType,
|
||||
pub attrs: FileAttrs,
|
||||
pub music_metadata: Option<MusicMetadata>,
|
||||
pub hash: u64,
|
||||
}
|
||||
|
||||
fn secs_since_epoch(time: SystemTime) -> u64 {
|
||||
return time
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0);
|
||||
}
|
||||
|
||||
impl Item {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
inode: INodeNo,
|
||||
parent_inode: INodeNo,
|
||||
name: String,
|
||||
original_path: PathBuf,
|
||||
local_path: PathBuf,
|
||||
file_type: FileType,
|
||||
attrs: FileAttrs,
|
||||
music_metadata: Option<MusicMetadata>,
|
||||
) -> Item {
|
||||
let mut item = Item {
|
||||
inode,
|
||||
parent_inode,
|
||||
name,
|
||||
original_path,
|
||||
local_path,
|
||||
file_type,
|
||||
attrs,
|
||||
music_metadata,
|
||||
hash: 0,
|
||||
};
|
||||
item.hash = item.compute_hash();
|
||||
|
||||
return item;
|
||||
}
|
||||
|
||||
pub fn compute_hash(&self) -> u64 {
|
||||
musicfs_core::compute_item_hash(
|
||||
self.inode.0,
|
||||
self.original_path.as_os_str().as_bytes(),
|
||||
secs_since_epoch(self.attrs.ctime),
|
||||
secs_since_epoch(self.attrs.mtime),
|
||||
secs_since_epoch(self.attrs.crtime),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
|
||||
#[test]
|
||||
fn compute_hash_deterministic() {
|
||||
let attrs = file_attrs_for_tempdir();
|
||||
|
||||
let item1 = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
PathBuf::from("/some/path"),
|
||||
PathBuf::from("test"),
|
||||
FileType::File,
|
||||
attrs.clone(),
|
||||
None,
|
||||
);
|
||||
|
||||
let item2 = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
PathBuf::from("/some/path"),
|
||||
PathBuf::from("test"),
|
||||
FileType::File,
|
||||
attrs,
|
||||
None,
|
||||
);
|
||||
|
||||
assert_eq!(item1.hash, item2.hash);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_hash_changes_on_path_change() {
|
||||
let attrs = file_attrs_for_tempdir();
|
||||
|
||||
let item1 = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
PathBuf::from("/some/path"),
|
||||
PathBuf::from("test"),
|
||||
FileType::File,
|
||||
attrs.clone(),
|
||||
None,
|
||||
);
|
||||
|
||||
let item2 = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
PathBuf::from("/different/path"),
|
||||
PathBuf::from("test"),
|
||||
FileType::File,
|
||||
attrs,
|
||||
None,
|
||||
);
|
||||
|
||||
assert_ne!(item1.hash, item2.hash);
|
||||
}
|
||||
|
||||
fn file_attrs_for_tempdir() -> FileAttrs {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = std::fs::metadata(tmp.path()).unwrap();
|
||||
return FileAttrs::from(&metadata);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
pub mod control;
|
||||
pub mod db;
|
||||
pub mod health;
|
||||
pub mod item;
|
||||
pub mod music;
|
||||
pub mod origins;
|
||||
pub mod virtual_dirs;
|
||||
|
||||
pub use musicfs_core::logging;
|
||||
pub use musicfs_proto as proto;
|
||||
@@ -0,0 +1,170 @@
|
||||
use std::{collections::HashMap, net::SocketAddr, path::Path, sync::Arc};
|
||||
|
||||
use clap::Parser;
|
||||
use fuser::INodeNo;
|
||||
use musicfs::db::entities::{Entity, Model};
|
||||
use musicfs::db::sync::sync_items_to_db;
|
||||
use musicfs::health::ServerStatus;
|
||||
use musicfs::item::Item;
|
||||
use musicfs::logging::{LogConfig, init};
|
||||
use musicfs::music::db::restore_music_metadata_from_db;
|
||||
use musicfs::origins::local::LocalOrigin;
|
||||
use musicfs::origins::network::NetworkOrigin;
|
||||
use musicfs::origins::{FuseFs, Origin};
|
||||
use musicfs::virtual_dirs::restore_virtual_paths;
|
||||
use sea_orm::entity::prelude::*;
|
||||
use tokio::signal::unix::{SignalKind, signal};
|
||||
use tracing::{debug, error, info};
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(version, about, long_about = None)]
|
||||
struct Args {
|
||||
#[arg(short, long, required = true)]
|
||||
mountpoint: String,
|
||||
|
||||
/// Local directory path (→ LocalOrigin) OR `http://host:port` URL of a
|
||||
/// musicfs-server (→ NetworkOrigin).
|
||||
#[arg(short, long, required = true)]
|
||||
source: String,
|
||||
|
||||
#[arg(short, long, required = true)]
|
||||
database: String,
|
||||
|
||||
/// Directory for daily-rotated log files.
|
||||
#[arg(long, default_value = "./logs")]
|
||||
log_dir: std::path::PathBuf,
|
||||
|
||||
/// gRPC server address for health, status, and control RPCs.
|
||||
#[arg(long, default_value = "127.0.0.1:50052")]
|
||||
listen: SocketAddr,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let args = Args::parse();
|
||||
|
||||
// Bind the guard for the whole process so the non-blocking file writer
|
||||
// flushes on exit. Initialized before anything else so even early
|
||||
// failures land in the log.
|
||||
let _guard = init(LogConfig {
|
||||
log_dir: args.log_dir.clone(),
|
||||
file_prefix: "musicfs".to_string(),
|
||||
max_files: 7,
|
||||
});
|
||||
|
||||
let mountpoint = args.mountpoint.clone();
|
||||
info!(
|
||||
mountpoint = %mountpoint,
|
||||
source = %args.source,
|
||||
database = %args.database,
|
||||
log_dir = %args.log_dir.display(),
|
||||
"musicfs starting"
|
||||
);
|
||||
|
||||
// sea-orm logs every statement at INFO by default, which floods normal
|
||||
// output; demote to DEBUG so queries stay hidden under RUST_LOG=info.
|
||||
let mut db_opts = sea_orm::ConnectOptions::new(args.database.clone());
|
||||
db_opts.sqlx_logging_level(log::LevelFilter::Debug);
|
||||
let db = sea_orm::Database::connect(db_opts)
|
||||
.await
|
||||
.unwrap_or_else(|e| {
|
||||
error!(database = %args.database, error = %e, "database connect failed");
|
||||
panic!("database connect: {e}");
|
||||
});
|
||||
info!(database = %args.database, "database connected");
|
||||
|
||||
let (snapshot, byte_source, watcher, server_status) = if looks_like_url(&args.source) {
|
||||
debug!(source = %args.source, "using NetworkOrigin");
|
||||
let origin = NetworkOrigin::new(args.source.clone(), mountpoint.clone(), db.clone())
|
||||
.unwrap_or_else(|e| {
|
||||
error!(source = %args.source, error = %e, "network origin init failed");
|
||||
panic!("network origin init: {e}");
|
||||
});
|
||||
let server_status = origin.server_status();
|
||||
let snapshot = origin.snapshot_async().await.unwrap_or_else(|e| {
|
||||
error!(source = %args.source, error = %e, "network initial snapshot failed");
|
||||
panic!("network initial snapshot: {e}");
|
||||
});
|
||||
info!(files = snapshot.len(), "network snapshot complete");
|
||||
(
|
||||
snapshot,
|
||||
origin.byte_source(),
|
||||
origin.watcher(),
|
||||
server_status,
|
||||
)
|
||||
} else {
|
||||
debug!(source = %args.source, "using LocalOrigin");
|
||||
let origin = LocalOrigin::new(args.source.clone(), mountpoint.clone());
|
||||
let mut snapshot = origin.snapshot().unwrap_or_else(|e| {
|
||||
error!(source = %args.source, error = %e, "local initial snapshot failed");
|
||||
panic!("local initial snapshot: {e}");
|
||||
});
|
||||
|
||||
let db_items: HashMap<i64, Model> = Entity::find()
|
||||
.all(&db)
|
||||
.await
|
||||
.unwrap_or_else(|e| {
|
||||
error!(error = %e, "loading db items failed");
|
||||
panic!("loading db items: {e}");
|
||||
})
|
||||
.into_iter()
|
||||
.map(|e| (e.inode, e))
|
||||
.collect();
|
||||
sync_items_to_db(&snapshot, &db_items, &db).await;
|
||||
restore_music_metadata_from_db(&mut snapshot, &db_items, &db).await;
|
||||
restore_virtual_paths(&mut snapshot, &db_items, Path::new(&args.source));
|
||||
|
||||
info!(files = snapshot.len(), "local snapshot complete");
|
||||
let server_status = ServerStatus::new_local(mountpoint.clone());
|
||||
(
|
||||
snapshot,
|
||||
origin.byte_source(),
|
||||
origin.watcher(),
|
||||
server_status,
|
||||
)
|
||||
};
|
||||
|
||||
let files: Arc<std::sync::Mutex<std::collections::BTreeMap<INodeNo, Item>>> =
|
||||
Arc::new(std::sync::Mutex::new(snapshot));
|
||||
let watcher_handle = watcher.watch(files.clone());
|
||||
let health_files = files.clone();
|
||||
let grpc_db = db.clone();
|
||||
|
||||
let fs = FuseFs {
|
||||
files,
|
||||
bytes: byte_source,
|
||||
client: db,
|
||||
runtime_handle: tokio::runtime::Handle::current(),
|
||||
};
|
||||
|
||||
let cfg = fuser::Config::default();
|
||||
let session = fuser::spawn_mount2(fs, &mountpoint, &cfg).unwrap_or_else(|e| {
|
||||
error!(mountpoint = %mountpoint, error = %e, "failed to mount FUSE filesystem");
|
||||
panic!("failed to mount FUSE filesystem: {e}");
|
||||
});
|
||||
info!(mountpoint = %mountpoint, "FUSE mounted");
|
||||
|
||||
musicfs::health::spawn_grpc_server(args.listen, server_status, health_files, grpc_db).await;
|
||||
|
||||
let mut sigint = signal(SignalKind::interrupt()).unwrap_or_else(|e| {
|
||||
error!(error = %e, "failed to register SIGINT handler");
|
||||
panic!("register SIGINT handler: {e}");
|
||||
});
|
||||
let mut sigterm = signal(SignalKind::terminate()).unwrap_or_else(|e| {
|
||||
error!(error = %e, "failed to register SIGTERM handler");
|
||||
panic!("register SIGTERM handler: {e}");
|
||||
});
|
||||
|
||||
tokio::select! {
|
||||
_ = sigint.recv() => info!("received SIGINT, shutting down"),
|
||||
_ = sigterm.recv() => info!("received SIGTERM, shutting down"),
|
||||
}
|
||||
|
||||
info!(mountpoint = %mountpoint, "unmounting");
|
||||
watcher_handle.stop();
|
||||
drop(session);
|
||||
}
|
||||
|
||||
fn looks_like_url(s: &str) -> bool {
|
||||
return s.starts_with("http://") || s.starts_with("https://");
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
use crate::db::entities::Model;
|
||||
use crate::item::Item;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use tracing::{debug, error};
|
||||
|
||||
pub mod entities {
|
||||
pub mod music_metadata {
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "music_metadata")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
pub track_title: String,
|
||||
pub album: String,
|
||||
pub track_number: i32,
|
||||
#[sea_orm(column_type = "Blob")]
|
||||
pub header: Vec<u8>,
|
||||
pub real_audio_start: i64,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
}
|
||||
|
||||
pub mod artists {
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "music_metadata_artists")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub artist: String,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
}
|
||||
|
||||
pub mod other_tags {
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "music_metadata_other_tags")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub position: i32,
|
||||
pub tag: String,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
}
|
||||
|
||||
pub mod pictures {
|
||||
use sea_orm::entity::prelude::*;
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)]
|
||||
#[sea_orm(table_name = "music_metadata_pictures")]
|
||||
pub struct Model {
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub inode: i64,
|
||||
#[sea_orm(primary_key, auto_increment = false)]
|
||||
pub position: i32,
|
||||
#[sea_orm(column_type = "Blob")]
|
||||
pub block_header: Vec<u8>,
|
||||
pub data_offset: i64,
|
||||
pub data_length: i64,
|
||||
}
|
||||
|
||||
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
|
||||
pub enum Relation {}
|
||||
|
||||
impl ActiveModelBehavior for ActiveModel {}
|
||||
}
|
||||
}
|
||||
|
||||
use entities::{artists, music_metadata as music_metadata_entity, other_tags, pictures};
|
||||
|
||||
pub async fn save_music_metadata(
|
||||
inode: i64,
|
||||
music_metadata: &MusicMetadata,
|
||||
client: &sea_orm::DatabaseConnection,
|
||||
) -> Result<(), sea_orm::DbErr> {
|
||||
use sea_orm::ActiveValue::Set;
|
||||
|
||||
music_metadata_entity::Entity::delete_by_id(inode)
|
||||
.exec(client)
|
||||
.await?;
|
||||
|
||||
music_metadata_entity::Entity::insert(music_metadata_entity::ActiveModel {
|
||||
inode: Set(inode),
|
||||
track_title: Set(music_metadata.track_title.clone()),
|
||||
album: Set(music_metadata.album.clone()),
|
||||
track_number: Set(music_metadata.track_number),
|
||||
header: Set(music_metadata.header.clone()),
|
||||
real_audio_start: Set(music_metadata.real_audio_start as i64),
|
||||
})
|
||||
.exec(client)
|
||||
.await?;
|
||||
|
||||
if !music_metadata.artist.is_empty() {
|
||||
artists::Entity::insert_many(music_metadata.artist.iter().map(|a| artists::ActiveModel {
|
||||
inode: Set(inode),
|
||||
artist: Set(a.clone()),
|
||||
}))
|
||||
.exec(client)
|
||||
.await?;
|
||||
}
|
||||
|
||||
if !music_metadata.other_tags.is_empty() {
|
||||
other_tags::Entity::insert_many(music_metadata.other_tags.iter().enumerate().map(
|
||||
|(pos, tag)| other_tags::ActiveModel {
|
||||
inode: Set(inode),
|
||||
position: Set(pos as i32),
|
||||
tag: Set(tag.clone()),
|
||||
},
|
||||
))
|
||||
.exec(client)
|
||||
.await?;
|
||||
}
|
||||
|
||||
if !music_metadata.picture_block_headers.is_empty() {
|
||||
pictures::Entity::insert_many(
|
||||
music_metadata
|
||||
.picture_block_headers
|
||||
.iter()
|
||||
.zip(music_metadata.picture_data_ranges.iter())
|
||||
.enumerate()
|
||||
.map(|(pos, (hdr, (offset, len)))| pictures::ActiveModel {
|
||||
inode: Set(inode),
|
||||
position: Set(pos as i32),
|
||||
block_header: Set(hdr.clone()),
|
||||
data_offset: Set(*offset as i64),
|
||||
data_length: Set(*len as i64),
|
||||
}),
|
||||
)
|
||||
.exec(client)
|
||||
.await?;
|
||||
}
|
||||
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
pub async fn restore_music_metadata_from_db(
|
||||
snapshot: &mut BTreeMap<INodeNo, Item>,
|
||||
db_items: &HashMap<i64, Model>,
|
||||
client: &sea_orm::DatabaseConnection,
|
||||
) {
|
||||
use artists::Column as ArtCol;
|
||||
use music_metadata_entity::Column as MmCol;
|
||||
use other_tags::Column as OtCol;
|
||||
use pictures::Column as PicCol;
|
||||
|
||||
let unchanged_music_inodes: Vec<i64> = db_items
|
||||
.values()
|
||||
.filter_map(|db_item| {
|
||||
let ino = INodeNo(db_item.inode as u64);
|
||||
snapshot
|
||||
.get(&ino)
|
||||
.filter(|item| item.hash as i64 == db_item.hash && item.music_metadata.is_some())
|
||||
.map(|_| db_item.inode)
|
||||
})
|
||||
.collect();
|
||||
|
||||
if unchanged_music_inodes.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let mm_rows: HashMap<i64, music_metadata_entity::Model> =
|
||||
match music_metadata_entity::Entity::find()
|
||||
.filter(MmCol::Inode.is_in(unchanged_music_inodes.clone()))
|
||||
.all(client)
|
||||
.await
|
||||
{
|
||||
Ok(rows) => rows.into_iter().map(|m| (m.inode, m)).collect(),
|
||||
Err(e) => {
|
||||
error!(error = %e, "restore_music_metadata: mm rows query failed");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let mut artists_by_inode: HashMap<i64, Vec<String>> = HashMap::new();
|
||||
let artist_rows = match artists::Entity::find()
|
||||
.filter(ArtCol::Inode.is_in(unchanged_music_inodes.clone()))
|
||||
.all(client)
|
||||
.await
|
||||
{
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
error!(error = %e, "restore_music_metadata: artists query failed");
|
||||
vec![]
|
||||
}
|
||||
};
|
||||
for row in artist_rows {
|
||||
artists_by_inode
|
||||
.entry(row.inode)
|
||||
.or_default()
|
||||
.push(row.artist);
|
||||
}
|
||||
|
||||
let mut other_tags_by_inode: HashMap<i64, Vec<(i32, String)>> = HashMap::new();
|
||||
let other_tag_rows = match other_tags::Entity::find()
|
||||
.filter(OtCol::Inode.is_in(unchanged_music_inodes.clone()))
|
||||
.all(client)
|
||||
.await
|
||||
{
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
error!(error = %e, "restore_music_metadata: other_tags query failed");
|
||||
vec![]
|
||||
}
|
||||
};
|
||||
for row in other_tag_rows {
|
||||
other_tags_by_inode
|
||||
.entry(row.inode)
|
||||
.or_default()
|
||||
.push((row.position, row.tag));
|
||||
}
|
||||
|
||||
let mut pictures_by_inode: HashMap<i64, Vec<pictures::Model>> = HashMap::new();
|
||||
let picture_rows = match pictures::Entity::find()
|
||||
.filter(PicCol::Inode.is_in(unchanged_music_inodes))
|
||||
.all(client)
|
||||
.await
|
||||
{
|
||||
Ok(r) => r,
|
||||
Err(e) => {
|
||||
error!(error = %e, "restore_music_metadata: pictures query failed");
|
||||
vec![]
|
||||
}
|
||||
};
|
||||
for row in picture_rows {
|
||||
pictures_by_inode.entry(row.inode).or_default().push(row);
|
||||
}
|
||||
|
||||
debug!(restored = mm_rows.len(), "restored music metadata from db");
|
||||
|
||||
for (inode, mm_row) in mm_rows {
|
||||
let ino = INodeNo(inode as u64);
|
||||
let Some(item) = snapshot.get_mut(&ino) else {
|
||||
continue;
|
||||
};
|
||||
|
||||
let mut sorted_tags = other_tags_by_inode.remove(&inode).unwrap_or_default();
|
||||
sorted_tags.sort_by_key(|(pos, _)| *pos);
|
||||
|
||||
let mut sorted_pics = pictures_by_inode.remove(&inode).unwrap_or_default();
|
||||
sorted_pics.sort_by_key(|p| p.position);
|
||||
|
||||
let picture_block_headers: Vec<Vec<u8>> =
|
||||
sorted_pics.iter().map(|p| p.block_header.clone()).collect();
|
||||
|
||||
let picture_data_ranges: Vec<(u64, u64)> = sorted_pics
|
||||
.iter()
|
||||
.map(|p| (p.data_offset as u64, p.data_length as u64))
|
||||
.collect();
|
||||
|
||||
let mut music_metadata = MusicMetadata {
|
||||
artist: artists_by_inode.remove(&inode).unwrap_or_default(),
|
||||
album_artist: None,
|
||||
album: mm_row.album,
|
||||
track_number: mm_row.track_number,
|
||||
track_title: mm_row.track_title,
|
||||
other_tags: sorted_tags.into_iter().map(|(_, tag)| tag).collect(),
|
||||
header: mm_row.header,
|
||||
picture_block_headers,
|
||||
picture_data_ranges,
|
||||
real_audio_start: mm_row.real_audio_start as u64,
|
||||
vorbis_comment_offset: 0,
|
||||
vorbis_comment_length: 0,
|
||||
};
|
||||
music_metadata.find_vorbis_offsets();
|
||||
item.music_metadata = Some(music_metadata);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
pub mod db;
|
||||
pub use musicfs_core::music::{metadata, parse};
|
||||
@@ -0,0 +1,215 @@
|
||||
use std::{
|
||||
fs,
|
||||
io::{self, Read, Seek, SeekFrom},
|
||||
path::Path,
|
||||
};
|
||||
|
||||
pub fn read_bytes_at(path: &Path, offset: u64, len: usize) -> io::Result<Vec<u8>> {
|
||||
let mut f = fs::File::open(path)?;
|
||||
f.seek(SeekFrom::Start(offset))?;
|
||||
let mut buf = vec![0u8; len];
|
||||
let n = f.read(&mut buf)?;
|
||||
buf.truncate(n);
|
||||
|
||||
return Ok(buf);
|
||||
}
|
||||
|
||||
fn assemble_virtual_read(
|
||||
reader: &dyn Fn(u64, usize) -> io::Result<Vec<u8>>,
|
||||
header: &[u8],
|
||||
pic_hdrs: &[Vec<u8>],
|
||||
pic_ranges: &[(u64, u64)],
|
||||
real_audio_start: u64,
|
||||
offset: u64,
|
||||
size: u32,
|
||||
) -> io::Result<Vec<u8>> {
|
||||
let end = offset + size as u64;
|
||||
let header_end = header.len() as u64;
|
||||
|
||||
if end <= header_end {
|
||||
return Ok(header[offset as usize..end as usize].to_vec());
|
||||
}
|
||||
|
||||
let mut buf = Vec::with_capacity(size as usize);
|
||||
if offset < header_end {
|
||||
buf.extend_from_slice(&header[offset as usize..]);
|
||||
}
|
||||
|
||||
let mut virt_pos = header_end;
|
||||
for (pic_hdr, (data_real_offset, data_len)) in pic_hdrs.iter().zip(pic_ranges.iter()) {
|
||||
let pic_hdr_len = pic_hdr.len() as u64;
|
||||
let pic_hdr_end = virt_pos + pic_hdr_len;
|
||||
let pic_end = pic_hdr_end + data_len;
|
||||
|
||||
if end <= virt_pos {
|
||||
break;
|
||||
}
|
||||
if offset >= pic_end {
|
||||
virt_pos = pic_end;
|
||||
continue;
|
||||
}
|
||||
|
||||
let hdr_from = (offset.max(virt_pos) - virt_pos) as usize;
|
||||
let hdr_to = ((end.min(pic_hdr_end)) - virt_pos) as usize;
|
||||
if hdr_from < hdr_to {
|
||||
buf.extend_from_slice(&pic_hdr[hdr_from..hdr_to.min(pic_hdr.len())]);
|
||||
}
|
||||
|
||||
let data_start = offset.max(pic_hdr_end);
|
||||
let data_end = end.min(pic_end);
|
||||
if data_start < data_end {
|
||||
let real_off = data_real_offset + (data_start - pic_hdr_end);
|
||||
let bytes = reader(real_off, (data_end - data_start) as usize)?;
|
||||
buf.extend_from_slice(&bytes);
|
||||
}
|
||||
|
||||
virt_pos = pic_end;
|
||||
}
|
||||
|
||||
let audio_start = offset.max(virt_pos);
|
||||
if audio_start < end {
|
||||
let real_off = real_audio_start + (audio_start - virt_pos);
|
||||
let bytes = reader(real_off, (end - audio_start) as usize)?;
|
||||
buf.extend_from_slice(&bytes);
|
||||
}
|
||||
|
||||
return Ok(buf);
|
||||
}
|
||||
|
||||
pub fn assemble_flac_read(
|
||||
reader: &dyn Fn(u64, usize) -> io::Result<Vec<u8>>,
|
||||
header: &[u8],
|
||||
pic_hdrs: &[Vec<u8>],
|
||||
pic_ranges: &[(u64, u64)],
|
||||
real_audio_start: u64,
|
||||
offset: u64,
|
||||
size: u32,
|
||||
) -> io::Result<Vec<u8>> {
|
||||
assemble_virtual_read(
|
||||
reader,
|
||||
header,
|
||||
pic_hdrs,
|
||||
pic_ranges,
|
||||
real_audio_start,
|
||||
offset,
|
||||
size,
|
||||
)
|
||||
}
|
||||
|
||||
pub fn assemble_mp3_read(
|
||||
reader: &dyn Fn(u64, usize) -> io::Result<Vec<u8>>,
|
||||
header: &[u8],
|
||||
pic_hdrs: &[Vec<u8>],
|
||||
pic_ranges: &[(u64, u64)],
|
||||
real_audio_start: u64,
|
||||
offset: u64,
|
||||
size: u32,
|
||||
) -> io::Result<Vec<u8>> {
|
||||
assemble_virtual_read(
|
||||
reader,
|
||||
header,
|
||||
pic_hdrs,
|
||||
pic_ranges,
|
||||
real_audio_start,
|
||||
offset,
|
||||
size,
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::Write;
|
||||
use std::path::PathBuf;
|
||||
|
||||
fn file_reader(path: PathBuf) -> impl Fn(u64, usize) -> io::Result<Vec<u8>> {
|
||||
return move |offset, len| read_bytes_at(&path, offset, len);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_bytes_at_full_file() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
f.write_all(b"hello world").unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path();
|
||||
|
||||
let result = read_bytes_at(path, 0, 11).unwrap();
|
||||
assert_eq!(result, b"hello world");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_bytes_at_offset() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
f.write_all(b"abcdefghij").unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path();
|
||||
|
||||
let result = read_bytes_at(path, 3, 4).unwrap();
|
||||
assert_eq!(result, b"defg");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_header_only() {
|
||||
let header = b"HEADERDATA";
|
||||
let noop = |_, _| Ok(vec![]);
|
||||
let result = assemble_flac_read(&noop, header, &[], &[], 0, 2, 4).unwrap();
|
||||
assert_eq!(result, b"ADER");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_audio_region() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let content: Vec<u8> = (0..20).collect();
|
||||
f.write_all(&content).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let reader = file_reader(path);
|
||||
let header = b"HDR";
|
||||
let result = assemble_flac_read(&reader, header, &[], &[], 10, 5, 4).unwrap();
|
||||
// offset=5, header_len=3, so we read from header[3..] (0 bytes) + audio at real_off=10+(5-3)=12
|
||||
// content[12..16] = [12, 13, 14, 15]
|
||||
assert_eq!(result, vec![12, 13, 14, 15]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_header_audio_boundary() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let content: Vec<u8> = (0..110).collect();
|
||||
f.write_all(&content).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let reader = file_reader(path);
|
||||
let header = b"ABCD";
|
||||
let result = assemble_flac_read(&reader, header, &[], &[], 100, 2, 6).unwrap();
|
||||
// offset=2, size=6, header_len=4
|
||||
// First 2 bytes from header[2..4] = "CD"
|
||||
// Remaining 4 bytes from audio at real_off=100+(2+2-4)=100
|
||||
// content[100..104] = [100, 101, 102, 103]
|
||||
let expected = [b'C', b'D', 100, 101, 102, 103];
|
||||
assert_eq!(result, expected);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_mp3_read_header_only() {
|
||||
let header = b"ID3\x04\x00DATAHERE";
|
||||
let noop = |_, _| Ok(vec![]);
|
||||
let result = assemble_mp3_read(&noop, header, &[], &[], 0, 4, 4).unwrap();
|
||||
assert_eq!(result, &header[4..8]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_mp3_read_audio_region() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let content: Vec<u8> = (0..20).collect();
|
||||
f.write_all(&content).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let reader = file_reader(path);
|
||||
let header = b"ID3";
|
||||
let result = assemble_mp3_read(&reader, header, &[], &[], 10, 5, 4).unwrap();
|
||||
assert_eq!(result, vec![12, 13, 14, 15]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
pub mod file_io;
|
||||
pub mod snapshot;
|
||||
pub mod watcher;
|
||||
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io,
|
||||
path::{Path, PathBuf},
|
||||
sync::Arc,
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
|
||||
use crate::item::Item;
|
||||
use crate::origins::{ByteSource, FileWatcher, Origin};
|
||||
pub struct LocalOrigin {
|
||||
pub(crate) source: PathBuf,
|
||||
pub(crate) destination: PathBuf,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for LocalOrigin {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
return f
|
||||
.debug_struct("LocalOrigin")
|
||||
.field("source", &self.source)
|
||||
.field("destination", &self.destination)
|
||||
.finish();
|
||||
}
|
||||
}
|
||||
|
||||
impl LocalOrigin {
|
||||
pub fn new(source: String, destination: String) -> LocalOrigin {
|
||||
return LocalOrigin {
|
||||
source: source.into(),
|
||||
destination: destination.into(),
|
||||
};
|
||||
}
|
||||
|
||||
pub fn source(&self) -> &Path {
|
||||
return &self.source;
|
||||
}
|
||||
}
|
||||
|
||||
impl Origin for LocalOrigin {
|
||||
fn snapshot(&self) -> io::Result<BTreeMap<INodeNo, Item>> {
|
||||
return snapshot::build_snapshot(&self.source, &self.destination);
|
||||
}
|
||||
|
||||
fn byte_source(&self) -> Arc<dyn ByteSource> {
|
||||
return Arc::new(LocalByteSource);
|
||||
}
|
||||
|
||||
fn watcher(&self) -> Box<dyn FileWatcher> {
|
||||
return Box::new(watcher::LocalOriginFileWatcher::new(
|
||||
self.source.clone(),
|
||||
self.destination.clone(),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
pub struct LocalByteSource;
|
||||
|
||||
impl ByteSource for LocalByteSource {
|
||||
fn read_at(
|
||||
&self,
|
||||
_inode: INodeNo,
|
||||
locator: &Path,
|
||||
offset: u64,
|
||||
len: usize,
|
||||
) -> io::Result<Vec<u8>> {
|
||||
return file_io::read_bytes_at(locator, offset, len);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
fs, io,
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, Mutex},
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::music::parse::parse_music_metadata_for_path;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
use crate::virtual_dirs::ensure_virtual_dirs;
|
||||
use tracing::{error, info};
|
||||
|
||||
pub fn fill_fileset(map: &Arc<Mutex<BTreeMap<INodeNo, Item>>>, source: &Path, destination: &Path) {
|
||||
match build_snapshot(source, destination) {
|
||||
Ok(new_snapshot) => {
|
||||
let count = new_snapshot.len();
|
||||
let mut files = map.lock().unwrap();
|
||||
|
||||
files.retain(|ino, _| new_snapshot.contains_key(ino));
|
||||
|
||||
for (ino, new_item) in new_snapshot {
|
||||
match files.get(&ino) {
|
||||
Some(existing) if existing.hash == new_item.hash => {}
|
||||
_ => {
|
||||
files.insert(ino, new_item);
|
||||
}
|
||||
}
|
||||
}
|
||||
info!(count, "snapshot rebuilt");
|
||||
}
|
||||
Err(e) => error!(source = %source.display(), error = %e, "error while reading source"),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn build_snapshot(
|
||||
source: &Path,
|
||||
destination: &Path,
|
||||
) -> Result<BTreeMap<INodeNo, Item>, io::Error> {
|
||||
let mut map = BTreeMap::new();
|
||||
|
||||
let local_root = Item::new(
|
||||
INodeNo::ROOT,
|
||||
INodeNo::ROOT,
|
||||
"/".to_string(),
|
||||
source.to_path_buf(),
|
||||
destination.to_path_buf(),
|
||||
FileType::Directory,
|
||||
FileAttrs::from(&fs::metadata(source)?),
|
||||
None,
|
||||
);
|
||||
map.insert(INodeNo::ROOT, local_root);
|
||||
|
||||
read_into_map(source, destination, &mut map)?;
|
||||
return Ok(map);
|
||||
}
|
||||
|
||||
pub fn read_into_map(
|
||||
source: &Path,
|
||||
destination: &Path,
|
||||
map: &mut BTreeMap<INodeNo, Item>,
|
||||
) -> Result<(), io::Error> {
|
||||
for item in fs::read_dir(source)? {
|
||||
let entry = item?;
|
||||
let item_path = entry.path();
|
||||
|
||||
if entry.file_type()?.is_dir() {
|
||||
read_into_map(&item_path, destination, map)?;
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
let metadata = entry.metadata()?;
|
||||
let music_metadata = parse_music_metadata_for_path(&item_path);
|
||||
|
||||
let mut local_path = PathBuf::new();
|
||||
if let Some(ref mm) = music_metadata {
|
||||
let joined;
|
||||
let artist_dir = match mm.album_artist.as_deref() {
|
||||
Some(a) => a,
|
||||
None => {
|
||||
joined = mm.artist.join("-");
|
||||
&joined
|
||||
}
|
||||
};
|
||||
local_path.push(artist_dir);
|
||||
local_path.push(&mm.album);
|
||||
}
|
||||
local_path.push(&name);
|
||||
|
||||
let inode = INodeNo(metadata.ino());
|
||||
let parent_inode = ensure_virtual_dirs(&local_path, source, map);
|
||||
let local_item = Item::new(
|
||||
inode,
|
||||
parent_inode,
|
||||
name,
|
||||
item_path,
|
||||
local_path,
|
||||
FileType::File,
|
||||
FileAttrs::from(&metadata),
|
||||
music_metadata,
|
||||
);
|
||||
|
||||
map.insert(inode, local_item);
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
path::PathBuf,
|
||||
sync::{Arc, Mutex, mpsc},
|
||||
thread,
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use notify::{Event, EventKind, RecursiveMode, Watcher};
|
||||
|
||||
use crate::item::Item;
|
||||
use crate::origins::{FileWatcher, WatcherHandle};
|
||||
use tracing::{debug, error, info, trace};
|
||||
|
||||
pub struct LocalOriginFileWatcher {
|
||||
source: PathBuf,
|
||||
destination: PathBuf,
|
||||
}
|
||||
|
||||
impl LocalOriginFileWatcher {
|
||||
pub fn new(source: PathBuf, destination: PathBuf) -> Self {
|
||||
return LocalOriginFileWatcher {
|
||||
source,
|
||||
destination,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
impl FileWatcher for LocalOriginFileWatcher {
|
||||
fn watch(&self, files: Arc<Mutex<BTreeMap<INodeNo, Item>>>) -> WatcherHandle {
|
||||
let source = self.source.clone();
|
||||
let destination = self.destination.clone();
|
||||
|
||||
info!(source = %source.display(), "starting file watcher");
|
||||
thread::spawn(move || {
|
||||
let (tx, rx): (
|
||||
mpsc::Sender<Result<Event, notify::Error>>,
|
||||
mpsc::Receiver<Result<Event, notify::Error>>,
|
||||
) = mpsc::channel();
|
||||
|
||||
let mut watcher: notify::INotifyWatcher = match notify::recommended_watcher(tx) {
|
||||
Ok(w) => w,
|
||||
Err(e) => {
|
||||
error!(error = %e, "watcher: failed to create notify watcher");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
if let Err(e) = watcher.watch(&source, RecursiveMode::Recursive) {
|
||||
error!(source = %source.display(), error = %e, "watcher: failed to watch source");
|
||||
return;
|
||||
}
|
||||
for res in rx {
|
||||
match res {
|
||||
Ok(event) => {
|
||||
trace!(?event, "watcher event");
|
||||
|
||||
match event.kind {
|
||||
EventKind::Any => {
|
||||
debug!("watcher: EventKind::Any, ignoring");
|
||||
}
|
||||
EventKind::Access(_access_kind) => {
|
||||
debug!("watcher: item accessed");
|
||||
}
|
||||
EventKind::Create(_) | EventKind::Modify(_) | EventKind::Remove(_) => {
|
||||
debug!("watcher: create/modify/remove; rebuilding snapshot");
|
||||
super::snapshot::fill_fileset(&files, &source, &destination);
|
||||
}
|
||||
EventKind::Other => {
|
||||
debug!("watcher: other event, ignoring");
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => error!(error = %e, "watcher: notify error"),
|
||||
}
|
||||
}
|
||||
});
|
||||
return WatcherHandle::detached();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,631 @@
|
||||
pub use musicfs_core::attrs;
|
||||
pub mod local;
|
||||
pub mod network;
|
||||
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io,
|
||||
path::Path,
|
||||
sync::{Arc, Mutex},
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use fuser::{Errno, FileAttr, Filesystem, Generation, INodeNo, Request};
|
||||
use sea_orm::{ActiveModelTrait, DatabaseConnection};
|
||||
use tracing::{debug, error, trace};
|
||||
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::music::db::save_music_metadata;
|
||||
use crate::origins::local::file_io;
|
||||
use crate::virtual_dirs::parent_inode_from_path;
|
||||
|
||||
/// Transport-agnostic byte-range reader. `locator` is whatever string the
|
||||
/// origin treats as a file key: a filesystem path for `LocalOrigin`, a remote
|
||||
/// key for the future network origin. Both produce bytes at `(offset, len)`.
|
||||
///
|
||||
/// `inode` is the FUSE-level inode; NetworkOrigin uses it as the server-side
|
||||
/// file id (and as the Postgres cache key). LocalOrigin ignores it.
|
||||
pub trait ByteSource: Send + Sync {
|
||||
fn read_at(
|
||||
&self,
|
||||
inode: INodeNo,
|
||||
locator: &Path,
|
||||
offset: u64,
|
||||
len: usize,
|
||||
) -> io::Result<Vec<u8>>;
|
||||
}
|
||||
|
||||
pub trait FileWatcher: Send + Sync {
|
||||
fn watch(&self, files: Arc<Mutex<BTreeMap<INodeNo, Item>>>) -> WatcherHandle;
|
||||
}
|
||||
|
||||
/// Returned by [`FileWatcher::watch`] so the caller can stop a background
|
||||
/// watcher before the process (and its tokio runtime) shuts down. Without
|
||||
/// this, an async watcher blocked on a timer/stream panics when the runtime is
|
||||
/// torn down out from under it.
|
||||
pub struct WatcherHandle {
|
||||
stop: Option<Box<dyn FnOnce() + Send>>,
|
||||
}
|
||||
|
||||
impl WatcherHandle {
|
||||
pub fn new(stop: impl FnOnce() + Send + 'static) -> Self {
|
||||
return WatcherHandle {
|
||||
stop: Some(Box::new(stop)),
|
||||
};
|
||||
}
|
||||
|
||||
/// A watcher with no shutdown work — its thread exits on its own.
|
||||
pub fn detached() -> Self {
|
||||
return WatcherHandle { stop: None };
|
||||
}
|
||||
|
||||
/// Signal the watcher to stop and wait for it to finish.
|
||||
pub fn stop(mut self) {
|
||||
if let Some(stop) = self.stop.take() {
|
||||
stop();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub trait Origin: Send + Sync {
|
||||
fn snapshot(&self) -> io::Result<BTreeMap<INodeNo, Item>>;
|
||||
fn byte_source(&self) -> Arc<dyn ByteSource>;
|
||||
fn watcher(&self) -> Box<dyn FileWatcher>;
|
||||
}
|
||||
|
||||
pub struct FuseFs {
|
||||
pub files: Arc<Mutex<BTreeMap<INodeNo, Item>>>,
|
||||
pub bytes: Arc<dyn ByteSource>,
|
||||
pub client: DatabaseConnection,
|
||||
pub runtime_handle: tokio::runtime::Handle,
|
||||
}
|
||||
|
||||
pub(crate) fn file_to_attr(item: &Item) -> FileAttr {
|
||||
let attrs = &item.attrs;
|
||||
let kind = match item.file_type {
|
||||
FileType::Directory => fuser::FileType::Directory,
|
||||
FileType::File => fuser::FileType::RegularFile,
|
||||
};
|
||||
|
||||
let size = match &item.music_metadata {
|
||||
Some(mm) if !mm.header.is_empty() => mm.virtual_size(attrs.size),
|
||||
_ => attrs.size,
|
||||
};
|
||||
|
||||
return FileAttr {
|
||||
ino: item.inode,
|
||||
size,
|
||||
blocks: attrs.blocks,
|
||||
atime: attrs.atime,
|
||||
mtime: attrs.mtime,
|
||||
ctime: attrs.ctime,
|
||||
crtime: attrs.crtime,
|
||||
kind,
|
||||
perm: attrs.perm,
|
||||
nlink: attrs.nlink,
|
||||
uid: attrs.uid,
|
||||
gid: attrs.gid,
|
||||
rdev: attrs.rdev,
|
||||
blksize: attrs.blksize,
|
||||
flags: 0,
|
||||
};
|
||||
}
|
||||
|
||||
impl Filesystem for FuseFs {
|
||||
fn open(
|
||||
&self,
|
||||
_req: &Request,
|
||||
ino: INodeNo,
|
||||
_flags: fuser::OpenFlags,
|
||||
reply: fuser::ReplyOpen,
|
||||
) {
|
||||
trace!(%ino, "open");
|
||||
if self.files.lock().unwrap().contains_key(&ino) {
|
||||
reply.opened(fuser::FileHandle(ino.0), fuser::FopenFlags::empty());
|
||||
} else {
|
||||
debug!(%ino, "open: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
}
|
||||
}
|
||||
|
||||
fn setattr(
|
||||
&self,
|
||||
_req: &Request,
|
||||
ino: INodeNo,
|
||||
_mode: Option<u32>,
|
||||
_uid: Option<u32>,
|
||||
_gid: Option<u32>,
|
||||
_size: Option<u64>,
|
||||
_atime: Option<fuser::TimeOrNow>,
|
||||
_mtime: Option<fuser::TimeOrNow>,
|
||||
_ctime: Option<std::time::SystemTime>,
|
||||
_fh: Option<fuser::FileHandle>,
|
||||
_crtime: Option<std::time::SystemTime>,
|
||||
_chgtime: Option<std::time::SystemTime>,
|
||||
_bkuptime: Option<std::time::SystemTime>,
|
||||
_flags: Option<fuser::BsdFileFlags>,
|
||||
reply: fuser::ReplyAttr,
|
||||
) {
|
||||
trace!(%ino, "setattr");
|
||||
match self.files.lock().unwrap().get(&ino) {
|
||||
Some(file) => reply.attr(&Duration::new(1, 0), &file_to_attr(file)),
|
||||
None => {
|
||||
debug!(%ino, "setattr: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn write(
|
||||
&self,
|
||||
_req: &Request,
|
||||
ino: INodeNo,
|
||||
_fh: fuser::FileHandle,
|
||||
offset: u64,
|
||||
data: &[u8],
|
||||
_write_flags: fuser::WriteFlags,
|
||||
_flags: fuser::OpenFlags,
|
||||
_lock_owner: Option<fuser::LockOwner>,
|
||||
reply: fuser::ReplyWrite,
|
||||
) {
|
||||
trace!(%ino, offset, len = data.len(), "write");
|
||||
let written = data.len() as u32;
|
||||
let write_start = offset;
|
||||
let write_end = write_start + data.len() as u64;
|
||||
|
||||
let updated_music_metadata = {
|
||||
let mut files = self.files.lock().unwrap();
|
||||
let item = match files.get_mut(&ino) {
|
||||
Some(item) => item,
|
||||
None => {
|
||||
debug!(%ino, "write: not found");
|
||||
reply.written(written);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
match &mut item.music_metadata {
|
||||
Some(mm) if mm.vorbis_comment_length > 0 => {
|
||||
let vc_data_offset = mm.vorbis_comment_offset;
|
||||
let vc_hdr_offset = vc_data_offset - 4;
|
||||
if write_start <= vc_hdr_offset && write_end >= vc_data_offset {
|
||||
let hdr_from = (vc_hdr_offset - write_start) as usize;
|
||||
let new_length = u32::from_be_bytes([
|
||||
0,
|
||||
data[hdr_from + 1],
|
||||
data[hdr_from + 2],
|
||||
data[hdr_from + 3],
|
||||
]) as u64;
|
||||
let vc_data_end = vc_data_offset + new_length;
|
||||
if write_end >= vc_data_end {
|
||||
let from = (vc_data_offset - write_start) as usize;
|
||||
let to = (vc_data_end - write_start) as usize;
|
||||
mm.update_from_vorbis_comment_data(&data[from..to]);
|
||||
debug!(%ino, "write: vorbis comment tag update detected");
|
||||
Some(mm.clone())
|
||||
} else {
|
||||
None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
Some(mm)
|
||||
if !mm.header.is_empty()
|
||||
&& write_start == 0
|
||||
&& data.len() >= 3
|
||||
&& &data[0..3] == b"ID3" =>
|
||||
{
|
||||
mm.update_from_id3_data(data);
|
||||
debug!(%ino, "write: ID3v2 tag update detected");
|
||||
Some(mm.clone())
|
||||
}
|
||||
Some(mm) if mm.header.is_empty() && data.len() == 128 && &data[0..3] == b"TAG" => {
|
||||
mm.update_from_id3v1_data(data);
|
||||
debug!(%ino, "write: ID3v1 tag update detected");
|
||||
Some(mm.clone())
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
};
|
||||
|
||||
if let Some(music_metadata) = updated_music_metadata {
|
||||
let client = self.client.clone();
|
||||
let ino_i64 = ino.0 as i64;
|
||||
self.runtime_handle.block_on(async move {
|
||||
if let Err(e) = save_music_metadata(ino_i64, &music_metadata, &client).await {
|
||||
error!(ino = ino_i64, error = %e, "write: save_music_metadata failed");
|
||||
} else {
|
||||
debug!(ino = ino_i64, "write: persisted updated music metadata");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
reply.written(written);
|
||||
}
|
||||
|
||||
fn getattr(
|
||||
&self,
|
||||
_req: &fuser::Request,
|
||||
ino: INodeNo,
|
||||
_fh: Option<fuser::FileHandle>,
|
||||
reply: fuser::ReplyAttr,
|
||||
) {
|
||||
trace!(%ino, "getattr");
|
||||
match self.files.lock().unwrap().get(&ino) {
|
||||
Some(file) => {
|
||||
let ttl = Duration::new(1, 0);
|
||||
let attr = file_to_attr(file);
|
||||
reply.attr(&ttl, &attr);
|
||||
}
|
||||
None => {
|
||||
debug!(%ino, "getattr: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn readdir(
|
||||
&self,
|
||||
_req: &fuser::Request,
|
||||
ino: INodeNo,
|
||||
fh: fuser::FileHandle,
|
||||
offset: u64,
|
||||
mut reply: fuser::ReplyDirectory,
|
||||
) {
|
||||
trace!(%ino, %fh, offset, "readdir");
|
||||
|
||||
let files = self.files.lock().unwrap();
|
||||
|
||||
let parent_inode = match files.get(&ino) {
|
||||
Some(dir) => dir.parent_inode,
|
||||
None => {
|
||||
debug!(%ino, "readdir: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
if offset < 1 {
|
||||
if reply.add(ino, 1, fuser::FileType::Directory, ".") {
|
||||
reply.ok();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if offset < 2 {
|
||||
if reply.add(parent_inode, 2, fuser::FileType::Directory, "..") {
|
||||
reply.ok();
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
let skip_count = if offset <= 2 {
|
||||
0
|
||||
} else {
|
||||
(offset - 2) as usize
|
||||
};
|
||||
|
||||
for (i, (key, value)) in files
|
||||
.iter()
|
||||
.filter(|(_, v)| v.parent_inode == ino && v.inode != ino)
|
||||
.skip(skip_count)
|
||||
.enumerate()
|
||||
{
|
||||
let entry_offset = (skip_count + i + 3) as u64;
|
||||
let file_type = match value.file_type {
|
||||
FileType::Directory => fuser::FileType::Directory,
|
||||
FileType::File => fuser::FileType::RegularFile,
|
||||
};
|
||||
if reply.add(*key, entry_offset, file_type, &value.name) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
reply.ok();
|
||||
}
|
||||
|
||||
fn rename(
|
||||
&self,
|
||||
_req: &Request,
|
||||
parent: INodeNo,
|
||||
name: &std::ffi::OsStr,
|
||||
newparent: INodeNo,
|
||||
newname: &std::ffi::OsStr,
|
||||
_flags: fuser::RenameFlags,
|
||||
reply: fuser::ReplyEmpty,
|
||||
) {
|
||||
trace!(
|
||||
%parent,
|
||||
name = %name.display(),
|
||||
%newparent,
|
||||
newname = %newname.display(),
|
||||
"rename"
|
||||
);
|
||||
let name_str = match name.to_str() {
|
||||
Some(s) => s,
|
||||
None => {
|
||||
error!(%parent, "rename: source name is not valid UTF-8");
|
||||
reply.error(Errno::EINVAL);
|
||||
return;
|
||||
}
|
||||
};
|
||||
let newname_str = match newname.to_str() {
|
||||
Some(s) => s,
|
||||
None => {
|
||||
error!(%newparent, "rename: target name is not valid UTF-8");
|
||||
reply.error(Errno::EINVAL);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let mut files = self.files.lock().unwrap();
|
||||
|
||||
let item_ino = match files
|
||||
.iter()
|
||||
.find(|(_, v)| v.parent_inode == parent && v.name == name_str)
|
||||
{
|
||||
Some((ino, _)) => *ino,
|
||||
None => {
|
||||
debug!(%parent, name = %name_str, "rename: source not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
let new_local_path = if newparent == INodeNo::ROOT {
|
||||
std::path::PathBuf::from(newname_str)
|
||||
} else {
|
||||
match files.get(&newparent) {
|
||||
Some(dir) => dir.local_path.join(newname_str),
|
||||
None => {
|
||||
debug!(%newparent, "rename: target parent not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
return;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
debug!(ino = %item_ino, from = %name_str, to = %newname_str, "rename");
|
||||
|
||||
let new_parent_inode = parent_inode_from_path(&new_local_path);
|
||||
|
||||
let item = files.get_mut(&item_ino).unwrap();
|
||||
item.name = newname_str.to_string();
|
||||
item.local_path = new_local_path.clone();
|
||||
item.parent_inode = new_parent_inode;
|
||||
|
||||
let inode_i64 = item_ino.0 as i64;
|
||||
let new_name_owned = newname_str.to_string();
|
||||
let new_local_path_str = new_local_path.to_string_lossy().into_owned();
|
||||
let client = self.client.clone();
|
||||
|
||||
drop(files);
|
||||
|
||||
self.runtime_handle.block_on(async move {
|
||||
use sea_orm::ActiveValue::Set;
|
||||
if let Err(e) = (crate::db::entities::ActiveModel {
|
||||
inode: Set(inode_i64),
|
||||
name: Set(new_name_owned),
|
||||
local_path: Set(new_local_path_str),
|
||||
..Default::default()
|
||||
}
|
||||
.update(&client)
|
||||
.await)
|
||||
{
|
||||
error!(ino = inode_i64, error = %e, "rename: db update failed");
|
||||
}
|
||||
});
|
||||
|
||||
reply.ok();
|
||||
}
|
||||
|
||||
fn read(
|
||||
&self,
|
||||
_req: &Request,
|
||||
ino: INodeNo,
|
||||
_fh: fuser::FileHandle,
|
||||
offset: u64,
|
||||
size: u32,
|
||||
_flags: fuser::OpenFlags,
|
||||
_lock_owner: Option<fuser::LockOwner>,
|
||||
reply: fuser::ReplyData,
|
||||
) {
|
||||
trace!(%ino, offset, size, "read");
|
||||
let (inode, locator, virtual_layout) = {
|
||||
let files = self.files.lock().unwrap();
|
||||
let item = match files.get(&ino) {
|
||||
Some(f) => f,
|
||||
None => {
|
||||
debug!(%ino, "read: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
return;
|
||||
}
|
||||
};
|
||||
let inode = item.inode;
|
||||
let locator = item.original_path.clone();
|
||||
let virtual_layout = item
|
||||
.music_metadata
|
||||
.as_ref()
|
||||
.filter(|mm| !mm.header.is_empty())
|
||||
.map(|mm| {
|
||||
(
|
||||
mm.header.starts_with(b"ID3"),
|
||||
mm.header.clone(),
|
||||
mm.picture_block_headers.clone(),
|
||||
mm.picture_data_ranges.clone(),
|
||||
mm.real_audio_start,
|
||||
)
|
||||
});
|
||||
(inode, locator, virtual_layout)
|
||||
};
|
||||
|
||||
let Some((is_mp3, header, pic_hdrs, pic_ranges, real_audio_start)) = virtual_layout else {
|
||||
match self.bytes.read_at(inode, &locator, offset, size as usize) {
|
||||
Ok(bytes) => reply.data(&bytes),
|
||||
Err(e) => {
|
||||
error!(%inode, offset, size, error = %e, "read: read_at failed; returning EIO");
|
||||
reply.error(Errno::EIO);
|
||||
}
|
||||
}
|
||||
return;
|
||||
};
|
||||
|
||||
let bytes = &self.bytes;
|
||||
let reader = |off: u64, len: usize| bytes.read_at(inode, &locator, off, len);
|
||||
let result = if is_mp3 {
|
||||
file_io::assemble_mp3_read(
|
||||
&reader,
|
||||
&header,
|
||||
&pic_hdrs,
|
||||
&pic_ranges,
|
||||
real_audio_start,
|
||||
offset,
|
||||
size,
|
||||
)
|
||||
} else {
|
||||
file_io::assemble_flac_read(
|
||||
&reader,
|
||||
&header,
|
||||
&pic_hdrs,
|
||||
&pic_ranges,
|
||||
real_audio_start,
|
||||
offset,
|
||||
size,
|
||||
)
|
||||
};
|
||||
match result {
|
||||
Ok(bytes) => reply.data(&bytes),
|
||||
Err(e) => {
|
||||
error!(%inode, offset, size, error = %e, "read: assembly failed; returning EIO");
|
||||
reply.error(Errno::EIO);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn lookup(
|
||||
&self,
|
||||
_req: &Request,
|
||||
parent: INodeNo,
|
||||
name: &std::ffi::OsStr,
|
||||
reply: fuser::ReplyEntry,
|
||||
) {
|
||||
trace!(%parent, name = %name.display(), "lookup");
|
||||
|
||||
match self
|
||||
.files
|
||||
.lock()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.find(|item| item.1.parent_inode == parent && item.1.name == name.to_str().unwrap())
|
||||
{
|
||||
Some(item) => {
|
||||
let ttl = Duration::new(1, 0);
|
||||
let attr = file_to_attr(item.1);
|
||||
|
||||
reply.entry(&ttl, &attr, Generation(0));
|
||||
}
|
||||
None => {
|
||||
debug!(%parent, name = %name.display(), "lookup: not found");
|
||||
reply.error(Errno::ENOENT);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::item::FileType;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
use fuser::INodeNo;
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
use std::path::PathBuf;
|
||||
|
||||
#[test]
|
||||
fn file_to_attr_uses_real_size_for_non_flac() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = std::fs::metadata(tmp.path()).unwrap();
|
||||
let attrs = FileAttrs::from(&metadata);
|
||||
let item = Item::new(
|
||||
INodeNo(42),
|
||||
INodeNo::ROOT,
|
||||
"test".to_string(),
|
||||
tmp.path().to_path_buf(),
|
||||
PathBuf::from("test"),
|
||||
FileType::File,
|
||||
attrs.clone(),
|
||||
None,
|
||||
);
|
||||
|
||||
let attr = file_to_attr(&item);
|
||||
assert_eq!(attr.size, metadata.size());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_to_attr_uses_virtual_size_for_flac() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let test_file = tmp.path().join("test.flac");
|
||||
std::fs::write(&test_file, vec![0u8; 1000]).unwrap();
|
||||
let metadata = std::fs::metadata(&test_file).unwrap();
|
||||
let attrs = FileAttrs::from(&metadata);
|
||||
let mm = MusicMetadata {
|
||||
real_audio_start: 500,
|
||||
header: vec![0u8; 100],
|
||||
picture_data_ranges: vec![(0, 50)],
|
||||
..MusicMetadata::default()
|
||||
};
|
||||
let item = Item::new(
|
||||
INodeNo(43),
|
||||
INodeNo::ROOT,
|
||||
"test_flac".to_string(),
|
||||
test_file.clone(),
|
||||
PathBuf::from("test_flac"),
|
||||
FileType::File,
|
||||
attrs.clone(),
|
||||
Some(mm.clone()),
|
||||
);
|
||||
|
||||
let attr = file_to_attr(&item);
|
||||
let expected_virtual_size = mm.virtual_size(attrs.size);
|
||||
assert_eq!(attr.size, expected_virtual_size);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_to_attr_kind_directory() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = std::fs::metadata(tmp.path()).unwrap();
|
||||
let item = Item::new(
|
||||
INodeNo(44),
|
||||
INodeNo::ROOT,
|
||||
"test_dir".to_string(),
|
||||
tmp.path().to_path_buf(),
|
||||
PathBuf::from("test_dir"),
|
||||
FileType::Directory,
|
||||
FileAttrs::from(&metadata),
|
||||
None,
|
||||
);
|
||||
|
||||
let attr = file_to_attr(&item);
|
||||
assert_eq!(attr.kind, fuser::FileType::Directory);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_to_attr_kind_file() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = std::fs::metadata(tmp.path()).unwrap();
|
||||
let item = Item::new(
|
||||
INodeNo(45),
|
||||
INodeNo::ROOT,
|
||||
"test_file".to_string(),
|
||||
tmp.path().to_path_buf(),
|
||||
PathBuf::from("test_file"),
|
||||
FileType::File,
|
||||
FileAttrs::from(&metadata),
|
||||
None,
|
||||
);
|
||||
|
||||
let attr = file_to_attr(&item);
|
||||
assert_eq!(attr.kind, fuser::FileType::RegularFile);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,374 @@
|
||||
pub mod transport;
|
||||
pub mod watcher;
|
||||
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io,
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, RwLock},
|
||||
time::{Duration, SystemTime},
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use sea_orm::EntityTrait;
|
||||
|
||||
use crate::db::cache::{delete_cached_bytes_for, get_cached_bytes, put_cached_bytes};
|
||||
use crate::db::entities as item_entities;
|
||||
use crate::db::sync::{run_db_blocking, sync_items_to_db};
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::music::db::restore_music_metadata_from_db;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
use crate::origins::{ByteSource, FileWatcher, Origin};
|
||||
use crate::proto::ManifestEntry as ProtoManifestEntry;
|
||||
use crate::virtual_dirs::{ensure_virtual_dirs, restore_virtual_paths};
|
||||
use tracing::{debug, error, info, trace};
|
||||
|
||||
use self::transport::NetworkTransport;
|
||||
|
||||
pub struct NetworkOrigin {
|
||||
pub(crate) endpoint: String,
|
||||
pub(crate) destination: PathBuf,
|
||||
handle: tokio::runtime::Handle,
|
||||
transport: NetworkTransport,
|
||||
client: sea_orm::DatabaseConnection,
|
||||
latest_manifest: Arc<RwLock<BTreeMap<u64, ProtoManifestEntry>>>,
|
||||
server_status: crate::health::ServerStatus,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for NetworkOrigin {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
return f
|
||||
.debug_struct("NetworkOrigin")
|
||||
.field("endpoint", &self.endpoint)
|
||||
.field("destination", &self.destination)
|
||||
.finish();
|
||||
}
|
||||
}
|
||||
|
||||
impl NetworkOrigin {
|
||||
pub fn new(
|
||||
endpoint: String,
|
||||
destination: String,
|
||||
client: sea_orm::DatabaseConnection,
|
||||
) -> io::Result<Self> {
|
||||
let handle = tokio::runtime::Handle::current();
|
||||
let transport = NetworkTransport::new(endpoint.clone()).map_err(io_err)?;
|
||||
let server_status =
|
||||
crate::health::ServerStatus::new_network(destination.clone(), endpoint.clone());
|
||||
return Ok(NetworkOrigin {
|
||||
endpoint,
|
||||
destination: destination.into(),
|
||||
handle,
|
||||
transport,
|
||||
client,
|
||||
latest_manifest: Arc::new(RwLock::new(BTreeMap::new())),
|
||||
server_status,
|
||||
});
|
||||
}
|
||||
|
||||
pub fn runtime_handle(&self) -> tokio::runtime::Handle {
|
||||
return self.handle.clone();
|
||||
}
|
||||
|
||||
pub fn server_status(&self) -> crate::health::ServerStatus {
|
||||
self.server_status.clone()
|
||||
}
|
||||
}
|
||||
|
||||
impl Origin for NetworkOrigin {
|
||||
fn snapshot(&self) -> io::Result<BTreeMap<INodeNo, Item>> {
|
||||
return run_db_blocking(self.snapshot_async()).map_err(io_err);
|
||||
}
|
||||
|
||||
fn byte_source(&self) -> Arc<dyn ByteSource> {
|
||||
return Arc::new(NetworkByteSource {
|
||||
transport: self.transport.clone(),
|
||||
runtime_handle: self.runtime_handle(),
|
||||
client: self.client.clone(),
|
||||
});
|
||||
}
|
||||
|
||||
fn watcher(&self) -> Box<dyn FileWatcher> {
|
||||
return Box::new(watcher::NetworkOriginFileWatcher::new(
|
||||
self.transport.clone(),
|
||||
self.runtime_handle(),
|
||||
self.client.clone(),
|
||||
self.destination.clone(),
|
||||
self.latest_manifest.clone(),
|
||||
self.server_status.clone(),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
impl NetworkOrigin {
|
||||
/// Async snapshot driven directly on the caller's runtime. `main` (already
|
||||
/// `#[tokio::main]`) awaits this; the sync `Origin::snapshot()` wrapper is
|
||||
/// only for non-async callers and must not be invoked from within a runtime
|
||||
/// (it builds and `block_on`s a throwaway one).
|
||||
pub async fn snapshot_async(&self) -> io::Result<BTreeMap<INodeNo, Item>> {
|
||||
// 1. Pull client's current (inode, hash) pairs from the DB.
|
||||
let client_entries: Vec<(u64, u64)> = item_entities::Entity::find()
|
||||
.all(&self.client)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
error!(error = %e, "network snapshot: client entries DB read failed");
|
||||
io_err(e)
|
||||
})?
|
||||
.into_iter()
|
||||
.filter(|m| m.file_type == "file")
|
||||
.map(|m| (m.inode as u64, m.hash as u64))
|
||||
.collect();
|
||||
|
||||
// 2. Ask server what changed.
|
||||
let response = self
|
||||
.transport
|
||||
.reconcile(client_entries)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
error!(error = %e, "network snapshot: reconcile with server failed");
|
||||
io_err(e)
|
||||
})?;
|
||||
|
||||
// 3. Invalidate cached bytes for changed + deleted inodes — they are
|
||||
// stale by definition.
|
||||
let mut changed_or_deleted: Vec<i64> =
|
||||
response.changed.iter().map(|ih| ih.inode as i64).collect();
|
||||
changed_or_deleted.extend(response.deleted.iter().map(|i| *i as i64));
|
||||
delete_cached_bytes_for(&changed_or_deleted, &self.client).await;
|
||||
|
||||
// latest_manifest is in-memory; on restart it's empty so the reconcile
|
||||
// delta misses unchanged files. Fetch full manifest then, delta otherwise.
|
||||
let mut current_manifest = self.latest_manifest.read().unwrap().clone();
|
||||
if current_manifest.is_empty() {
|
||||
let entries = self.transport.get_manifest().await.map_err(|e| {
|
||||
error!(error = %e, "network snapshot: get_manifest from server failed");
|
||||
io_err(e)
|
||||
})?;
|
||||
current_manifest = entries.into_iter().map(|e| (e.id, e)).collect();
|
||||
} else {
|
||||
let wanted: Vec<u64> = response.changed.iter().map(|ih| ih.inode).collect();
|
||||
if !wanted.is_empty() {
|
||||
let wanted_count = wanted.len();
|
||||
let entries = self.transport.get_metadata(wanted).await.map_err(|e| {
|
||||
error!(
|
||||
wanted = wanted_count,
|
||||
error = %e,
|
||||
"network snapshot: get_metadata from server failed"
|
||||
);
|
||||
io_err(e)
|
||||
})?;
|
||||
for entry in entries {
|
||||
current_manifest.insert(entry.id, entry);
|
||||
}
|
||||
}
|
||||
}
|
||||
for inode in &response.deleted {
|
||||
current_manifest.remove(inode);
|
||||
}
|
||||
*self.latest_manifest.write().unwrap() = current_manifest.clone();
|
||||
|
||||
let snapshot =
|
||||
build_snapshot_from_manifest(¤t_manifest, &self.destination, &self.client)
|
||||
.await?;
|
||||
|
||||
info!(
|
||||
changed = response.changed.len(),
|
||||
deleted = response.deleted.len(),
|
||||
files = snapshot.len(),
|
||||
"network snapshot complete"
|
||||
);
|
||||
return Ok(snapshot);
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a complete Items snapshot from the server manifest, sync it to the
|
||||
/// DB, and restore music metadata + virtual paths from existing DB rows.
|
||||
///
|
||||
/// Shared between `snapshot_async` (initial mount) and the watcher's
|
||||
/// `reconcile_once` (runtime updates) so both paths produce identical
|
||||
/// snapshots and keep the DB in sync.
|
||||
pub(crate) async fn build_snapshot_from_manifest(
|
||||
manifest: &BTreeMap<u64, ProtoManifestEntry>,
|
||||
destination: &Path,
|
||||
client: &sea_orm::DatabaseConnection,
|
||||
) -> io::Result<BTreeMap<INodeNo, Item>> {
|
||||
let source_root = PathBuf::from("/");
|
||||
let mut snapshot = BTreeMap::new();
|
||||
let root_attrs = FileAttrs {
|
||||
size: 0,
|
||||
blocks: 0,
|
||||
atime: SystemTime::UNIX_EPOCH,
|
||||
mtime: SystemTime::UNIX_EPOCH,
|
||||
ctime: SystemTime::UNIX_EPOCH,
|
||||
crtime: SystemTime::UNIX_EPOCH,
|
||||
perm: 0o755,
|
||||
nlink: 2,
|
||||
uid: 0,
|
||||
gid: 0,
|
||||
rdev: 0,
|
||||
blksize: 4096,
|
||||
};
|
||||
snapshot.insert(
|
||||
INodeNo::ROOT,
|
||||
Item::new(
|
||||
INodeNo::ROOT,
|
||||
INodeNo::ROOT,
|
||||
"/".to_string(),
|
||||
destination.to_path_buf(),
|
||||
destination.to_path_buf(),
|
||||
FileType::Directory,
|
||||
root_attrs,
|
||||
None,
|
||||
),
|
||||
);
|
||||
|
||||
for (_id, entry) in manifest {
|
||||
let item = manifest_entry_to_item(entry, &source_root, &mut snapshot);
|
||||
snapshot.insert(item.inode, item);
|
||||
}
|
||||
|
||||
let db_items: std::collections::HashMap<i64, item_entities::Model> =
|
||||
item_entities::Entity::find()
|
||||
.all(client)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
error!(error = %e, "build_snapshot_from_manifest: DB read failed");
|
||||
io_err(e)
|
||||
})?
|
||||
.into_iter()
|
||||
.map(|e| (e.inode, e))
|
||||
.collect();
|
||||
sync_items_to_db(&snapshot, &db_items, client).await;
|
||||
restore_music_metadata_from_db(&mut snapshot, &db_items, client).await;
|
||||
restore_virtual_paths(&mut snapshot, &db_items, &source_root);
|
||||
|
||||
return Ok(snapshot);
|
||||
}
|
||||
|
||||
fn manifest_entry_to_item(
|
||||
entry: &ProtoManifestEntry,
|
||||
source_root: &Path,
|
||||
snapshot: &mut BTreeMap<INodeNo, Item>,
|
||||
) -> Item {
|
||||
let inode = INodeNo(entry.id);
|
||||
let original_path = PathBuf::from(&entry.rel_path);
|
||||
let attrs = FileAttrs {
|
||||
size: entry.size,
|
||||
blocks: 0,
|
||||
atime: SystemTime::UNIX_EPOCH + Duration::from_secs(entry.mtime),
|
||||
mtime: SystemTime::UNIX_EPOCH + Duration::from_secs(entry.mtime),
|
||||
ctime: SystemTime::UNIX_EPOCH + Duration::from_secs(entry.ctime),
|
||||
crtime: SystemTime::UNIX_EPOCH + Duration::from_secs(entry.crtime),
|
||||
perm: 0o644,
|
||||
nlink: 1,
|
||||
uid: 0,
|
||||
gid: 0,
|
||||
rdev: 0,
|
||||
blksize: 4096,
|
||||
};
|
||||
let music_metadata: Option<MusicMetadata> =
|
||||
entry.music_metadata.clone().map(music_metadata_from_proto);
|
||||
|
||||
let mut local_path = PathBuf::new();
|
||||
if let Some(mm) = &music_metadata {
|
||||
let joined;
|
||||
let artist_dir = match mm.album_artist.as_deref() {
|
||||
Some(a) => a,
|
||||
None => {
|
||||
joined = mm.artist.join("-");
|
||||
&joined
|
||||
}
|
||||
};
|
||||
local_path.push(artist_dir);
|
||||
local_path.push(&mm.album);
|
||||
}
|
||||
let name = Path::new(&entry.rel_path)
|
||||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| entry.rel_path.clone());
|
||||
local_path.push(&name);
|
||||
|
||||
let parent_inode = ensure_virtual_dirs(&local_path, source_root, snapshot);
|
||||
return Item::new(
|
||||
inode,
|
||||
parent_inode,
|
||||
name,
|
||||
original_path,
|
||||
local_path,
|
||||
FileType::File,
|
||||
attrs,
|
||||
music_metadata,
|
||||
);
|
||||
}
|
||||
|
||||
fn music_metadata_from_proto(mm: crate::proto::MusicMetadata) -> MusicMetadata {
|
||||
MusicMetadata {
|
||||
artist: mm.artist,
|
||||
album_artist: mm.album_artist,
|
||||
album: mm.album,
|
||||
track_number: mm.track_number,
|
||||
track_title: mm.track_title,
|
||||
other_tags: mm.other_tags,
|
||||
header: mm.header,
|
||||
picture_block_headers: mm.picture_block_headers,
|
||||
picture_data_ranges: mm
|
||||
.picture_data_ranges
|
||||
.into_iter()
|
||||
.map(|p| (p.offset, p.length))
|
||||
.collect(),
|
||||
real_audio_start: mm.real_audio_start,
|
||||
vorbis_comment_offset: mm.vorbis_comment_offset,
|
||||
vorbis_comment_length: mm.vorbis_comment_length,
|
||||
}
|
||||
}
|
||||
|
||||
fn io_err<E: std::fmt::Display>(e: E) -> io::Error {
|
||||
return io::Error::new(io::ErrorKind::Other, e.to_string());
|
||||
}
|
||||
|
||||
pub struct NetworkByteSource {
|
||||
transport: NetworkTransport,
|
||||
runtime_handle: tokio::runtime::Handle,
|
||||
client: sea_orm::DatabaseConnection,
|
||||
}
|
||||
|
||||
impl ByteSource for NetworkByteSource {
|
||||
fn read_at(
|
||||
&self,
|
||||
inode: INodeNo,
|
||||
_locator: &Path,
|
||||
offset: u64,
|
||||
len: usize,
|
||||
) -> io::Result<Vec<u8>> {
|
||||
let inode_i64 = inode.0 as i64;
|
||||
trace!(%inode, offset, len, "network read_at");
|
||||
|
||||
let cached = self
|
||||
.runtime_handle
|
||||
.block_on(get_cached_bytes(inode_i64, &self.client));
|
||||
if let Some(data) = cached {
|
||||
debug!(%inode, "read_at cache hit");
|
||||
return slice_range(&data, offset, len);
|
||||
}
|
||||
|
||||
debug!(%inode, "read_at cache miss; fetching from server");
|
||||
let (data, _total_size) = self
|
||||
.runtime_handle
|
||||
.block_on(self.transport.fetch_file_range(inode.0, 0, 0))
|
||||
.map_err(|e| {
|
||||
error!(%inode, error = %e, "read_at: fetch_file_range failed");
|
||||
io_err(e)
|
||||
})?;
|
||||
self.runtime_handle
|
||||
.block_on(put_cached_bytes(inode_i64, data.clone(), &self.client));
|
||||
|
||||
return slice_range(&data, offset, len);
|
||||
}
|
||||
}
|
||||
|
||||
fn slice_range(data: &[u8], offset: u64, len: usize) -> io::Result<Vec<u8>> {
|
||||
let start = (offset as usize).min(data.len());
|
||||
let end = (start + len).min(data.len());
|
||||
return Ok(data[start..end].to_vec());
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result, anyhow};
|
||||
use tokio_stream::StreamExt;
|
||||
use tonic::codec::Streaming;
|
||||
|
||||
use crate::proto::{
|
||||
ChangeEvent, GetFileRequest, GetManifestRequest, GetMetadataRequest, InodeHash, ManifestEntry,
|
||||
MusicFsClient, ReconcileRequest, ReconcileResponse, SubscribeEventsRequest,
|
||||
};
|
||||
|
||||
/// Thin wrapper over the generated tonic client. Owns the connection and
|
||||
/// exposes four operations matching the four RPCs NetworkOrigin needs.
|
||||
///
|
||||
/// All methods are async and must be polled on the runtime whose Handle was
|
||||
/// passed in at construction (or a child of it). The sync FUSE path uses
|
||||
/// `runtime_handle.block_on(...)` to enter this runtime.
|
||||
#[derive(Clone)]
|
||||
pub struct NetworkTransport {
|
||||
client: Arc<tokio::sync::Mutex<MusicFsClient<tonic::transport::Channel>>>,
|
||||
endpoint: tonic::transport::Endpoint,
|
||||
}
|
||||
|
||||
impl NetworkTransport {
|
||||
/// Build a transport that connects lazily. The first RPC establishes the
|
||||
/// connection; reconnects happen automatically if the channel drops.
|
||||
pub fn new(url: String) -> Result<Self> {
|
||||
let endpoint: tonic::transport::Endpoint = url
|
||||
.try_into()
|
||||
.map_err(|e| anyhow!("invalid server URL: {e}"))?;
|
||||
let endpoint = endpoint
|
||||
.timeout(Duration::from_secs(3))
|
||||
.connect_timeout(Duration::from_secs(5))
|
||||
.http2_keep_alive_interval(Duration::from_secs(10))
|
||||
.keep_alive_timeout(Duration::from_secs(5));
|
||||
let channel = endpoint.connect_lazy();
|
||||
let client = Arc::new(tokio::sync::Mutex::new(MusicFsClient::new(channel)));
|
||||
return Ok(NetworkTransport { client, endpoint });
|
||||
}
|
||||
|
||||
pub async fn reconcile(&self, entries: Vec<(u64, u64)>) -> Result<ReconcileResponse> {
|
||||
let request = ReconcileRequest {
|
||||
entries: entries
|
||||
.into_iter()
|
||||
.map(|(inode, hash)| InodeHash { inode, hash })
|
||||
.collect(),
|
||||
};
|
||||
let mut client = self.client.lock().await;
|
||||
let response = client
|
||||
.reconcile(request)
|
||||
.await
|
||||
.context("Reconcile RPC failed")?;
|
||||
return Ok(response.into_inner());
|
||||
}
|
||||
|
||||
pub async fn get_metadata(&self, inodes: Vec<u64>) -> Result<Vec<ManifestEntry>> {
|
||||
let request = GetMetadataRequest { inodes };
|
||||
let mut client = self.client.lock().await;
|
||||
let mut stream: Streaming<ManifestEntry> = client
|
||||
.get_metadata(request)
|
||||
.await
|
||||
.context("GetMetadata RPC failed")?
|
||||
.into_inner();
|
||||
let mut out = Vec::new();
|
||||
while let Some(entry) = stream.next().await {
|
||||
out.push(entry.context("GetMetadata stream error")?);
|
||||
}
|
||||
return Ok(out);
|
||||
}
|
||||
|
||||
pub async fn get_manifest(&self) -> Result<Vec<ManifestEntry>> {
|
||||
let mut client = self.client.lock().await;
|
||||
let mut stream: Streaming<ManifestEntry> = client
|
||||
.get_manifest(GetManifestRequest {})
|
||||
.await
|
||||
.context("GetManifest RPC failed")?
|
||||
.into_inner();
|
||||
let mut out = Vec::new();
|
||||
while let Some(entry) = stream.next().await {
|
||||
out.push(entry.context("GetManifest stream error")?);
|
||||
}
|
||||
return Ok(out);
|
||||
}
|
||||
|
||||
pub async fn fetch_file_range(
|
||||
&self,
|
||||
id: u64,
|
||||
start: u64,
|
||||
length: u64,
|
||||
) -> Result<(Vec<u8>, u64)> {
|
||||
let request = GetFileRequest { id, start, length };
|
||||
let mut client = self.client.lock().await;
|
||||
let mut stream = client
|
||||
.get_file(request)
|
||||
.await
|
||||
.context("GetFile RPC failed")?
|
||||
.into_inner();
|
||||
let first = stream
|
||||
.next()
|
||||
.await
|
||||
.ok_or_else(|| anyhow!("GetFile returned empty stream for id {id}"))?
|
||||
.context("GetFile stream error")?;
|
||||
let total_size = first.total_size;
|
||||
let mut data = first.data;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk.context("GetFile stream error")?;
|
||||
data.extend_from_slice(&chunk.data);
|
||||
}
|
||||
return Ok((data, total_size));
|
||||
}
|
||||
|
||||
/// Take a fresh receiver on the SubscribeEvents stream. Each call opens a
|
||||
/// new server-streaming RPC; the caller owns the lifetime.
|
||||
pub async fn subscribe_events(&self) -> Result<Streaming<ChangeEvent>> {
|
||||
let mut client = self.client.lock().await;
|
||||
let stream = client
|
||||
.subscribe_events(SubscribeEventsRequest {})
|
||||
.await
|
||||
.context("SubscribeEvents RPC failed")?
|
||||
.into_inner();
|
||||
return Ok(stream);
|
||||
}
|
||||
|
||||
/// Reconnect — used by the watcher when the channel has gone bad.
|
||||
#[allow(dead_code)]
|
||||
pub async fn reconnect(&self) -> Result<()> {
|
||||
let channel = self.endpoint.connect().await.context("reconnect failed")?;
|
||||
let new_client = MusicFsClient::new(channel);
|
||||
let mut guard = self.client.lock().await;
|
||||
*guard = new_client;
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,197 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
path::PathBuf,
|
||||
sync::{Arc, RwLock},
|
||||
thread,
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use sea_orm::{DatabaseConnection, EntityTrait};
|
||||
use tokio::sync::Notify;
|
||||
use tokio_stream::StreamExt;
|
||||
|
||||
use crate::item::Item;
|
||||
use crate::origins::network::transport::NetworkTransport;
|
||||
use crate::origins::{FileWatcher, WatcherHandle};
|
||||
use crate::proto::ManifestEntry as ProtoManifestEntry;
|
||||
use tracing::{info, warn};
|
||||
|
||||
const INITIAL_RETRY_DELAY: Duration = Duration::from_secs(1);
|
||||
const MAX_RETRY_DELAY: Duration = Duration::from_secs(30);
|
||||
|
||||
pub struct NetworkOriginFileWatcher {
|
||||
transport: NetworkTransport,
|
||||
runtime_handle: tokio::runtime::Handle,
|
||||
client: DatabaseConnection,
|
||||
destination: PathBuf,
|
||||
latest_manifest: Arc<RwLock<BTreeMap<u64, ProtoManifestEntry>>>,
|
||||
server_status: crate::health::ServerStatus,
|
||||
}
|
||||
|
||||
impl NetworkOriginFileWatcher {
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
transport: NetworkTransport,
|
||||
runtime_handle: tokio::runtime::Handle,
|
||||
client: DatabaseConnection,
|
||||
destination: PathBuf,
|
||||
latest_manifest: Arc<RwLock<BTreeMap<u64, ProtoManifestEntry>>>,
|
||||
server_status: crate::health::ServerStatus,
|
||||
) -> Self {
|
||||
return NetworkOriginFileWatcher {
|
||||
transport,
|
||||
runtime_handle,
|
||||
client,
|
||||
destination,
|
||||
latest_manifest,
|
||||
server_status,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
impl FileWatcher for NetworkOriginFileWatcher {
|
||||
fn watch(&self, files: Arc<std::sync::Mutex<BTreeMap<INodeNo, Item>>>) -> WatcherHandle {
|
||||
let runtime_handle = self.runtime_handle.clone();
|
||||
let shutdown = Arc::new(Notify::new());
|
||||
let state = WatcherState {
|
||||
transport: self.transport.clone(),
|
||||
client: self.client.clone(),
|
||||
destination: self.destination.clone(),
|
||||
latest_manifest: self.latest_manifest.clone(),
|
||||
server_status: self.server_status.clone(),
|
||||
files,
|
||||
};
|
||||
let loop_shutdown = shutdown.clone();
|
||||
let join = thread::spawn(move || {
|
||||
runtime_handle.block_on(state.run_loop(loop_shutdown));
|
||||
});
|
||||
return WatcherHandle::new(move || {
|
||||
shutdown.notify_one();
|
||||
let _ = join.join();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
struct WatcherState {
|
||||
transport: NetworkTransport,
|
||||
client: DatabaseConnection,
|
||||
destination: PathBuf,
|
||||
latest_manifest: Arc<RwLock<BTreeMap<u64, ProtoManifestEntry>>>,
|
||||
server_status: crate::health::ServerStatus,
|
||||
files: Arc<std::sync::Mutex<BTreeMap<INodeNo, Item>>>,
|
||||
}
|
||||
|
||||
impl WatcherState {
|
||||
async fn run_loop(self, shutdown: Arc<Notify>) {
|
||||
let mut retry_delay = INITIAL_RETRY_DELAY;
|
||||
loop {
|
||||
let subscribed = tokio::select! {
|
||||
biased;
|
||||
_ = shutdown.notified() => return,
|
||||
s = self.transport.subscribe_events() => s,
|
||||
};
|
||||
match subscribed {
|
||||
Ok(mut stream) => {
|
||||
info!("network watcher: subscribed to /events");
|
||||
self.server_status.set_connected(true);
|
||||
retry_delay = INITIAL_RETRY_DELAY;
|
||||
loop {
|
||||
let item = tokio::select! {
|
||||
biased;
|
||||
_ = shutdown.notified() => return,
|
||||
item = stream.next() => item,
|
||||
};
|
||||
match item {
|
||||
Some(Ok(_event)) => {
|
||||
if let Err(e) = self.reconcile_once().await {
|
||||
warn!(error = %e, "network watcher: reconcile after event failed");
|
||||
}
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
warn!(error = %e, "network watcher: stream error; reconnecting");
|
||||
self.server_status.set_connected(false);
|
||||
break;
|
||||
}
|
||||
None => {
|
||||
self.server_status.set_connected(false);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
self.server_status.set_connected(false);
|
||||
warn!(error = %e, "network watcher: subscribe failed; will retry");
|
||||
}
|
||||
}
|
||||
tokio::select! {
|
||||
biased;
|
||||
_ = shutdown.notified() => return,
|
||||
_ = tokio::time::sleep(retry_delay) => {}
|
||||
}
|
||||
retry_delay = (retry_delay * 2).min(MAX_RETRY_DELAY);
|
||||
if let Err(e) = self.reconcile_once().await {
|
||||
warn!(error = %e, "network watcher: poll reconcile failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn reconcile_once(&self) -> anyhow::Result<()> {
|
||||
let client_entries: Vec<(u64, u64)> = crate::db::entities::Entity::find()
|
||||
.all(&self.client)
|
||||
.await?
|
||||
.into_iter()
|
||||
.filter(|m| m.file_type == "file")
|
||||
.map(|m| (m.inode as u64, m.hash as u64))
|
||||
.collect();
|
||||
|
||||
let response = self.transport.reconcile(client_entries).await?;
|
||||
|
||||
let mut changed_or_deleted: Vec<i64> =
|
||||
response.changed.iter().map(|ih| ih.inode as i64).collect();
|
||||
changed_or_deleted.extend(response.deleted.iter().map(|i| *i as i64));
|
||||
crate::db::cache::delete_cached_bytes_for(&changed_or_deleted, &self.client).await;
|
||||
|
||||
let wanted: Vec<u64> = response.changed.iter().map(|ih| ih.inode).collect();
|
||||
let mut current_manifest = self.latest_manifest.read().unwrap().clone();
|
||||
if !wanted.is_empty() {
|
||||
let entries = self.transport.get_metadata(wanted).await?;
|
||||
for entry in entries {
|
||||
current_manifest.insert(entry.id, entry);
|
||||
}
|
||||
}
|
||||
for inode in &response.deleted {
|
||||
current_manifest.remove(inode);
|
||||
}
|
||||
*self.latest_manifest.write().unwrap() = current_manifest.clone();
|
||||
|
||||
let new_snapshot =
|
||||
super::build_snapshot_from_manifest(¤t_manifest, &self.destination, &self.client)
|
||||
.await?;
|
||||
|
||||
{
|
||||
let mut files = self.files.lock().unwrap();
|
||||
files.retain(|ino, _| new_snapshot.contains_key(ino));
|
||||
for (ino, new_item) in &new_snapshot {
|
||||
match files.get(ino) {
|
||||
Some(existing) if existing.hash == new_item.hash => {}
|
||||
_ => {
|
||||
files.insert(*ino, new_item.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
self.server_status.mark_reconciled();
|
||||
|
||||
info!(
|
||||
changed = response.changed.len(),
|
||||
deleted = response.deleted.len(),
|
||||
total = current_manifest.len(),
|
||||
"network watcher: reconcile applied"
|
||||
);
|
||||
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,531 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
fs,
|
||||
hash::Hasher,
|
||||
path::{Path, PathBuf},
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use twox_hash::XxHash64;
|
||||
|
||||
use crate::db::entities::Model;
|
||||
use crate::item::{FileType, Item};
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::origins::attrs::FileAttrs;
|
||||
|
||||
pub fn virtual_inode(path: &str) -> INodeNo {
|
||||
let mut hasher = XxHash64::with_seed(5678);
|
||||
hasher.write(path.as_bytes());
|
||||
|
||||
return INodeNo(hasher.finish() | (1u64 << 63));
|
||||
}
|
||||
|
||||
pub fn parent_inode_from_path(local_path: &Path) -> INodeNo {
|
||||
let components: Vec<_> = local_path
|
||||
.components()
|
||||
.filter(|c| matches!(c, std::path::Component::Normal(_)))
|
||||
.collect();
|
||||
if components.len() <= 1 {
|
||||
return INodeNo::ROOT;
|
||||
}
|
||||
let mut parent_path = String::new();
|
||||
for (i, comp) in components[..components.len() - 1].iter().enumerate() {
|
||||
if i > 0 {
|
||||
parent_path.push('/');
|
||||
}
|
||||
parent_path.push_str(comp.as_os_str().to_str().unwrap_or_default());
|
||||
}
|
||||
|
||||
return virtual_inode(&parent_path);
|
||||
}
|
||||
|
||||
pub fn ensure_virtual_dirs(
|
||||
local_path: &Path,
|
||||
source: &Path,
|
||||
map: &mut BTreeMap<INodeNo, Item>,
|
||||
) -> INodeNo {
|
||||
let components: Vec<_> = local_path
|
||||
.components()
|
||||
.filter(|c| matches!(c, std::path::Component::Normal(_)))
|
||||
.collect();
|
||||
|
||||
if components.len() <= 1 {
|
||||
return INodeNo::ROOT;
|
||||
}
|
||||
|
||||
let mut current_parent = INodeNo::ROOT;
|
||||
let mut current_path = String::new();
|
||||
|
||||
for component in &components[..components.len() - 1] {
|
||||
let comp_str = component.as_os_str().to_str().unwrap_or_default();
|
||||
if !current_path.is_empty() {
|
||||
current_path.push('/');
|
||||
}
|
||||
current_path.push_str(comp_str);
|
||||
|
||||
let virt_ino = virtual_inode(¤t_path);
|
||||
|
||||
if !map.contains_key(&virt_ino) {
|
||||
let virt_item = Item::new(
|
||||
virt_ino,
|
||||
current_parent,
|
||||
comp_str.to_string(),
|
||||
source.to_path_buf(),
|
||||
PathBuf::from(¤t_path),
|
||||
FileType::Directory,
|
||||
FileAttrs::from(&fs::metadata(source).unwrap()),
|
||||
None,
|
||||
);
|
||||
map.insert(virt_ino, virt_item);
|
||||
}
|
||||
|
||||
current_parent = virt_ino;
|
||||
}
|
||||
|
||||
return current_parent;
|
||||
}
|
||||
|
||||
pub fn restore_virtual_paths(
|
||||
snapshot: &mut BTreeMap<INodeNo, Item>,
|
||||
db_items: &std::collections::HashMap<i64, Model>,
|
||||
source: &Path,
|
||||
) {
|
||||
let restorations: Vec<(INodeNo, String, PathBuf)> = db_items
|
||||
.values()
|
||||
.filter_map(|db_item| {
|
||||
let ino = INodeNo(db_item.inode as u64);
|
||||
if ino == INodeNo::ROOT {
|
||||
return None;
|
||||
}
|
||||
snapshot
|
||||
.get(&ino)
|
||||
.filter(|item| item.hash as i64 == db_item.hash)
|
||||
.map(|_| {
|
||||
(
|
||||
ino,
|
||||
db_item.name.clone(),
|
||||
PathBuf::from(&db_item.local_path),
|
||||
)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
for (_, _, local_path) in &restorations {
|
||||
ensure_virtual_dirs(local_path, source, snapshot);
|
||||
}
|
||||
for (ino, name, local_path) in restorations {
|
||||
if let Some(item) = snapshot.get_mut(&ino) {
|
||||
item.name = name;
|
||||
item.parent_inode = parent_inode_from_path(&local_path);
|
||||
item.local_path = local_path;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Computes the new (name, local_path) for an Item based on its updated
|
||||
/// `MusicMetadata`. Mirrors the layout logic in
|
||||
/// `origins::local::snapshot::read_into_map`:
|
||||
/// - artist dir = `album_artist` if set, else `artists.join("-")`
|
||||
/// - if `track_title` is non-empty, the filename becomes
|
||||
/// `{track_number:02} - {track_title}.{ext}` (zero-padded track number
|
||||
/// when > 0, otherwise just `{track_title}.{ext}`), ext preserved from
|
||||
/// `current_name`
|
||||
/// - otherwise the filename is unchanged
|
||||
/// Returns `(new_name, new_local_path)`.
|
||||
pub fn compute_new_layout(current_name: &str, mm: &MusicMetadata) -> (String, PathBuf) {
|
||||
let artist_dir = artist_dir(mm);
|
||||
let filename = match &mm.track_title {
|
||||
title if !title.is_empty() => {
|
||||
let stem = if mm.track_number > 0 {
|
||||
format!("{:02} - {}", mm.track_number, title)
|
||||
} else {
|
||||
title.clone()
|
||||
};
|
||||
rename_with_extension(current_name, &stem)
|
||||
}
|
||||
_ => current_name.to_string(),
|
||||
};
|
||||
|
||||
let mut local_path = PathBuf::new();
|
||||
local_path.push(&artist_dir);
|
||||
local_path.push(&mm.album);
|
||||
local_path.push(&filename);
|
||||
|
||||
(filename, local_path)
|
||||
}
|
||||
|
||||
fn artist_dir(mm: &MusicMetadata) -> String {
|
||||
match &mm.album_artist {
|
||||
Some(a) if !a.is_empty() => a.clone(),
|
||||
_ => mm.artist.join("-"),
|
||||
}
|
||||
}
|
||||
|
||||
fn rename_with_extension(current_name: &str, new_stem: &str) -> String {
|
||||
let current_path = Path::new(current_name);
|
||||
match current_path.extension() {
|
||||
Some(ext) => format!("{new_stem}.{}", ext.to_string_lossy()),
|
||||
None => new_stem.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the virtual inodes of directories in the `old_local_path`'s
|
||||
/// parent chain that will have no remaining files after this item moves
|
||||
/// away. Walks the chain from the deepest directory upward; stops at the
|
||||
/// first directory that still has other files living in it.
|
||||
///
|
||||
/// `moving_inode` is the actual inode of the file being relocated (the
|
||||
/// request's `inode`), used to exclude it from the "any other files here?"
|
||||
/// check. `old_local_path` is the LOCAL (virtual FUSE) path of the file
|
||||
/// before its metadata changed.
|
||||
pub fn find_orphaned_dirs(
|
||||
files: &BTreeMap<INodeNo, Item>,
|
||||
moving_inode: INodeNo,
|
||||
old_local_path: &Path,
|
||||
) -> Vec<INodeNo> {
|
||||
let components: Vec<_> = old_local_path
|
||||
.components()
|
||||
.filter(|c| matches!(c, std::path::Component::Normal(_)))
|
||||
.collect();
|
||||
if components.len() <= 1 {
|
||||
return vec![];
|
||||
}
|
||||
|
||||
let mut orphaned = vec![];
|
||||
for depth in (1..components.len()).rev() {
|
||||
let dir_path: PathBuf = components[..depth].iter().map(|c| c.as_os_str()).collect();
|
||||
let dir_path_str = dir_path.to_string_lossy().into_owned();
|
||||
let dir_inode = virtual_inode(&dir_path_str);
|
||||
|
||||
// Path-prefix match: a file at "Artist/Album2/b.flac" still keeps
|
||||
// "Artist" non-orphan even though its direct parent is "Artist/Album2".
|
||||
let prefix = format!("{dir_path_str}/");
|
||||
let has_other_files = files.values().any(|item| {
|
||||
item.inode != moving_inode
|
||||
&& item.file_type == FileType::File
|
||||
&& item.local_path.starts_with(&prefix)
|
||||
});
|
||||
if has_other_files {
|
||||
break;
|
||||
}
|
||||
orphaned.push(dir_inode);
|
||||
}
|
||||
orphaned
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::collections::HashMap;
|
||||
|
||||
// ---- compute_new_layout ----
|
||||
|
||||
fn mm(artist: &str, album: &str, title: &str) -> MusicMetadata {
|
||||
MusicMetadata {
|
||||
artist: vec![artist.to_string()],
|
||||
album_artist: None,
|
||||
album: album.to_string(),
|
||||
track_number: 1,
|
||||
track_title: title.to_string(),
|
||||
other_tags: vec![],
|
||||
header: vec![],
|
||||
picture_block_headers: vec![],
|
||||
picture_data_ranges: vec![],
|
||||
real_audio_start: 0,
|
||||
vorbis_comment_offset: 0,
|
||||
vorbis_comment_length: 0,
|
||||
}
|
||||
}
|
||||
|
||||
fn file_item(inode: u64, name: &str, local_path: &str) -> Item {
|
||||
Item {
|
||||
inode: INodeNo(inode),
|
||||
parent_inode: parent_inode_from_path(Path::new(local_path)),
|
||||
name: name.to_string(),
|
||||
original_path: format!("/torrent/{name}").into(),
|
||||
local_path: local_path.into(),
|
||||
file_type: FileType::File,
|
||||
attrs: FileAttrs {
|
||||
size: 1,
|
||||
blocks: 1,
|
||||
atime: std::time::SystemTime::UNIX_EPOCH,
|
||||
mtime: std::time::SystemTime::UNIX_EPOCH,
|
||||
ctime: std::time::SystemTime::UNIX_EPOCH,
|
||||
crtime: std::time::SystemTime::UNIX_EPOCH,
|
||||
perm: 0o644,
|
||||
nlink: 1,
|
||||
uid: 0,
|
||||
gid: 0,
|
||||
rdev: 0,
|
||||
blksize: 4096,
|
||||
},
|
||||
music_metadata: None,
|
||||
hash: inode,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_renames_file_when_track_title_provided() {
|
||||
let mm = mm(
|
||||
"Pink Floyd",
|
||||
"Wish You Were Here",
|
||||
"Shine On You Crazy Diamond",
|
||||
);
|
||||
let (name, path) = compute_new_layout("01-track.flac", &mm);
|
||||
assert_eq!(name, "01 - Shine On You Crazy Diamond.flac");
|
||||
assert_eq!(
|
||||
path,
|
||||
PathBuf::from("Pink Floyd/Wish You Were Here/01 - Shine On You Crazy Diamond.flac")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_preserves_extension_through_rename() {
|
||||
let mm = mm("A", "B", "New Title");
|
||||
let (name, _) = compute_new_layout("old.mp3", &mm);
|
||||
assert_eq!(name, "01 - New Title.mp3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_handles_multi_dot_extensions() {
|
||||
let mm = mm("A", "B", "New");
|
||||
let (name, _) = compute_new_layout("old.tar.gz", &mm);
|
||||
assert_eq!(name, "01 - New.gz", "only the final extension is preserved");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_keeps_original_name_when_track_title_empty() {
|
||||
let mut metadata = mm("A", "B", "ignored");
|
||||
metadata.track_title = String::new();
|
||||
let (name, path) = compute_new_layout("original-name.flac", &metadata);
|
||||
assert_eq!(name, "original-name.flac");
|
||||
assert_eq!(path, PathBuf::from("A/B/original-name.flac"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_uses_album_artist_when_present() {
|
||||
let mut metadata = mm("Secondary", "Album", "Title");
|
||||
metadata.album_artist = Some("Primary Artist".to_string());
|
||||
let (_, path) = compute_new_layout("track.flac", &metadata);
|
||||
assert_eq!(path, PathBuf::from("Primary Artist/Album/01 - Title.flac"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compute_new_layout_joins_multiple_artists_with_dash_when_no_album_artist() {
|
||||
let mut metadata = mm("Foo", "Album", "Title");
|
||||
metadata.artist = vec!["Foo".to_string(), "Bar".to_string()];
|
||||
let (_, path) = compute_new_layout("track.flac", &metadata);
|
||||
assert_eq!(path, PathBuf::from("Foo-Bar/Album/01 - Title.flac"));
|
||||
}
|
||||
|
||||
// ---- find_orphaned_dirs ----
|
||||
|
||||
#[test]
|
||||
fn find_orphaned_dirs_returns_both_album_and_artist_when_last_file_moves() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
file_item(100, "track.flac", "Artist/Album/track.flac"),
|
||||
);
|
||||
|
||||
let orphaned =
|
||||
find_orphaned_dirs(&files, INodeNo(100), Path::new("Artist/Album/track.flac"));
|
||||
assert_eq!(orphaned.len(), 2);
|
||||
assert!(orphaned.contains(&virtual_inode("Artist/Album")));
|
||||
assert!(orphaned.contains(&virtual_inode("Artist")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_orphaned_dirs_returns_album_only_when_other_album_keeps_artist() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
file_item(100, "a.flac", "Artist/Album1/a.flac"),
|
||||
);
|
||||
files.insert(
|
||||
INodeNo(101),
|
||||
file_item(101, "b.flac", "Artist/Album2/b.flac"),
|
||||
);
|
||||
|
||||
let orphaned = find_orphaned_dirs(&files, INodeNo(100), Path::new("Artist/Album1/a.flac"));
|
||||
assert_eq!(orphaned, vec![virtual_inode("Artist/Album1")]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_orphaned_dirs_returns_nothing_when_other_files_remain_in_same_album() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
file_item(100, "a.flac", "Artist/Album/a.flac"),
|
||||
);
|
||||
files.insert(
|
||||
INodeNo(101),
|
||||
file_item(101, "b.flac", "Artist/Album/b.flac"),
|
||||
);
|
||||
|
||||
let orphaned = find_orphaned_dirs(&files, INodeNo(100), Path::new("Artist/Album/a.flac"));
|
||||
assert!(orphaned.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_orphaned_dirs_returns_nothing_for_file_at_root() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(INodeNo(100), file_item(100, "track.flac", "track.flac"));
|
||||
|
||||
let orphaned = find_orphaned_dirs(&files, INodeNo(100), Path::new("track.flac"));
|
||||
assert!(orphaned.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_orphaned_dirs_excludes_moving_file_from_other_file_check() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
file_item(100, "only.flac", "Artist/Album/only.flac"),
|
||||
);
|
||||
// If we forgot to exclude the moving inode, this test would return
|
||||
// empty (thinking the dir is still populated). Must return both dirs.
|
||||
let orphaned =
|
||||
find_orphaned_dirs(&files, INodeNo(100), Path::new("Artist/Album/only.flac"));
|
||||
assert_eq!(orphaned.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn virtual_inode_deterministic() {
|
||||
let ino1 = virtual_inode("foo");
|
||||
let ino2 = virtual_inode("foo");
|
||||
assert_eq!(ino1, ino2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn virtual_inode_high_bit_set() {
|
||||
let ino = virtual_inode("test");
|
||||
assert_eq!(ino.0 & (1u64 << 63), 1u64 << 63);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn virtual_inode_different_inputs() {
|
||||
let ino1 = virtual_inode("foo");
|
||||
let ino2 = virtual_inode("bar");
|
||||
assert_ne!(ino1, ino2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parent_inode_single_component() {
|
||||
let ino = parent_inode_from_path(Path::new("file.txt"));
|
||||
assert_eq!(ino, INodeNo::ROOT);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parent_inode_multi_component() {
|
||||
let ino = parent_inode_from_path(Path::new("a/b/c"));
|
||||
let expected = virtual_inode("a/b");
|
||||
assert_eq!(ino, expected);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parent_inode_absolute_path_filtered() {
|
||||
let ino = parent_inode_from_path(Path::new("/file"));
|
||||
assert_eq!(ino, INodeNo::ROOT);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ensure_virtual_dirs_single_component() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
let mut map = BTreeMap::new();
|
||||
|
||||
let ino = ensure_virtual_dirs(Path::new("file.txt"), source, &mut map);
|
||||
assert_eq!(ino, INodeNo::ROOT);
|
||||
assert!(map.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ensure_virtual_dirs_creates_hierarchy() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
let mut map = BTreeMap::new();
|
||||
|
||||
let ino = ensure_virtual_dirs(Path::new("a/b/file"), source, &mut map);
|
||||
|
||||
let a_ino = virtual_inode("a");
|
||||
let ab_ino = virtual_inode("a/b");
|
||||
|
||||
assert_eq!(ino, ab_ino);
|
||||
assert_eq!(map.len(), 2);
|
||||
assert!(map.contains_key(&a_ino));
|
||||
assert!(map.contains_key(&ab_ino));
|
||||
|
||||
let a_item = &map[&a_ino];
|
||||
assert_eq!(a_item.parent_inode, INodeNo::ROOT);
|
||||
|
||||
let ab_item = &map[&ab_ino];
|
||||
assert_eq!(ab_item.parent_inode, a_ino);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ensure_virtual_dirs_idempotent() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
let mut map = BTreeMap::new();
|
||||
|
||||
ensure_virtual_dirs(Path::new("a/b/file"), source, &mut map);
|
||||
let size_after_first = map.len();
|
||||
|
||||
ensure_virtual_dirs(Path::new("a/b/file"), source, &mut map);
|
||||
let size_after_second = map.len();
|
||||
|
||||
assert_eq!(size_after_first, size_after_second);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_virtual_paths_skips_root() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
let mut snapshot = BTreeMap::new();
|
||||
|
||||
let real_ino = INodeNo(42);
|
||||
let real_item = Item::new(
|
||||
real_ino,
|
||||
INodeNo::ROOT,
|
||||
"real_file".to_string(),
|
||||
source.to_path_buf(),
|
||||
PathBuf::from("real_file"),
|
||||
FileType::File,
|
||||
FileAttrs::from(&fs::metadata(source).unwrap()),
|
||||
None,
|
||||
);
|
||||
snapshot.insert(real_ino, real_item);
|
||||
|
||||
let mut db_items = HashMap::new();
|
||||
db_items.insert(
|
||||
1i64,
|
||||
Model {
|
||||
inode: 1,
|
||||
hash: 0,
|
||||
name: "root".to_string(),
|
||||
original_path: "/tmp/test".to_string(),
|
||||
local_path: "/tmp/test".to_string(),
|
||||
file_type: "directory".to_string(),
|
||||
},
|
||||
);
|
||||
db_items.insert(
|
||||
42i64,
|
||||
Model {
|
||||
inode: 42,
|
||||
hash: 0,
|
||||
name: "real_file".to_string(),
|
||||
original_path: "real_file".to_string(),
|
||||
local_path: "real_file".to_string(),
|
||||
file_type: "file".to_string(),
|
||||
},
|
||||
);
|
||||
|
||||
restore_virtual_paths(&mut snapshot, &db_items, source);
|
||||
|
||||
for item in snapshot.values() {
|
||||
assert_ne!(item.name, "/");
|
||||
assert_ne!(item.name, "tmp");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,562 @@
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
sync::{Arc, Mutex},
|
||||
time::SystemTime,
|
||||
};
|
||||
|
||||
use fuser::INodeNo;
|
||||
use musicfs::control::ClientControlServiceImpl;
|
||||
use musicfs::db::entities;
|
||||
use musicfs::item::{FileType, Item};
|
||||
use musicfs::music::db::entities::{artists, music_metadata as mm_entity};
|
||||
use musicfs::music::metadata::MusicMetadata;
|
||||
use musicfs::origins::attrs::FileAttrs;
|
||||
use musicfs_proto::{
|
||||
ClientControl, GetMusicMetadataRequest, ListFilesRequest, UpdateMusicMetadataRequest,
|
||||
};
|
||||
use sea_orm::{DatabaseBackend, MockDatabase, MockExecResult};
|
||||
use tonic::{Code, Request};
|
||||
|
||||
// ── Test helpers ───────────────────────────────────────────────────────
|
||||
|
||||
fn make_attrs() -> FileAttrs {
|
||||
FileAttrs {
|
||||
size: 4096,
|
||||
blocks: 8,
|
||||
atime: SystemTime::UNIX_EPOCH,
|
||||
mtime: SystemTime::UNIX_EPOCH,
|
||||
ctime: SystemTime::UNIX_EPOCH,
|
||||
crtime: SystemTime::UNIX_EPOCH,
|
||||
perm: 0o644,
|
||||
nlink: 1,
|
||||
uid: 0,
|
||||
gid: 0,
|
||||
rdev: 0,
|
||||
blksize: 4096,
|
||||
}
|
||||
}
|
||||
|
||||
fn make_flac_item(inode: u64, name: &str, metadata: Option<MusicMetadata>) -> Item {
|
||||
Item {
|
||||
inode: INodeNo(inode),
|
||||
parent_inode: INodeNo(1),
|
||||
name: name.to_string(),
|
||||
original_path: format!("/{name}").into(),
|
||||
local_path: format!("Artist/Album/{name}").into(),
|
||||
file_type: FileType::File,
|
||||
attrs: make_attrs(),
|
||||
music_metadata: metadata,
|
||||
hash: inode,
|
||||
}
|
||||
}
|
||||
|
||||
fn make_flac_metadata() -> MusicMetadata {
|
||||
MusicMetadata {
|
||||
artist: vec!["Test Artist".to_string()],
|
||||
album_artist: Some("Test Artist".to_string()),
|
||||
album: "Test Album".to_string(),
|
||||
track_number: 1,
|
||||
track_title: "Test Track".to_string(),
|
||||
other_tags: vec![],
|
||||
header: vec![],
|
||||
picture_block_headers: vec![],
|
||||
picture_data_ranges: vec![],
|
||||
real_audio_start: 0,
|
||||
vorbis_comment_offset: 0,
|
||||
vorbis_comment_length: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Mock DB for get tests (no INSERT operations expected).
|
||||
fn mock_db_readonly() -> sea_orm::DatabaseConnection {
|
||||
MockDatabase::new(DatabaseBackend::Postgres).into_connection()
|
||||
}
|
||||
|
||||
/// Mock DB for update with one artist.
|
||||
/// save_music_metadata does: DELETE, INSERT music_metadata, INSERT artists.
|
||||
/// persist_layout_changes does: UPDATE items (RETURNING), then one DELETE
|
||||
/// per orphaned virtual directory (worst case = 2: artist + album dirs).
|
||||
fn mock_db_update_with_artist(inode: i64) -> sea_orm::DatabaseConnection {
|
||||
MockDatabase::new(DatabaseBackend::Postgres)
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.append_query_results([vec![mm_entity::Model {
|
||||
inode,
|
||||
track_title: String::new(),
|
||||
album: String::new(),
|
||||
track_number: 0,
|
||||
header: vec![],
|
||||
real_audio_start: 0,
|
||||
}]])
|
||||
.append_query_results([vec![artists::Model {
|
||||
inode,
|
||||
artist: String::new(),
|
||||
}]])
|
||||
.append_query_results([vec![entities::Model {
|
||||
inode,
|
||||
name: String::new(),
|
||||
original_path: String::new(),
|
||||
local_path: String::new(),
|
||||
file_type: "file".to_string(),
|
||||
hash: 0,
|
||||
}]])
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.into_connection()
|
||||
}
|
||||
|
||||
/// Mock DB for update with no artists.
|
||||
/// save_music_metadata does: DELETE, INSERT music_metadata only.
|
||||
/// persist_layout_changes does: UPDATE items (RETURNING), then up to two
|
||||
/// DELETEs for orphaned virtual directories.
|
||||
fn mock_db_update_no_artist(inode: i64) -> sea_orm::DatabaseConnection {
|
||||
MockDatabase::new(DatabaseBackend::Postgres)
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.append_query_results([vec![mm_entity::Model {
|
||||
inode,
|
||||
track_title: String::new(),
|
||||
album: String::new(),
|
||||
track_number: 0,
|
||||
header: vec![],
|
||||
real_audio_start: 0,
|
||||
}]])
|
||||
.append_query_results([vec![entities::Model {
|
||||
inode,
|
||||
name: String::new(),
|
||||
original_path: String::new(),
|
||||
local_path: String::new(),
|
||||
file_type: "file".to_string(),
|
||||
hash: 0,
|
||||
}]])
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.append_exec_results([MockExecResult {
|
||||
rows_affected: 1,
|
||||
..Default::default()
|
||||
}])
|
||||
.into_connection()
|
||||
}
|
||||
|
||||
fn make_dir_item(inode: u64, name: &str) -> Item {
|
||||
Item {
|
||||
inode: INodeNo(inode),
|
||||
parent_inode: INodeNo(1),
|
||||
name: name.to_string(),
|
||||
original_path: format!("/{name}").into(),
|
||||
local_path: name.into(),
|
||||
file_type: FileType::Directory,
|
||||
attrs: make_attrs(),
|
||||
music_metadata: None,
|
||||
hash: inode,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_removes_orphaned_virtual_dirs_when_last_file_moves_out() {
|
||||
let source_dir = tempfile::tempdir().unwrap();
|
||||
let source = source_dir.path();
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(0),
|
||||
Item {
|
||||
inode: INodeNo(0),
|
||||
parent_inode: INodeNo::ROOT,
|
||||
name: "/".to_string(),
|
||||
original_path: source.to_path_buf(),
|
||||
local_path: "/".into(),
|
||||
file_type: FileType::Directory,
|
||||
attrs: make_attrs(),
|
||||
music_metadata: None,
|
||||
hash: 0,
|
||||
},
|
||||
);
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "song.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files.clone(), mock_db_update_with_artist(100));
|
||||
|
||||
// Pre-create the virtual dirs for the file's current layout so we can
|
||||
// assert they get pruned after the metadata update moves the file.
|
||||
{
|
||||
use musicfs::virtual_dirs::{ensure_virtual_dirs, virtual_inode};
|
||||
let mut guard = files.lock().unwrap();
|
||||
ensure_virtual_dirs(
|
||||
std::path::Path::new("Artist/Album/song.flac"),
|
||||
source,
|
||||
&mut guard,
|
||||
);
|
||||
assert!(guard.contains_key(&virtual_inode("Artist")));
|
||||
assert!(guard.contains_key(&virtual_inode("Artist/Album")));
|
||||
}
|
||||
|
||||
svc.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 100,
|
||||
artist: vec!["New Artist".to_string()],
|
||||
album_artist: Some("New Artist".to_string()),
|
||||
album: "New Album".to_string(),
|
||||
track_number: 1,
|
||||
track_title: "New Title".to_string(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect("update should succeed");
|
||||
|
||||
let guard = files.lock().unwrap();
|
||||
use musicfs::virtual_dirs::virtual_inode;
|
||||
assert!(
|
||||
!guard.contains_key(&virtual_inode("Artist")),
|
||||
"old Artist dir must be removed when no files remain under it"
|
||||
);
|
||||
assert!(
|
||||
!guard.contains_key(&virtual_inode("Artist/Album")),
|
||||
"old Album dir must be removed"
|
||||
);
|
||||
assert!(guard.contains_key(&virtual_inode("New Artist")));
|
||||
assert!(guard.contains_key(&virtual_inode("New Artist/New Album")));
|
||||
|
||||
let item = guard.get(&INodeNo(100)).expect("file still present");
|
||||
assert_eq!(
|
||||
item.local_path,
|
||||
std::path::PathBuf::from("New Artist/New Album/01 - New Title.flac")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_keeps_shared_artist_dir_when_other_album_remains() {
|
||||
let source_dir = tempfile::tempdir().unwrap();
|
||||
let source = source_dir.path();
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(0),
|
||||
Item {
|
||||
inode: INodeNo(0),
|
||||
parent_inode: INodeNo::ROOT,
|
||||
name: "/".to_string(),
|
||||
original_path: source.to_path_buf(),
|
||||
local_path: "/".into(),
|
||||
file_type: FileType::Directory,
|
||||
attrs: make_attrs(),
|
||||
music_metadata: None,
|
||||
hash: 0,
|
||||
},
|
||||
);
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "a.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
files.insert(
|
||||
INodeNo(101),
|
||||
make_flac_item(101, "b.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
files.get_mut(&INodeNo(101)).unwrap().local_path = "Artist/Other Album/b.flac".into();
|
||||
|
||||
// Pre-create virtual dirs for both files' layouts: pruning after the
|
||||
// update should remove Album/ (file 100's old parent) but keep Artist/
|
||||
// (still has Other Album/ with file 101 in it).
|
||||
{
|
||||
use musicfs::virtual_dirs::ensure_virtual_dirs;
|
||||
ensure_virtual_dirs(
|
||||
std::path::Path::new("Artist/Album/a.flac"),
|
||||
source,
|
||||
&mut files,
|
||||
);
|
||||
ensure_virtual_dirs(
|
||||
std::path::Path::new("Artist/Other Album/b.flac"),
|
||||
source,
|
||||
&mut files,
|
||||
);
|
||||
}
|
||||
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files.clone(), mock_db_update_with_artist(100));
|
||||
|
||||
svc.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 100,
|
||||
artist: vec!["New Artist".to_string()],
|
||||
album_artist: Some("New Artist".to_string()),
|
||||
album: "New Album".to_string(),
|
||||
track_number: 1,
|
||||
track_title: "New Title".to_string(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect("update should succeed");
|
||||
|
||||
let guard = files.lock().unwrap();
|
||||
use musicfs::virtual_dirs::virtual_inode;
|
||||
assert!(
|
||||
guard.contains_key(&virtual_inode("Artist")),
|
||||
"Artist dir must remain while Other Album still has files"
|
||||
);
|
||||
assert!(
|
||||
!guard.contains_key(&virtual_inode("Artist/Album")),
|
||||
"Artist/Album dir must be removed"
|
||||
);
|
||||
}
|
||||
|
||||
// ── ListFiles ─────────────────────────────────────────────────────────
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_files_returns_all_files_with_full_field_projection() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "a.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
files.insert(INodeNo(101), make_flac_item(101, "b.flac", None));
|
||||
// Directory entries must be filtered out.
|
||||
files.insert(INodeNo(2), make_dir_item(2, "Artist"));
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let resp = svc
|
||||
.list_files(Request::new(ListFilesRequest {}))
|
||||
.await
|
||||
.expect("list should succeed");
|
||||
|
||||
let mut entries = resp.into_inner().files;
|
||||
assert_eq!(entries.len(), 2, "directories must be filtered out");
|
||||
entries.sort_by_key(|e| e.inode);
|
||||
|
||||
assert_eq!(entries[0].inode, 100);
|
||||
assert_eq!(entries[0].name, "a.flac");
|
||||
assert_eq!(entries[0].original_path, "/a.flac");
|
||||
assert_eq!(entries[0].local_path, "Artist/Album/a.flac");
|
||||
assert!(entries[0].metadata.is_some());
|
||||
|
||||
assert_eq!(entries[1].inode, 101);
|
||||
assert_eq!(entries[1].name, "b.flac");
|
||||
assert!(entries[1].metadata.is_none(), "no metadata on this file");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_files_empty_map_returns_empty_list() {
|
||||
let files = Arc::new(Mutex::new(BTreeMap::new()));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let resp = svc
|
||||
.list_files(Request::new(ListFilesRequest {}))
|
||||
.await
|
||||
.expect("list should succeed");
|
||||
|
||||
assert!(resp.into_inner().files.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_files_skips_directories_and_root() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(INodeNo(0), make_dir_item(0, "/")); // root
|
||||
files.insert(INodeNo(1), make_dir_item(1, "Artist"));
|
||||
files.insert(INodeNo(2), make_dir_item(2, "Album"));
|
||||
files.insert(INodeNo(100), make_flac_item(100, "track.flac", None));
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let resp = svc
|
||||
.list_files(Request::new(ListFilesRequest {}))
|
||||
.await
|
||||
.expect("list should succeed");
|
||||
|
||||
let inodes: Vec<u64> = resp
|
||||
.into_inner()
|
||||
.files
|
||||
.into_iter()
|
||||
.map(|e| e.inode)
|
||||
.collect();
|
||||
assert_eq!(inodes, vec![100]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_returns_metadata_for_valid_inode() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "song.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let resp = svc
|
||||
.get_music_metadata(Request::new(GetMusicMetadataRequest { inode: 100 }))
|
||||
.await
|
||||
.expect("get should succeed");
|
||||
|
||||
let md = resp.into_inner().metadata.expect("metadata present");
|
||||
assert_eq!(md.track_title, "Test Track");
|
||||
assert_eq!(md.album, "Test Album");
|
||||
assert_eq!(md.artist, vec!["Test Artist".to_string()]);
|
||||
assert_eq!(md.track_number, 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_returns_not_found_for_missing_inode() {
|
||||
let files = Arc::new(Mutex::new(BTreeMap::new()));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let err = svc
|
||||
.get_music_metadata(Request::new(GetMusicMetadataRequest { inode: 999 }))
|
||||
.await
|
||||
.expect_err("should be NOT_FOUND");
|
||||
|
||||
assert_eq!(err.code(), Code::NotFound);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_returns_not_found_when_item_has_no_metadata() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(INodeNo(100), make_flac_item(100, "song.flac", None));
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let err = svc
|
||||
.get_music_metadata(Request::new(GetMusicMetadataRequest { inode: 100 }))
|
||||
.await
|
||||
.expect_err("should be NOT_FOUND");
|
||||
|
||||
assert_eq!(err.code(), Code::NotFound);
|
||||
}
|
||||
|
||||
// ── UpdateMusicMetadata ───────────────────────────────────────────────
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_changes_tags_and_response_reflects_new_values() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "song.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files.clone(), mock_db_update_with_artist(100));
|
||||
|
||||
let resp = svc
|
||||
.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 100,
|
||||
artist: vec!["Updated Artist".to_string()],
|
||||
album_artist: Some("Updated Artist".to_string()),
|
||||
album: "Updated Album".to_string(),
|
||||
track_number: 7,
|
||||
track_title: "Updated Title".to_string(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect("update should succeed");
|
||||
|
||||
let md = resp.into_inner().metadata.expect("metadata present");
|
||||
assert_eq!(md.track_title, "Updated Title");
|
||||
assert_eq!(md.album, "Updated Album");
|
||||
assert_eq!(md.artist, vec!["Updated Artist".to_string()]);
|
||||
assert_eq!(md.track_number, 7);
|
||||
|
||||
let guard = files.lock().unwrap();
|
||||
let item = guard.get(&INodeNo(100)).expect("item still in map");
|
||||
let mm = item.music_metadata.as_ref().expect("metadata present");
|
||||
assert_eq!(mm.track_title, "Updated Title");
|
||||
assert_eq!(mm.album, "Updated Album");
|
||||
assert_eq!(mm.artist, vec!["Updated Artist".to_string()]);
|
||||
assert_eq!(mm.track_number, 7);
|
||||
|
||||
// Layout was recomputed from the new metadata.
|
||||
assert_eq!(item.name, "07 - Updated Title.flac");
|
||||
assert_eq!(
|
||||
item.local_path,
|
||||
std::path::PathBuf::from("Updated Artist/Updated Album/07 - Updated Title.flac")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_returns_not_found_for_missing_inode() {
|
||||
let files = Arc::new(Mutex::new(BTreeMap::new()));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let err = svc
|
||||
.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 999,
|
||||
artist: vec![],
|
||||
album_artist: None,
|
||||
album: String::new(),
|
||||
track_number: 0,
|
||||
track_title: String::new(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect_err("should be NOT_FOUND");
|
||||
|
||||
assert_eq!(err.code(), Code::NotFound);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_returns_not_found_when_item_has_no_metadata() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(INodeNo(100), make_flac_item(100, "song.flac", None));
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files, mock_db_readonly());
|
||||
|
||||
let err = svc
|
||||
.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 100,
|
||||
artist: vec!["X".to_string()],
|
||||
album_artist: None,
|
||||
album: "X".to_string(),
|
||||
track_number: 1,
|
||||
track_title: "X".to_string(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect_err("should be NOT_FOUND");
|
||||
|
||||
assert_eq!(err.code(), Code::NotFound);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_clears_tags_when_empty_values_sent() {
|
||||
let mut files = BTreeMap::new();
|
||||
files.insert(
|
||||
INodeNo(100),
|
||||
make_flac_item(100, "song.flac", Some(make_flac_metadata())),
|
||||
);
|
||||
let files = Arc::new(Mutex::new(files));
|
||||
let svc = ClientControlServiceImpl::new(files.clone(), mock_db_update_no_artist(100));
|
||||
|
||||
let resp = svc
|
||||
.update_music_metadata(Request::new(UpdateMusicMetadataRequest {
|
||||
inode: 100,
|
||||
artist: vec![],
|
||||
album_artist: None,
|
||||
album: String::new(),
|
||||
track_number: 0,
|
||||
track_title: String::new(),
|
||||
other_tags: vec![],
|
||||
}))
|
||||
.await
|
||||
.expect("update should succeed");
|
||||
|
||||
let md = resp.into_inner().metadata.expect("metadata present");
|
||||
assert!(md.artist.is_empty());
|
||||
assert!(md.album.is_empty());
|
||||
assert!(md.track_title.is_empty());
|
||||
|
||||
let guard = files.lock().unwrap();
|
||||
let mm = guard
|
||||
.get(&INodeNo(100))
|
||||
.unwrap()
|
||||
.music_metadata
|
||||
.as_ref()
|
||||
.unwrap();
|
||||
assert!(mm.artist.is_empty());
|
||||
assert!(mm.album.is_empty());
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
use std::io::Write;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use musicfs::origins::local::file_io::{assemble_flac_read, read_bytes_at};
|
||||
|
||||
fn file_reader(path: PathBuf) -> impl Fn(u64, usize) -> std::io::Result<Vec<u8>> {
|
||||
return move |offset, len| read_bytes_at(&path, offset, len);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_picture_header_region() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let data: Vec<u8> = (0..100).collect();
|
||||
f.write_all(&data).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let header = b"ABCD";
|
||||
let pic_hdr = vec![0x86u8, 0x00, 0x00, 0x05];
|
||||
let pic_ranges = [(10u64, 5u64)];
|
||||
let real_audio_start = 50u64;
|
||||
|
||||
let reader = file_reader(path);
|
||||
let result = assemble_flac_read(
|
||||
&reader,
|
||||
header,
|
||||
&[pic_hdr.clone()],
|
||||
&pic_ranges,
|
||||
real_audio_start,
|
||||
4,
|
||||
4,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result, pic_hdr);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_picture_data_region() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let data: Vec<u8> = (0..100).collect();
|
||||
f.write_all(&data).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let header = b"ABCD";
|
||||
let pic_hdr = vec![0x86u8, 0x00, 0x00, 0x05];
|
||||
let pic_ranges = [(10u64, 5u64)];
|
||||
let real_audio_start = 50u64;
|
||||
|
||||
let reader = file_reader(path);
|
||||
let result = assemble_flac_read(
|
||||
&reader,
|
||||
header,
|
||||
&[pic_hdr.clone()],
|
||||
&pic_ranges,
|
||||
real_audio_start,
|
||||
8,
|
||||
5,
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(result, vec![10, 11, 12, 13, 14]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn assemble_flac_read_full_virtual_layout() {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
let data: Vec<u8> = (0..200).map(|i| i as u8).collect();
|
||||
f.write_all(&data).unwrap();
|
||||
f.flush().unwrap();
|
||||
let path = f.path().to_path_buf();
|
||||
|
||||
let header = vec![0xAA; 8];
|
||||
let real_audio_start = 100u64;
|
||||
|
||||
let reader = file_reader(path);
|
||||
let result = assemble_flac_read(&reader, &header, &[], &[], real_audio_start, 0, 18).unwrap();
|
||||
|
||||
let mut expected = vec![0xAA; 8];
|
||||
expected.extend_from_slice(&(100..110).map(|i| i as u8).collect::<Vec<u8>>());
|
||||
assert_eq!(result, expected);
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
use std::fs;
|
||||
use std::io::Write;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use fuser::INodeNo;
|
||||
|
||||
use musicfs::item::FileType;
|
||||
use musicfs::origins::local::snapshot::{build_snapshot, fill_fileset};
|
||||
|
||||
#[test]
|
||||
fn build_snapshot_empty_dir() {
|
||||
let source = tempfile::tempdir().unwrap();
|
||||
let dest = tempfile::tempdir().unwrap();
|
||||
|
||||
let snapshot = build_snapshot(source.path(), dest.path()).unwrap();
|
||||
|
||||
assert_eq!(snapshot.len(), 1);
|
||||
assert!(snapshot.contains_key(&INodeNo::ROOT));
|
||||
|
||||
let root = &snapshot[&INodeNo::ROOT];
|
||||
assert_eq!(root.inode, INodeNo::ROOT);
|
||||
assert_eq!(root.parent_inode, INodeNo::ROOT);
|
||||
assert_eq!(root.file_type, FileType::Directory);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_snapshot_flat_files() {
|
||||
let source = tempfile::tempdir().unwrap();
|
||||
let dest = tempfile::tempdir().unwrap();
|
||||
|
||||
let mut f = fs::File::create(source.path().join("a.txt")).unwrap();
|
||||
f.write_all(b"content a").unwrap();
|
||||
|
||||
let mut f = fs::File::create(source.path().join("b.txt")).unwrap();
|
||||
f.write_all(b"content b").unwrap();
|
||||
|
||||
let mut f = fs::File::create(source.path().join("c.txt")).unwrap();
|
||||
f.write_all(b"content c").unwrap();
|
||||
|
||||
let snapshot = build_snapshot(source.path(), dest.path()).unwrap();
|
||||
|
||||
assert_eq!(snapshot.len(), 4);
|
||||
assert!(snapshot.contains_key(&INodeNo::ROOT));
|
||||
|
||||
let file_entries: Vec<_> = snapshot
|
||||
.values()
|
||||
.filter(|item| item.file_type == FileType::File)
|
||||
.collect();
|
||||
assert_eq!(file_entries.len(), 3);
|
||||
|
||||
for file_entry in file_entries {
|
||||
assert_eq!(file_entry.parent_inode, INodeNo::ROOT);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_snapshot_nested_dirs() {
|
||||
let source = tempfile::tempdir().unwrap();
|
||||
let dest = tempfile::tempdir().unwrap();
|
||||
|
||||
fs::create_dir(source.path().join("subdir")).unwrap();
|
||||
let nested_path = source.path().join("subdir").join("file.txt");
|
||||
let mut f = fs::File::create(&nested_path).unwrap();
|
||||
f.write_all(b"nested content").unwrap();
|
||||
|
||||
let snapshot = build_snapshot(source.path(), dest.path()).unwrap();
|
||||
|
||||
assert!(snapshot.contains_key(&INodeNo::ROOT));
|
||||
|
||||
let file_entry = snapshot
|
||||
.values()
|
||||
.find(|item| item.name == "file.txt" && item.file_type == FileType::File)
|
||||
.expect("file.txt not found");
|
||||
|
||||
assert_eq!(file_entry.parent_inode, INodeNo::ROOT);
|
||||
assert_eq!(file_entry.original_path, nested_path);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fill_fileset_removes_deleted() {
|
||||
let source = tempfile::tempdir().unwrap();
|
||||
let dest = tempfile::tempdir().unwrap();
|
||||
|
||||
let mut f = fs::File::create(source.path().join("file1.txt")).unwrap();
|
||||
f.write_all(b"content 1").unwrap();
|
||||
|
||||
let mut f = fs::File::create(source.path().join("file2.txt")).unwrap();
|
||||
f.write_all(b"content 2").unwrap();
|
||||
|
||||
let initial_snapshot = build_snapshot(source.path(), dest.path()).unwrap();
|
||||
let map = Arc::new(Mutex::new(initial_snapshot));
|
||||
|
||||
assert_eq!(map.lock().unwrap().len(), 3);
|
||||
|
||||
fs::remove_file(source.path().join("file1.txt")).unwrap();
|
||||
|
||||
fill_fileset(&map, source.path(), dest.path());
|
||||
|
||||
let final_map = map.lock().unwrap();
|
||||
assert_eq!(final_map.len(), 2);
|
||||
assert!(final_map.contains_key(&INodeNo::ROOT));
|
||||
|
||||
let remaining_files: Vec<_> = final_map
|
||||
.values()
|
||||
.filter(|item| item.file_type == FileType::File)
|
||||
.collect();
|
||||
assert_eq!(remaining_files.len(), 1);
|
||||
assert_eq!(remaining_files[0].name, "file2.txt");
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
[package]
|
||||
name = "musicfs-core"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
symphonia.workspace = true
|
||||
twox-hash.workspace = true
|
||||
tracing.workspace = true
|
||||
tracing-subscriber.workspace = true
|
||||
tracing-appender.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile.workspace = true
|
||||
@@ -0,0 +1,88 @@
|
||||
use std::{
|
||||
fs,
|
||||
time::{Duration, SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
use std::os::unix::fs::{MetadataExt, PermissionsExt};
|
||||
|
||||
/// Owned, `Clone`-able snapshot of the file attributes that musicfs needs to
|
||||
/// serve FUSE `getattr` and to compute the per-item identity hash.
|
||||
///
|
||||
/// Decoupled from `std::fs::Metadata` so that non-disk origins (e.g. a future
|
||||
/// `NetworkOrigin` reading a server manifest) can construct equivalent
|
||||
/// attributes without a real inode on disk. The on-disk origin builds this
|
||||
/// via `From<&fs::Metadata>`; other origins build it from their own metadata
|
||||
/// source using the same field types.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct FileAttrs {
|
||||
pub size: u64,
|
||||
pub blocks: u64,
|
||||
pub atime: SystemTime,
|
||||
pub mtime: SystemTime,
|
||||
pub ctime: SystemTime,
|
||||
pub crtime: SystemTime,
|
||||
pub perm: u16,
|
||||
pub nlink: u32,
|
||||
pub uid: u32,
|
||||
pub gid: u32,
|
||||
pub rdev: u32,
|
||||
pub blksize: u32,
|
||||
}
|
||||
|
||||
impl From<&fs::Metadata> for FileAttrs {
|
||||
fn from(metadata: &fs::Metadata) -> Self {
|
||||
return FileAttrs {
|
||||
size: metadata.size(),
|
||||
blocks: metadata.blocks(),
|
||||
atime: metadata.accessed().unwrap_or(UNIX_EPOCH),
|
||||
mtime: metadata.modified().unwrap_or(UNIX_EPOCH),
|
||||
ctime: UNIX_EPOCH + Duration::from_secs(metadata.ctime() as u64),
|
||||
crtime: metadata.created().unwrap_or(UNIX_EPOCH),
|
||||
perm: metadata.permissions().mode() as u16,
|
||||
nlink: metadata.nlink() as u32,
|
||||
uid: metadata.uid(),
|
||||
gid: metadata.gid(),
|
||||
rdev: metadata.rdev() as u32,
|
||||
blksize: metadata.blksize() as u32,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn file_attrs_from_metadata_copies_size_and_block_fields() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = fs::metadata(tmp.path()).unwrap();
|
||||
|
||||
let attrs = FileAttrs::from(&metadata);
|
||||
|
||||
assert_eq!(attrs.size, metadata.size());
|
||||
assert_eq!(attrs.blocks, metadata.blocks());
|
||||
assert_eq!(attrs.blksize, metadata.blksize() as u32);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_attrs_from_metadata_copies_unix_ownership() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = fs::metadata(tmp.path()).unwrap();
|
||||
|
||||
let attrs = FileAttrs::from(&metadata);
|
||||
|
||||
assert_eq!(attrs.uid, metadata.uid());
|
||||
assert_eq!(attrs.gid, metadata.gid());
|
||||
assert_eq!(attrs.perm, metadata.permissions().mode() as u16);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_attrs_from_metadata_preserves_mtime() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let metadata = fs::metadata(tmp.path()).unwrap();
|
||||
|
||||
let attrs = FileAttrs::from(&metadata);
|
||||
|
||||
assert_eq!(attrs.mtime, metadata.modified().unwrap_or(UNIX_EPOCH));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
use std::hash::Hasher;
|
||||
|
||||
use twox_hash::XxHash64;
|
||||
|
||||
pub fn compute_item_hash(
|
||||
inode: u64,
|
||||
original_path: &[u8],
|
||||
ctime_secs: u64,
|
||||
mtime_secs: u64,
|
||||
crtime_secs: u64,
|
||||
) -> u64 {
|
||||
let seed = 1234;
|
||||
let mut hasher = XxHash64::with_seed(seed);
|
||||
hasher.write_u64(inode);
|
||||
hasher.write(original_path);
|
||||
hasher.write_u64(ctime_secs);
|
||||
hasher.write_u64(mtime_secs);
|
||||
hasher.write_u64(crtime_secs);
|
||||
hasher.finish()
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
pub mod attrs;
|
||||
pub mod hash;
|
||||
pub mod logging;
|
||||
pub mod music;
|
||||
|
||||
pub use attrs::FileAttrs;
|
||||
pub use hash::compute_item_hash;
|
||||
pub use music::metadata::MusicMetadata;
|
||||
@@ -0,0 +1,89 @@
|
||||
//! Process-wide logging initialization.
|
||||
//!
|
||||
//! Wires three `fmt` layers under a single global [`EnvFilter`] (driven by
|
||||
//! `RUST_LOG`, defaulting to `info`):
|
||||
//!
|
||||
//! - **stdout** — `INFO`/`DEBUG`/`TRACE` (the verbose side)
|
||||
//! - **stderr** — `WARN`/`ERROR` (the severe side)
|
||||
//! - **file** — everything passing the global filter, daily-rotated with
|
||||
//! retention, written without ANSI colors so log files stay clean.
|
||||
//!
|
||||
//! [`init`] returns the [`WorkerGuard`] that owns the non-blocking file
|
||||
//! buffer; the caller binds it for the whole process lifetime so the buffer
|
||||
//! flushes on exit. Dropping it early would silently drop pending log lines.
|
||||
//!
|
||||
//! The FUSE/RPC hot paths carry `trace!`/`debug!` points that fire per syscall
|
||||
//! — thousands per second under load. They are inert unless
|
||||
//! `RUST_LOG=trace`/`debug` is set; do not enable those levels in production
|
||||
//! casually.
|
||||
|
||||
use std::path::PathBuf;
|
||||
|
||||
use tracing::Level;
|
||||
use tracing_subscriber::{EnvFilter, filter::filter_fn, fmt, prelude::*};
|
||||
|
||||
/// Configuration handed to [`init`] by each binary.
|
||||
pub struct LogConfig {
|
||||
/// Directory the daily-rotated log files are written into.
|
||||
pub log_dir: PathBuf,
|
||||
/// Filename prefix; conventionally the binary name (`musicfs` or
|
||||
/// `musicfs-server`).
|
||||
pub file_prefix: String,
|
||||
/// Retention: keep at most this many rotated log files.
|
||||
pub max_files: usize,
|
||||
}
|
||||
|
||||
/// Initialize the global tracing subscriber and return the file-writer
|
||||
/// [`WorkerGuard`]. Bind it for the process lifetime so pending file writes
|
||||
/// flush on exit.
|
||||
///
|
||||
/// Panics if the log directory cannot be created or the rolling file appender
|
||||
/// cannot be initialized — both are fatal startup conditions worth failing
|
||||
/// fast on, before any real work begins.
|
||||
pub fn init(cfg: LogConfig) -> tracing_appender::non_blocking::WorkerGuard {
|
||||
std::fs::create_dir_all(&cfg.log_dir).unwrap_or_else(|e| {
|
||||
panic!(
|
||||
"logging: failed to create log dir {}: {e}",
|
||||
cfg.log_dir.display()
|
||||
)
|
||||
});
|
||||
|
||||
// One global gate for all three layers. RUST_LOG wins; default `info`.
|
||||
let env_filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info"));
|
||||
|
||||
let file_appender = tracing_appender::rolling::RollingFileAppender::builder()
|
||||
.rotation(tracing_appender::rolling::Rotation::DAILY)
|
||||
.filename_prefix(&cfg.file_prefix)
|
||||
.filename_suffix("log")
|
||||
.max_log_files(cfg.max_files)
|
||||
.build(&cfg.log_dir)
|
||||
.unwrap_or_else(|e| {
|
||||
panic!(
|
||||
"logging: failed to init rolling file appender in {}: {e}",
|
||||
cfg.log_dir.display()
|
||||
)
|
||||
});
|
||||
let (file_writer, guard) = tracing_appender::non_blocking(file_appender);
|
||||
|
||||
// tracing's Level ordering is ERROR < WARN < INFO < DEBUG < TRACE, so
|
||||
// `>= INFO` selects the verbose side (INFO/DEBUG/TRACE) routed to stdout,
|
||||
// and `<= WARN` selects the severe side (WARN/ERROR) routed to stderr.
|
||||
let stdout_layer = fmt::layer()
|
||||
.with_writer(std::io::stdout)
|
||||
.with_filter(filter_fn(|m| *m.level() >= Level::INFO));
|
||||
|
||||
let stderr_layer = fmt::layer()
|
||||
.with_writer(std::io::stderr)
|
||||
.with_filter(filter_fn(|m| *m.level() <= Level::WARN));
|
||||
|
||||
let file_layer = fmt::layer().with_ansi(false).with_writer(file_writer);
|
||||
|
||||
tracing_subscriber::registry()
|
||||
.with(env_filter)
|
||||
.with(stdout_layer)
|
||||
.with(stderr_layer)
|
||||
.with(file_layer)
|
||||
.init();
|
||||
|
||||
guard
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
use std::path::Path;
|
||||
|
||||
use crate::music::flac::FlacMusicMetadataEncoder;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::music::mp3::Mp3MusicMetadataEncoder;
|
||||
|
||||
/// Bakes the current (possibly overridden) tag fields of a [`MusicMetadata`]
|
||||
/// into its in-memory `header`. The original media file is never modified;
|
||||
/// the rebuilt header is served on the fly at read time, ahead of the
|
||||
/// externalized frames and the original audio.
|
||||
pub trait MusicMetadataEncoder {
|
||||
fn encode(&self, metadata: &mut MusicMetadata);
|
||||
}
|
||||
|
||||
/// Selects the right [`MusicMetadataEncoder`] for a given file.
|
||||
pub struct MusicMetadataEncoderFactory;
|
||||
|
||||
impl MusicMetadataEncoderFactory {
|
||||
pub fn for_path(path: &Path) -> Option<Box<dyn MusicMetadataEncoder>> {
|
||||
match path
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.map(str::to_ascii_lowercase)
|
||||
.as_deref()
|
||||
{
|
||||
Some("flac") => Some(Box::new(FlacMusicMetadataEncoder)),
|
||||
Some("mp3") => Some(Box::new(Mp3MusicMetadataEncoder)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,445 @@
|
||||
use std::{
|
||||
fs,
|
||||
io::{Cursor, Read, Seek, SeekFrom},
|
||||
path::Path,
|
||||
};
|
||||
|
||||
use symphonia::core::{
|
||||
formats::FormatOptions, io::MediaSourceStream, meta::MetadataOptions, probe::Hint,
|
||||
};
|
||||
|
||||
use crate::music::encoder::MusicMetadataEncoder;
|
||||
use crate::music::metadata::{MusicMetadata, extract_standard_tags};
|
||||
use crate::music::parser::MusicMetadataParser;
|
||||
use tracing::warn;
|
||||
|
||||
const BLOCK_PADDING: u8 = 1;
|
||||
const BLOCK_VORBIS_COMMENT: u8 = 4;
|
||||
const BLOCK_PICTURE: u8 = 6;
|
||||
const BLOCK_LAST_FLAG: u8 = 0x80;
|
||||
const BLOCK_TYPE_MASK: u8 = 0x7f;
|
||||
const PADDING_SIZE: usize = 8192;
|
||||
|
||||
/// FLAC parser. Owns all FLAC container parsing; returns `None` (never panics)
|
||||
/// on a file it can't read — a corrupt or mid-copy file is logged and served
|
||||
/// as plain passthrough rather than taking down the snapshot/watcher.
|
||||
pub struct FlacMusicMetadataParser;
|
||||
|
||||
impl MusicMetadataParser for FlacMusicMetadataParser {
|
||||
fn parse(&self, path: &Path) -> Option<MusicMetadata> {
|
||||
match parse_flac_metadata(path) {
|
||||
Some(mm) => Some(mm),
|
||||
None => {
|
||||
warn!(
|
||||
path = %path.display(),
|
||||
"failed to parse FLAC metadata; serving as passthrough"
|
||||
);
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// FLAC encoder. Rebuilds the FLAC metadata header from the current tag fields,
|
||||
/// re-injecting the (possibly overridden) Vorbis comment.
|
||||
pub struct FlacMusicMetadataEncoder;
|
||||
|
||||
impl MusicMetadataEncoder for FlacMusicMetadataEncoder {
|
||||
fn encode(&self, metadata: &mut MusicMetadata) {
|
||||
if metadata.header.is_empty() {
|
||||
return;
|
||||
}
|
||||
let blocks = extract_non_vorbis_blocks(&metadata.header);
|
||||
let (header, vorbis_comment_offset, vorbis_comment_length) =
|
||||
build_flac_header(blocks, metadata);
|
||||
metadata.header = header;
|
||||
metadata.vorbis_comment_offset = vorbis_comment_offset;
|
||||
metadata.vorbis_comment_length = vorbis_comment_length;
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_flac_metadata(path: &Path) -> Option<MusicMetadata> {
|
||||
let src = fs::File::open(path).ok()?;
|
||||
let mss = MediaSourceStream::new(Box::new(src), Default::default());
|
||||
let mut hint = Hint::new();
|
||||
hint.with_extension("flac");
|
||||
|
||||
let meta_opts: MetadataOptions = Default::default();
|
||||
let fmt_opts: FormatOptions = Default::default();
|
||||
|
||||
let mut format = symphonia::default::get_probe()
|
||||
.format(&hint, mss, &fmt_opts, &meta_opts)
|
||||
.ok()?;
|
||||
|
||||
let metadata = format.format.metadata();
|
||||
|
||||
let mut music_metadata = MusicMetadata::default();
|
||||
if let Some(revision) = metadata.current() {
|
||||
extract_standard_tags(revision, &mut music_metadata);
|
||||
}
|
||||
|
||||
if let Some(parsed) = parse_flac(path) {
|
||||
music_metadata.real_audio_start = parsed.audio_start;
|
||||
music_metadata.picture_block_headers = parsed.picture_block_headers;
|
||||
music_metadata.picture_data_ranges = parsed.picture_data_ranges;
|
||||
let (header, vorbis_comment_offset, vorbis_comment_length) =
|
||||
build_flac_header(parsed.other_blocks, &music_metadata);
|
||||
music_metadata.header = header;
|
||||
music_metadata.vorbis_comment_offset = vorbis_comment_offset;
|
||||
music_metadata.vorbis_comment_length = vorbis_comment_length;
|
||||
}
|
||||
|
||||
Some(music_metadata)
|
||||
}
|
||||
|
||||
impl MusicMetadata {
|
||||
/// Locate the Vorbis comment block within the rebuilt FLAC header so writes
|
||||
/// to it can be intercepted. No-op for non-FLAC (e.g. ID3) headers.
|
||||
pub fn find_vorbis_offsets(&mut self) {
|
||||
let mut cursor = Cursor::new(&self.header);
|
||||
let mut magic = [0u8; 4];
|
||||
if cursor.read_exact(&mut magic).is_err() {
|
||||
return;
|
||||
}
|
||||
if &magic != b"fLaC" {
|
||||
return;
|
||||
}
|
||||
loop {
|
||||
let mut hdr = [0u8; 4];
|
||||
if cursor.read_exact(&mut hdr).is_err() {
|
||||
break;
|
||||
}
|
||||
let (is_last, block_type, length) = read_block_header(&hdr);
|
||||
if block_type == BLOCK_VORBIS_COMMENT {
|
||||
self.vorbis_comment_offset = cursor.position();
|
||||
self.vorbis_comment_length = length;
|
||||
return;
|
||||
}
|
||||
if cursor.seek(SeekFrom::Current(length as i64)).is_err() {
|
||||
break;
|
||||
}
|
||||
if is_last {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Apply a freshly written Vorbis comment block and rebuild the header.
|
||||
pub fn update_from_vorbis_comment_data(&mut self, data: &[u8]) {
|
||||
parse_vorbis_comment_block(data, self);
|
||||
let other_blocks = extract_non_vorbis_blocks(&self.header);
|
||||
let (header, vorbis_comment_offset, vorbis_comment_length) =
|
||||
build_flac_header(other_blocks, self);
|
||||
self.header = header;
|
||||
self.vorbis_comment_offset = vorbis_comment_offset;
|
||||
self.vorbis_comment_length = vorbis_comment_length;
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a 4-byte FLAC metadata block header into (is_last, block_type, data_length).
|
||||
fn read_block_header(hdr: &[u8; 4]) -> (bool, u8, u64) {
|
||||
let is_last = (hdr[0] & BLOCK_LAST_FLAG) != 0;
|
||||
let block_type = hdr[0] & BLOCK_TYPE_MASK;
|
||||
let length = u32::from_be_bytes([0, hdr[1], hdr[2], hdr[3]]) as u64;
|
||||
(is_last, block_type, length)
|
||||
}
|
||||
|
||||
struct FlacParsed {
|
||||
other_blocks: Vec<(u8, Vec<u8>)>,
|
||||
picture_block_headers: Vec<Vec<u8>>,
|
||||
picture_data_ranges: Vec<(u64, u64)>,
|
||||
audio_start: u64,
|
||||
}
|
||||
|
||||
fn parse_flac(path: &Path) -> Option<FlacParsed> {
|
||||
let mut f = fs::File::open(path).ok()?;
|
||||
|
||||
let mut magic = [0u8; 4];
|
||||
f.read_exact(&mut magic).ok()?;
|
||||
if &magic != b"fLaC" {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut other_blocks: Vec<(u8, Vec<u8>)> = vec![];
|
||||
let mut picture_block_headers: Vec<Vec<u8>> = vec![];
|
||||
let mut picture_data_ranges: Vec<(u64, u64)> = vec![];
|
||||
let mut pos = 4u64;
|
||||
|
||||
loop {
|
||||
let mut hdr = [0u8; 4];
|
||||
f.read_exact(&mut hdr).ok()?;
|
||||
let (is_last, block_type, length) = read_block_header(&hdr);
|
||||
pos += 4;
|
||||
|
||||
if block_type == BLOCK_PICTURE {
|
||||
// PICTURE: keep block header (we'll fix is_last later), record data range
|
||||
picture_block_headers.push(hdr.to_vec());
|
||||
picture_data_ranges.push((pos, length));
|
||||
f.seek(SeekFrom::Current(length as i64)).ok()?;
|
||||
} else {
|
||||
let mut data = vec![0u8; length as usize];
|
||||
f.read_exact(&mut data).ok()?;
|
||||
// Skip VORBIS_COMMENT — we rebuild it
|
||||
if block_type != BLOCK_VORBIS_COMMENT {
|
||||
other_blocks.push((block_type, data));
|
||||
}
|
||||
}
|
||||
|
||||
pos += length;
|
||||
if is_last {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Fix is_last on the last picture block header: it must be 1 when audio follows
|
||||
if let Some(last_hdr) = picture_block_headers.last_mut() {
|
||||
last_hdr[0] = BLOCK_LAST_FLAG | (last_hdr[0] & BLOCK_TYPE_MASK);
|
||||
}
|
||||
|
||||
Some(FlacParsed {
|
||||
other_blocks,
|
||||
picture_block_headers,
|
||||
picture_data_ranges,
|
||||
audio_start: pos,
|
||||
})
|
||||
}
|
||||
|
||||
fn build_flac_header(blocks: Vec<(u8, Vec<u8>)>, metadata: &MusicMetadata) -> (Vec<u8>, u64, u64) {
|
||||
let vorbis = build_vorbis_comment(metadata);
|
||||
let vorbis_len = vorbis.len() as u64;
|
||||
|
||||
let mut patched: Vec<(u8, Vec<u8>)> = blocks;
|
||||
patched.push((BLOCK_VORBIS_COMMENT, vorbis));
|
||||
patched.push((BLOCK_PADDING, vec![0u8; PADDING_SIZE])); // allows metaflac in-place writes
|
||||
|
||||
let vorbis_idx = patched.len() - 2;
|
||||
let has_pictures = !metadata.picture_data_ranges.is_empty();
|
||||
let mut out = Vec::new();
|
||||
out.extend_from_slice(b"fLaC");
|
||||
|
||||
let last = patched.len() - 1;
|
||||
let mut vorbis_offset = 0u64;
|
||||
for (i, (block_type, data)) in patched.iter().enumerate() {
|
||||
let is_last_block = i == last && !has_pictures;
|
||||
let flag: u8 = if is_last_block { BLOCK_LAST_FLAG } else { 0x00 };
|
||||
let length = data.len() as u32;
|
||||
if i == vorbis_idx {
|
||||
vorbis_offset = out.len() as u64 + 4; // data starts after 4-byte block header
|
||||
}
|
||||
out.push(flag | block_type);
|
||||
out.push((length >> 16) as u8);
|
||||
out.push((length >> 8) as u8);
|
||||
out.push(length as u8);
|
||||
out.extend_from_slice(data);
|
||||
}
|
||||
|
||||
(out, vorbis_offset, vorbis_len)
|
||||
}
|
||||
|
||||
fn parse_vorbis_comment_block(data: &[u8], out: &mut MusicMetadata) {
|
||||
let mut cursor = Cursor::new(data);
|
||||
let mut len_bytes = [0u8; 4];
|
||||
|
||||
if cursor.read_exact(&mut len_bytes).is_err() {
|
||||
return;
|
||||
}
|
||||
let vendor_len = u32::from_le_bytes(len_bytes) as i64;
|
||||
if cursor.seek(SeekFrom::Current(vendor_len)).is_err() {
|
||||
return;
|
||||
}
|
||||
if cursor.read_exact(&mut len_bytes).is_err() {
|
||||
return;
|
||||
}
|
||||
let count = u32::from_le_bytes(len_bytes);
|
||||
|
||||
out.artist.clear();
|
||||
out.other_tags.clear();
|
||||
|
||||
for _ in 0..count {
|
||||
if cursor.read_exact(&mut len_bytes).is_err() {
|
||||
break;
|
||||
}
|
||||
let comment_len = u32::from_le_bytes(len_bytes) as usize;
|
||||
let mut comment_bytes = vec![0u8; comment_len];
|
||||
if cursor.read_exact(&mut comment_bytes).is_err() {
|
||||
break;
|
||||
}
|
||||
let comment = String::from_utf8_lossy(&comment_bytes).into_owned();
|
||||
if let Some((key, value)) = comment.split_once('=') {
|
||||
match key.to_ascii_uppercase().as_str() {
|
||||
"TITLE" => out.track_title = value.to_string(),
|
||||
"ALBUM" => out.album = value.to_string(),
|
||||
"TRACKNUMBER" => out.track_number = value.parse().unwrap_or(0),
|
||||
"ARTIST" => out.artist.push(value.to_string()),
|
||||
_ => out.other_tags.push(comment),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn extract_non_vorbis_blocks(header: &[u8]) -> Vec<(u8, Vec<u8>)> {
|
||||
let mut cursor = Cursor::new(header);
|
||||
let mut blocks = vec![];
|
||||
|
||||
let mut magic = [0u8; 4];
|
||||
if cursor.read_exact(&mut magic).is_err() {
|
||||
return blocks;
|
||||
}
|
||||
|
||||
loop {
|
||||
let mut hdr = [0u8; 4];
|
||||
if cursor.read_exact(&mut hdr).is_err() {
|
||||
break;
|
||||
}
|
||||
let (is_last, block_type, length) = read_block_header(&hdr);
|
||||
let mut data = vec![0u8; length as usize];
|
||||
if cursor.read_exact(&mut data).is_err() {
|
||||
break;
|
||||
}
|
||||
if block_type != BLOCK_VORBIS_COMMENT && block_type != BLOCK_PADDING {
|
||||
blocks.push((block_type, data));
|
||||
}
|
||||
if is_last {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
blocks
|
||||
}
|
||||
|
||||
fn build_vorbis_comment(metadata: &MusicMetadata) -> Vec<u8> {
|
||||
let vendor = b"musicfs";
|
||||
let mut out = Vec::new();
|
||||
|
||||
out.extend_from_slice(&(vendor.len() as u32).to_le_bytes());
|
||||
out.extend_from_slice(vendor);
|
||||
|
||||
let mut comments: Vec<String> = vec![
|
||||
format!("TITLE={}", metadata.track_title),
|
||||
format!("ALBUM={}", metadata.album),
|
||||
format!("TRACKNUMBER={}", metadata.track_number),
|
||||
];
|
||||
for artist in &metadata.artist {
|
||||
comments.push(format!("ARTIST={}", artist));
|
||||
}
|
||||
comments.extend(metadata.other_tags.iter().cloned());
|
||||
|
||||
out.extend_from_slice(&(comments.len() as u32).to_le_bytes());
|
||||
for comment in &comments {
|
||||
let bytes = comment.as_bytes();
|
||||
out.extend_from_slice(&(bytes.len() as u32).to_le_bytes());
|
||||
out.extend_from_slice(bytes);
|
||||
}
|
||||
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn read_block_header_normal() {
|
||||
let hdr = [0x04, 0x00, 0x01, 0x00];
|
||||
let (is_last, block_type, length) = read_block_header(&hdr);
|
||||
assert!(!is_last);
|
||||
assert_eq!(block_type, 4);
|
||||
assert_eq!(length, 256);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_block_header_last() {
|
||||
let hdr = [0x84, 0x00, 0x00, 0x10];
|
||||
let (is_last, block_type, length) = read_block_header(&hdr);
|
||||
assert!(is_last);
|
||||
assert_eq!(block_type, 4);
|
||||
assert_eq!(length, 16);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_vorbis_comment_roundtrip() {
|
||||
let mut metadata = MusicMetadata::default();
|
||||
metadata.artist = vec!["Artist1".to_string(), "Artist2".to_string()];
|
||||
metadata.album = "Album".to_string();
|
||||
metadata.track_number = 3;
|
||||
metadata.track_title = "Title".to_string();
|
||||
metadata.other_tags = vec!["GENRE=Rock".to_string()];
|
||||
|
||||
let vorbis_bytes = build_vorbis_comment(&metadata);
|
||||
let mut parsed = MusicMetadata::default();
|
||||
parse_vorbis_comment_block(&vorbis_bytes, &mut parsed);
|
||||
|
||||
assert_eq!(parsed.artist, vec!["Artist1", "Artist2"]);
|
||||
assert_eq!(parsed.album, "Album");
|
||||
assert_eq!(parsed.track_number, 3);
|
||||
assert_eq!(parsed.track_title, "Title");
|
||||
assert_eq!(parsed.other_tags, vec!["GENRE=Rock"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_vorbis_comment_multi_artist() {
|
||||
let mut vorbis_bytes = Vec::new();
|
||||
let vendor = b"test";
|
||||
vorbis_bytes.extend_from_slice(&(vendor.len() as u32).to_le_bytes());
|
||||
vorbis_bytes.extend_from_slice(vendor);
|
||||
|
||||
let comments = vec!["ARTIST=Artist1", "ARTIST=Artist2"];
|
||||
vorbis_bytes.extend_from_slice(&(comments.len() as u32).to_le_bytes());
|
||||
for comment in &comments {
|
||||
let bytes = comment.as_bytes();
|
||||
vorbis_bytes.extend_from_slice(&(bytes.len() as u32).to_le_bytes());
|
||||
vorbis_bytes.extend_from_slice(bytes);
|
||||
}
|
||||
|
||||
let mut metadata = MusicMetadata::default();
|
||||
parse_vorbis_comment_block(&vorbis_bytes, &mut metadata);
|
||||
|
||||
assert_eq!(metadata.artist.len(), 2);
|
||||
assert_eq!(metadata.artist[0], "Artist1");
|
||||
assert_eq!(metadata.artist[1], "Artist2");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_non_vorbis_blocks_strips_vc_and_padding() {
|
||||
let mut header = Vec::new();
|
||||
header.extend_from_slice(b"fLaC");
|
||||
|
||||
let streaminfo_data = vec![0u8; 34];
|
||||
header.push(0x00);
|
||||
header.extend_from_slice(&[0x00, 0x00, 0x22]);
|
||||
header.extend_from_slice(&streaminfo_data);
|
||||
|
||||
let vorbis_data = vec![0u8; 50];
|
||||
header.push(0x04);
|
||||
header.extend_from_slice(&[0x00, 0x00, 0x32]);
|
||||
header.extend_from_slice(&vorbis_data);
|
||||
|
||||
let padding_data = vec![0u8; 100];
|
||||
header.push(0x81);
|
||||
header.extend_from_slice(&[0x00, 0x00, 0x64]);
|
||||
header.extend_from_slice(&padding_data);
|
||||
|
||||
let blocks = extract_non_vorbis_blocks(&header);
|
||||
assert_eq!(blocks.len(), 1);
|
||||
assert_eq!(blocks[0].0, 0);
|
||||
assert_eq!(blocks[0].1.len(), 34);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn find_vorbis_offsets_correct() {
|
||||
let mut metadata = MusicMetadata::default();
|
||||
metadata.artist = vec!["TestArtist".to_string()];
|
||||
metadata.album = "TestAlbum".to_string();
|
||||
metadata.track_number = 1;
|
||||
metadata.track_title = "TestTitle".to_string();
|
||||
|
||||
let other_blocks = vec![(0, vec![0u8; 34])];
|
||||
let (header, expected_offset, expected_length) = build_flac_header(other_blocks, &metadata);
|
||||
metadata.header = header;
|
||||
|
||||
metadata.find_vorbis_offsets();
|
||||
|
||||
assert_eq!(metadata.vorbis_comment_offset, expected_offset);
|
||||
assert_eq!(metadata.vorbis_comment_length, expected_length);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
use symphonia::core::meta::{MetadataRevision, StandardTagKey};
|
||||
|
||||
#[derive(Debug, Default, Clone, PartialEq, Eq)]
|
||||
pub struct MusicMetadata {
|
||||
pub artist: Vec<String>,
|
||||
pub album_artist: Option<String>,
|
||||
pub album: String,
|
||||
pub track_number: i32,
|
||||
pub track_title: String,
|
||||
pub other_tags: Vec<String>,
|
||||
pub header: Vec<u8>,
|
||||
pub picture_block_headers: Vec<Vec<u8>>,
|
||||
pub picture_data_ranges: Vec<(u64, u64)>,
|
||||
pub real_audio_start: u64,
|
||||
pub vorbis_comment_offset: u64,
|
||||
pub vorbis_comment_length: u64,
|
||||
}
|
||||
|
||||
impl MusicMetadata {
|
||||
pub fn virtual_size(&self, real_file_size: u64) -> u64 {
|
||||
let pictures_size: u64 = self
|
||||
.picture_block_headers
|
||||
.iter()
|
||||
.zip(self.picture_data_ranges.iter())
|
||||
.map(|(prefix, (_, len))| prefix.len() as u64 + len)
|
||||
.sum();
|
||||
self.header.len() as u64 + pictures_size + (real_file_size - self.real_audio_start)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn extract_standard_tags(revision: &MetadataRevision, out: &mut MusicMetadata) {
|
||||
for tag in revision.tags() {
|
||||
let value = tag.value.to_string();
|
||||
match tag.std_key {
|
||||
Some(StandardTagKey::Artist) => out.artist.push(value),
|
||||
Some(StandardTagKey::AlbumArtist) => out.album_artist = Some(value),
|
||||
Some(StandardTagKey::Album) => out.album = value,
|
||||
Some(StandardTagKey::TrackNumber) => {
|
||||
out.track_number = value.parse::<i32>().unwrap_or(0)
|
||||
}
|
||||
Some(StandardTagKey::TrackTitle) => out.track_title = value,
|
||||
_ => out.other_tags.push(format!("{}={}", tag.key, value)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn virtual_size_calculation() {
|
||||
let mut metadata = MusicMetadata::default();
|
||||
metadata.header = vec![0u8; 100];
|
||||
metadata.picture_block_headers = vec![vec![0u8; 4]];
|
||||
metadata.picture_data_ranges = vec![(0, 50)];
|
||||
metadata.real_audio_start = 200;
|
||||
let file_size = 1000u64;
|
||||
let virtual_size = metadata.virtual_size(file_size);
|
||||
assert_eq!(virtual_size, 954);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
pub mod encoder;
|
||||
pub mod flac;
|
||||
pub mod metadata;
|
||||
pub mod mp3;
|
||||
pub mod parse;
|
||||
pub mod parser;
|
||||
@@ -0,0 +1,507 @@
|
||||
use std::{
|
||||
fs,
|
||||
io::{Read, Seek, SeekFrom},
|
||||
path::Path,
|
||||
};
|
||||
|
||||
use symphonia::core::{
|
||||
formats::FormatOptions, io::MediaSourceStream, meta::MetadataOptions, probe::Hint,
|
||||
};
|
||||
|
||||
use crate::music::encoder::MusicMetadataEncoder;
|
||||
use crate::music::metadata::{MusicMetadata, extract_standard_tags};
|
||||
use crate::music::parser::MusicMetadataParser;
|
||||
use tracing::warn;
|
||||
|
||||
/// Frames we replace from our own tag fields. Every other frame in the source
|
||||
/// ID3 tag is preserved verbatim via externalization.
|
||||
const OVERRIDE_FRAME_IDS: [[u8; 4]; 4] = [*b"TIT2", *b"TALB", *b"TPE1", *b"TRCK"];
|
||||
|
||||
/// MP3 parser. Reads ID3 tags (via symphonia) and locates where the audio
|
||||
/// frames begin (end of the ID3v2 tag).
|
||||
pub struct Mp3MusicMetadataParser;
|
||||
|
||||
impl MusicMetadataParser for Mp3MusicMetadataParser {
|
||||
fn parse(&self, path: &Path) -> Option<MusicMetadata> {
|
||||
match parse_mp3_metadata(path) {
|
||||
Some(mm) => Some(mm),
|
||||
None => {
|
||||
warn!(
|
||||
path = %path.display(),
|
||||
"failed to parse MP3 metadata; serving as passthrough"
|
||||
);
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Decode a 28-bit syncsafe integer (7 bits per byte) as used by ID3v2 sizes.
|
||||
fn syncsafe(b: [u8; 4]) -> u64 {
|
||||
((b[0] as u64) << 21) | ((b[1] as u64) << 14) | ((b[2] as u64) << 7) | (b[3] as u64)
|
||||
}
|
||||
|
||||
/// Byte offset where the MP3 audio frames begin: the end of the ID3v2 tag, or
|
||||
/// 0 when there is no tag. Reads only the 10-byte ID3 header.
|
||||
fn id3_audio_start(path: &Path) -> u64 {
|
||||
let mut f = match fs::File::open(path) {
|
||||
Ok(f) => f,
|
||||
Err(_) => return 0,
|
||||
};
|
||||
let mut hdr = [0u8; 10];
|
||||
if f.read_exact(&mut hdr).is_err() {
|
||||
return 0;
|
||||
}
|
||||
if &hdr[0..3] != b"ID3" {
|
||||
return 0;
|
||||
}
|
||||
let size = syncsafe([hdr[6], hdr[7], hdr[8], hdr[9]]);
|
||||
// ID3v2.4 footer flag adds another 10 bytes after the tag body.
|
||||
let footer = if hdr[5] & 0x10 != 0 { 10 } else { 0 };
|
||||
10 + size + footer
|
||||
}
|
||||
|
||||
fn parse_mp3_metadata(path: &Path) -> Option<MusicMetadata> {
|
||||
let src = fs::File::open(path).ok()?;
|
||||
let mss = MediaSourceStream::new(Box::new(src), Default::default());
|
||||
let mut hint = Hint::new();
|
||||
hint.with_extension("mp3");
|
||||
|
||||
let meta_opts: MetadataOptions = Default::default();
|
||||
let fmt_opts: FormatOptions = Default::default();
|
||||
|
||||
let mut probed = symphonia::default::get_probe()
|
||||
.format(&hint, mss, &fmt_opts, &meta_opts)
|
||||
.ok()?;
|
||||
|
||||
let mut music_metadata = MusicMetadata::default();
|
||||
|
||||
// ID3v2 tags at the start of the file surface in the probe-level metadata;
|
||||
// fall back to the in-stream metadata otherwise.
|
||||
let mut got_tags = false;
|
||||
if let Some(metadata) = probed.metadata.get() {
|
||||
if let Some(revision) = metadata.current() {
|
||||
extract_standard_tags(revision, &mut music_metadata);
|
||||
got_tags = true;
|
||||
}
|
||||
}
|
||||
if !got_tags {
|
||||
let metadata = probed.format.metadata();
|
||||
if let Some(revision) = metadata.current() {
|
||||
extract_standard_tags(revision, &mut music_metadata);
|
||||
}
|
||||
}
|
||||
|
||||
music_metadata.real_audio_start = id3_audio_start(path);
|
||||
|
||||
// Record the original frames we will preserve (cover art, lyrics, …) so the
|
||||
// encoder can stitch them back in from the original file at read time.
|
||||
let (prefixes, ranges) = parse_id3_preserved_frames(path);
|
||||
music_metadata.picture_block_headers = prefixes;
|
||||
music_metadata.picture_data_ranges = ranges;
|
||||
|
||||
// Untagged files would otherwise build a degenerate empty artist/album path.
|
||||
if music_metadata.artist.is_empty() {
|
||||
music_metadata.artist = vec!["Unknown Artist".to_string()];
|
||||
}
|
||||
if music_metadata.album.is_empty() {
|
||||
music_metadata.album = "Unknown Album".to_string();
|
||||
}
|
||||
|
||||
Some(music_metadata)
|
||||
}
|
||||
|
||||
impl MusicMetadata {
|
||||
/// Apply a freshly written ID3v2 tag and rebuild the in-memory header.
|
||||
pub fn update_from_id3_data(&mut self, data: &[u8]) {
|
||||
parse_id3_tag_frames(data, self);
|
||||
self.header = build_id3v2_header(self);
|
||||
}
|
||||
|
||||
/// Apply a freshly written ID3v1 tag (128-byte "TAG" block) and rebuild
|
||||
/// the in-memory ID3v2 header. ID3v1 is Latin-1, max 30 chars per field;
|
||||
/// non-ASCII bytes are replaced with '?' since Latin-1 ≠ UTF-8.
|
||||
pub fn update_from_id3v1_data(&mut self, data: &[u8]) {
|
||||
if data.len() < 128 || &data[0..3] != b"TAG" {
|
||||
return;
|
||||
}
|
||||
fn latin1_str(bytes: &[u8]) -> Option<String> {
|
||||
let end = bytes.iter().position(|&b| b == 0).unwrap_or(bytes.len());
|
||||
if end == 0 {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
bytes[..end]
|
||||
.iter()
|
||||
.map(|&b| if b < 0x80 { char::from(b) } else { '?' })
|
||||
.collect(),
|
||||
)
|
||||
}
|
||||
if let Some(t) = latin1_str(&data[3..33]) {
|
||||
self.track_title = t;
|
||||
}
|
||||
if let Some(a) = latin1_str(&data[33..63]) {
|
||||
self.artist = vec![a];
|
||||
}
|
||||
if let Some(a) = latin1_str(&data[63..93]) {
|
||||
self.album = a;
|
||||
}
|
||||
// ID3v1.1: byte 125 == 0 means byte 126 is the track number
|
||||
if data[125] == 0 && data[126] != 0 {
|
||||
self.track_number = data[126] as i32;
|
||||
}
|
||||
self.header = build_id3v2_header(self);
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_id3_text(data: &[u8]) -> String {
|
||||
if data.is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
match data[0] {
|
||||
0x03 => String::from_utf8_lossy(&data[1..])
|
||||
.trim_end_matches('\0')
|
||||
.to_string(),
|
||||
0x00 => data[1..]
|
||||
.iter()
|
||||
.take_while(|&&b| b != 0)
|
||||
.map(|&b| char::from(b))
|
||||
.collect(),
|
||||
0x01 | 0x02 => {
|
||||
let raw = &data[1..];
|
||||
let (src, le) = if raw.len() >= 2 && raw[0] == 0xFF && raw[1] == 0xFE {
|
||||
(&raw[2..], true)
|
||||
} else if raw.len() >= 2 && raw[0] == 0xFE && raw[1] == 0xFF {
|
||||
(&raw[2..], false)
|
||||
} else {
|
||||
(raw, true)
|
||||
};
|
||||
let words: Vec<u16> = src
|
||||
.chunks_exact(2)
|
||||
.map(|c| {
|
||||
if le {
|
||||
u16::from_le_bytes([c[0], c[1]])
|
||||
} else {
|
||||
u16::from_be_bytes([c[0], c[1]])
|
||||
}
|
||||
})
|
||||
.take_while(|&w| w != 0)
|
||||
.collect();
|
||||
String::from_utf16_lossy(&words)
|
||||
}
|
||||
_ => String::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_id3_tag_frames(data: &[u8], out: &mut MusicMetadata) {
|
||||
if data.len() < 10 || &data[0..3] != b"ID3" {
|
||||
return;
|
||||
}
|
||||
let version = data[3];
|
||||
if data[5] & 0x80 != 0 {
|
||||
return; // unsynchronised — can't copy frames verbatim
|
||||
}
|
||||
let tag_end = (10 + syncsafe([data[6], data[7], data[8], data[9]]) as usize).min(data.len());
|
||||
|
||||
out.artist.clear();
|
||||
let mut offset = 10usize;
|
||||
while offset + 10 <= tag_end {
|
||||
if data[offset..offset + 4].iter().all(|&b| b == 0) {
|
||||
break; // padding
|
||||
}
|
||||
let size = if version >= 4 {
|
||||
syncsafe([
|
||||
data[offset + 4],
|
||||
data[offset + 5],
|
||||
data[offset + 6],
|
||||
data[offset + 7],
|
||||
]) as usize
|
||||
} else {
|
||||
u32::from_be_bytes([
|
||||
data[offset + 4],
|
||||
data[offset + 5],
|
||||
data[offset + 6],
|
||||
data[offset + 7],
|
||||
]) as usize
|
||||
};
|
||||
if size == 0 || offset + 10 + size > tag_end {
|
||||
break;
|
||||
}
|
||||
let body = &data[offset + 10..offset + 10 + size];
|
||||
match &data[offset..offset + 4] {
|
||||
b"TIT2" => out.track_title = decode_id3_text(body),
|
||||
b"TALB" => out.album = decode_id3_text(body),
|
||||
b"TPE1" => {
|
||||
let text = decode_id3_text(body);
|
||||
out.artist = text
|
||||
.split('\0')
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(str::to_string)
|
||||
.collect();
|
||||
}
|
||||
b"TRCK" => {
|
||||
let text = decode_id3_text(body);
|
||||
out.track_number = text
|
||||
.split('/')
|
||||
.next()
|
||||
.and_then(|s| s.parse().ok())
|
||||
.unwrap_or(0);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
offset += 10 + size;
|
||||
}
|
||||
}
|
||||
|
||||
/// MP3 encoder. Builds a fresh ID3v2.3 tag from the current (possibly
|
||||
/// overridden) tag fields. Preserved frames are not copied into the header;
|
||||
/// their lengths are counted so the tag's size field spans them.
|
||||
pub struct Mp3MusicMetadataEncoder;
|
||||
|
||||
impl MusicMetadataEncoder for Mp3MusicMetadataEncoder {
|
||||
fn encode(&self, metadata: &mut MusicMetadata) {
|
||||
metadata.header = build_id3v2_header(metadata);
|
||||
}
|
||||
}
|
||||
|
||||
/// Encode a 28-bit syncsafe integer (7 bits per byte) for ID3v2 size fields.
|
||||
fn syncsafe_encode(n: u64) -> [u8; 4] {
|
||||
[
|
||||
((n >> 21) & 0x7f) as u8,
|
||||
((n >> 14) & 0x7f) as u8,
|
||||
((n >> 7) & 0x7f) as u8,
|
||||
(n & 0x7f) as u8,
|
||||
]
|
||||
}
|
||||
|
||||
/// Build a single UTF-16 ID3v2.3 text frame.
|
||||
fn build_id3_text_frame(id: &[u8; 4], text: &str) -> Vec<u8> {
|
||||
let mut body = Vec::with_capacity(3 + text.len() * 2);
|
||||
body.push(0x01); // text encoding: UTF-16 with BOM
|
||||
body.extend_from_slice(&[0xFF, 0xFE]); // little-endian BOM
|
||||
for unit in text.encode_utf16() {
|
||||
body.extend_from_slice(&unit.to_le_bytes());
|
||||
}
|
||||
|
||||
let mut frame = Vec::with_capacity(10 + body.len());
|
||||
frame.extend_from_slice(id);
|
||||
frame.extend_from_slice(&(body.len() as u32).to_be_bytes());
|
||||
frame.extend_from_slice(&[0, 0]); // frame flags
|
||||
frame.extend_from_slice(&body);
|
||||
frame
|
||||
}
|
||||
|
||||
/// Build the in-memory portion of the virtual ID3v2.3 tag: the 10-byte tag
|
||||
/// header plus our four override frames. Preserved frames and audio are
|
||||
/// stitched in by the read path; the tag size field accounts for them.
|
||||
fn build_id3v2_header(m: &MusicMetadata) -> Vec<u8> {
|
||||
let mut frames = Vec::new();
|
||||
frames.extend(build_id3_text_frame(b"TIT2", &m.track_title));
|
||||
frames.extend(build_id3_text_frame(b"TALB", &m.album));
|
||||
frames.extend(build_id3_text_frame(b"TPE1", &m.artist.join("\0")));
|
||||
frames.extend(build_id3_text_frame(b"TRCK", &m.track_number.to_string()));
|
||||
|
||||
let preserved_len: u64 = m
|
||||
.picture_block_headers
|
||||
.iter()
|
||||
.zip(m.picture_data_ranges.iter())
|
||||
.map(|(prefix, (_, len))| prefix.len() as u64 + len)
|
||||
.sum();
|
||||
|
||||
let body_len = frames.len() as u64 + preserved_len;
|
||||
|
||||
let mut header = Vec::with_capacity(10 + frames.len());
|
||||
header.extend_from_slice(b"ID3");
|
||||
header.extend_from_slice(&[0x03, 0x00, 0x00]); // v2.3.0, no flags
|
||||
header.extend_from_slice(&syncsafe_encode(body_len));
|
||||
header.extend_from_slice(&frames);
|
||||
header
|
||||
}
|
||||
|
||||
/// Walk the source ID3v2 tag and record every frame we do NOT override
|
||||
/// (cover art, lyrics, other text frames) as a whole-frame range in the
|
||||
/// original file. Each gets an empty in-memory prefix — the frame is copied
|
||||
/// verbatim from disk at read time. Returns empty lists when there is no tag
|
||||
/// or the tag is unsynchronised (which can't be externalized verbatim).
|
||||
fn parse_id3_preserved_frames(path: &Path) -> (Vec<Vec<u8>>, Vec<(u64, u64)>) {
|
||||
let mut prefixes: Vec<Vec<u8>> = Vec::new();
|
||||
let mut ranges: Vec<(u64, u64)> = Vec::new();
|
||||
|
||||
let mut f = match fs::File::open(path) {
|
||||
Ok(f) => f,
|
||||
Err(_) => return (prefixes, ranges),
|
||||
};
|
||||
let mut hdr = [0u8; 10];
|
||||
if f.read_exact(&mut hdr).is_err() || &hdr[0..3] != b"ID3" {
|
||||
return (prefixes, ranges);
|
||||
}
|
||||
let version = hdr[3];
|
||||
// Unsynchronised tags store bytes we can't copy verbatim; fall back to
|
||||
// passthrough by preserving nothing (the whole original tag is dropped,
|
||||
// but that only loses art on a rare encoding — acceptable for v1).
|
||||
if hdr[5] & 0x80 != 0 {
|
||||
return (prefixes, ranges);
|
||||
}
|
||||
|
||||
let tag_end = id3_audio_start(path);
|
||||
let mut offset: u64 = 10;
|
||||
while offset + 10 <= tag_end {
|
||||
if f.seek(SeekFrom::Start(offset)).is_err() {
|
||||
break;
|
||||
}
|
||||
let mut fh = [0u8; 10];
|
||||
if f.read_exact(&mut fh).is_err() {
|
||||
break;
|
||||
}
|
||||
// A zero frame id marks the start of padding.
|
||||
if fh[0..4].iter().all(|&b| b == 0) {
|
||||
break;
|
||||
}
|
||||
let size = if version >= 4 {
|
||||
syncsafe([fh[4], fh[5], fh[6], fh[7]])
|
||||
} else {
|
||||
u32::from_be_bytes([fh[4], fh[5], fh[6], fh[7]]) as u64
|
||||
};
|
||||
let frame_total = 10 + size;
|
||||
if size == 0 || offset + frame_total > tag_end {
|
||||
break;
|
||||
}
|
||||
let id = &fh[0..4];
|
||||
let is_override = OVERRIDE_FRAME_IDS.iter().any(|o| &o[..] == id);
|
||||
if !is_override {
|
||||
prefixes.push(Vec::new());
|
||||
ranges.push((offset, frame_total));
|
||||
}
|
||||
offset += frame_total;
|
||||
}
|
||||
|
||||
(prefixes, ranges)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::Write;
|
||||
|
||||
#[test]
|
||||
fn syncsafe_decodes_seven_bit_groups() {
|
||||
assert_eq!(syncsafe([0, 0, 0, 0x23]), 35);
|
||||
assert_eq!(syncsafe([0, 0, 1, 0]), 128);
|
||||
}
|
||||
|
||||
fn write_temp(bytes: &[u8]) -> tempfile::NamedTempFile {
|
||||
let mut f = tempfile::NamedTempFile::new().unwrap();
|
||||
f.write_all(bytes).unwrap();
|
||||
f.flush().unwrap();
|
||||
f
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn id3_audio_start_after_tag() {
|
||||
let mut buf = Vec::new();
|
||||
buf.extend_from_slice(b"ID3");
|
||||
buf.extend_from_slice(&[0x04, 0x00, 0x00]); // version + flags
|
||||
buf.extend_from_slice(&[0x00, 0x00, 0x00, 0x23]); // syncsafe size = 35
|
||||
buf.extend_from_slice(&[0u8; 35]); // tag body
|
||||
let f = write_temp(&buf);
|
||||
assert_eq!(id3_audio_start(f.path()), 45);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn id3_audio_start_with_footer() {
|
||||
let mut buf = Vec::new();
|
||||
buf.extend_from_slice(b"ID3");
|
||||
buf.extend_from_slice(&[0x04, 0x00, 0x10]); // footer flag set
|
||||
buf.extend_from_slice(&[0x00, 0x00, 0x00, 0x23]); // size = 35
|
||||
buf.extend_from_slice(&[0u8; 45]);
|
||||
let f = write_temp(&buf);
|
||||
assert_eq!(id3_audio_start(f.path()), 55);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn id3_audio_start_no_tag() {
|
||||
// Raw MP3 frame sync, no ID3 tag.
|
||||
let f = write_temp(&[0xFF, 0xFB, 0x40, 0xC0, 0x00, 0x00]);
|
||||
assert_eq!(id3_audio_start(f.path()), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syncsafe_encode_inverts_syncsafe() {
|
||||
for n in [0u64, 35, 128, 30696, 0x0FFF_FFFF] {
|
||||
assert_eq!(syncsafe(syncsafe_encode(n)), n);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_id3v2_header_structure() {
|
||||
let mm = MusicMetadata {
|
||||
track_title: "Title".to_string(),
|
||||
album: "Album".to_string(),
|
||||
artist: vec!["A".to_string(), "B".to_string()],
|
||||
track_number: 7,
|
||||
// one preserved frame of total length 20
|
||||
picture_block_headers: vec![Vec::new()],
|
||||
picture_data_ranges: vec![(100, 20)],
|
||||
..MusicMetadata::default()
|
||||
};
|
||||
let header = build_id3v2_header(&mm);
|
||||
|
||||
assert_eq!(&header[0..3], b"ID3");
|
||||
assert_eq!(&header[3..6], &[0x03, 0x00, 0x00]);
|
||||
|
||||
let frames_len = header.len() as u64 - 10;
|
||||
let declared = syncsafe([header[6], header[7], header[8], header[9]]);
|
||||
// size field spans our frames + the preserved frame's 20 bytes
|
||||
assert_eq!(declared, frames_len + 20);
|
||||
|
||||
assert!(header.windows(4).any(|w| w == b"TIT2"));
|
||||
assert!(header.windows(4).any(|w| w == b"TPE1"));
|
||||
}
|
||||
|
||||
fn id3_frame(id: &[u8; 4], data: &[u8], syncsafe_size: bool) -> Vec<u8> {
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(id);
|
||||
if syncsafe_size {
|
||||
v.extend_from_slice(&syncsafe_encode(data.len() as u64));
|
||||
} else {
|
||||
v.extend_from_slice(&(data.len() as u32).to_be_bytes());
|
||||
}
|
||||
v.extend_from_slice(&[0, 0]); // flags
|
||||
v.extend_from_slice(data);
|
||||
v
|
||||
}
|
||||
|
||||
fn build_tag(version: u8, frames: &[u8]) -> Vec<u8> {
|
||||
let mut buf = Vec::new();
|
||||
buf.extend_from_slice(b"ID3");
|
||||
buf.extend_from_slice(&[version, 0x00, 0x00]);
|
||||
buf.extend_from_slice(&syncsafe_encode(frames.len() as u64));
|
||||
buf.extend_from_slice(frames);
|
||||
buf
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preserved_frames_v23_keeps_apic_drops_overrides() {
|
||||
let mut frames = Vec::new();
|
||||
frames.extend(id3_frame(b"APIC", &[1, 2, 3, 4, 5], false));
|
||||
frames.extend(id3_frame(b"TIT2", &[0x03, b'h', b'i'], false));
|
||||
let f = write_temp(&build_tag(0x03, &frames));
|
||||
|
||||
let (prefixes, ranges) = parse_id3_preserved_frames(f.path());
|
||||
// APIC starts right after the 10-byte tag header; total = 10 + 5
|
||||
assert_eq!(ranges, vec![(10, 15)]);
|
||||
assert_eq!(prefixes, vec![Vec::<u8>::new()]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn preserved_frames_v24_syncsafe_sizes() {
|
||||
let mut frames = Vec::new();
|
||||
frames.extend(id3_frame(b"USLT", &[9, 9, 9], true));
|
||||
frames.extend(id3_frame(b"TALB", &[0x03, b'x'], true));
|
||||
let f = write_temp(&build_tag(0x04, &frames));
|
||||
|
||||
let (prefixes, ranges) = parse_id3_preserved_frames(f.path());
|
||||
assert_eq!(ranges, vec![(10, 13)]); // USLT: 10 + 3
|
||||
assert_eq!(prefixes, vec![Vec::<u8>::new()]);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
use std::path::Path;
|
||||
|
||||
use tracing::{debug, trace};
|
||||
|
||||
use crate::music::encoder::MusicMetadataEncoderFactory;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::music::parser::MusicMetadataParserFactory;
|
||||
|
||||
/// Parse a single media file into a fully-encoded [`MusicMetadata`].
|
||||
///
|
||||
/// `None` is returned for files musicfs does not treat as music (unknown
|
||||
/// extension or unparseable container). When metadata is produced, the
|
||||
/// matching [`MusicMetadataEncoder`] has already baked the tag fields into
|
||||
/// the in-memory `header`, so callers can serve virtualized bytes directly.
|
||||
///
|
||||
/// A music-extension file that fails to parse is logged at `warn!` by the
|
||||
/// selected parser (see [`crate::music::flac`] / [`crate::music::mp3`]); this
|
||||
/// function simply propagates the resulting `None` without re-logging.
|
||||
///
|
||||
/// This helper is the single source of truth for "parse + encode" — used by
|
||||
/// the local FUSE origin's snapshot builder, the network origin's manifest
|
||||
/// builder, and the server's manifest endpoint.
|
||||
pub fn parse_music_metadata_for_path(path: &Path) -> Option<MusicMetadata> {
|
||||
let parser = match MusicMetadataParserFactory::for_path(path) {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
trace!(path = %path.display(), "non-music extension; skipping");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
debug!(path = %path.display(), "music parser selected");
|
||||
let mut metadata = parser.parse(path)?;
|
||||
if let Some(encoder) = MusicMetadataEncoderFactory::for_path(path) {
|
||||
encoder.encode(&mut metadata);
|
||||
}
|
||||
return Some(metadata);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::path::Path;
|
||||
|
||||
#[test]
|
||||
fn none_for_non_music_extension() {
|
||||
let path = Path::new("notes.txt");
|
||||
assert!(parse_music_metadata_for_path(path).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn none_for_missing_file_with_music_extension() {
|
||||
let path = Path::new("/nonexistent/track.flac");
|
||||
assert!(parse_music_metadata_for_path(path).is_none());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
use std::path::Path;
|
||||
|
||||
use crate::music::flac::FlacMusicMetadataParser;
|
||||
use crate::music::metadata::MusicMetadata;
|
||||
use crate::music::mp3::Mp3MusicMetadataParser;
|
||||
|
||||
/// A parser that turns a single media file into our shared [`MusicMetadata`].
|
||||
/// Each supported container format provides one implementation.
|
||||
pub trait MusicMetadataParser {
|
||||
fn parse(&self, path: &Path) -> Option<MusicMetadata>;
|
||||
}
|
||||
|
||||
/// Selects the right [`MusicMetadataParser`] for a given file.
|
||||
pub struct MusicMetadataParserFactory;
|
||||
|
||||
impl MusicMetadataParserFactory {
|
||||
/// Returns a parser based on the file extension, or `None` for files we
|
||||
/// don't treat as music.
|
||||
pub fn for_path(path: &Path) -> Option<Box<dyn MusicMetadataParser>> {
|
||||
match path
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.map(str::to_ascii_lowercase)
|
||||
.as_deref()
|
||||
{
|
||||
Some("flac") => Some(Box::new(FlacMusicMetadataParser)),
|
||||
Some("mp3") => Some(Box::new(Mp3MusicMetadataParser)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
[package]
|
||||
name = "musicfs-proto"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
prost.workspace = true
|
||||
tonic.workspace = true
|
||||
tonic-prost.workspace = true
|
||||
@@ -0,0 +1,835 @@
|
||||
// @generated
|
||||
// This file is @generated by prost-build.
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct ReconcileRequest {
|
||||
#[prost(message, repeated, tag = "1")]
|
||||
pub entries: ::prost::alloc::vec::Vec<InodeHash>,
|
||||
}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct ReconcileResponse {
|
||||
#[prost(message, repeated, tag = "1")]
|
||||
pub changed: ::prost::alloc::vec::Vec<InodeHash>,
|
||||
#[prost(uint64, repeated, tag = "2")]
|
||||
pub deleted: ::prost::alloc::vec::Vec<u64>,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct InodeHash {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub inode: u64,
|
||||
#[prost(uint64, tag = "2")]
|
||||
pub hash: u64,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct GetMetadataRequest {
|
||||
#[prost(uint64, repeated, tag = "1")]
|
||||
pub inodes: ::prost::alloc::vec::Vec<u64>,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct GetManifestRequest {}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct GetFileRequest {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub id: u64,
|
||||
/// When both are zero the server streams the entire file.
|
||||
#[prost(uint64, tag = "2")]
|
||||
pub start: u64,
|
||||
#[prost(uint64, tag = "3")]
|
||||
pub length: u64,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct SubscribeEventsRequest {}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct ManifestEntry {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub id: u64,
|
||||
#[prost(string, tag = "2")]
|
||||
pub rel_path: ::prost::alloc::string::String,
|
||||
#[prost(uint64, tag = "3")]
|
||||
pub size: u64,
|
||||
#[prost(uint64, tag = "4")]
|
||||
pub mtime: u64,
|
||||
#[prost(uint64, tag = "5")]
|
||||
pub ctime: u64,
|
||||
#[prost(uint64, tag = "6")]
|
||||
pub crtime: u64,
|
||||
#[prost(message, optional, tag = "7")]
|
||||
pub music_metadata: ::core::option::Option<MusicMetadata>,
|
||||
}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct MusicMetadata {
|
||||
#[prost(string, repeated, tag = "1")]
|
||||
pub artist: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
#[prost(string, optional, tag = "2")]
|
||||
pub album_artist: ::core::option::Option<::prost::alloc::string::String>,
|
||||
#[prost(string, tag = "3")]
|
||||
pub album: ::prost::alloc::string::String,
|
||||
#[prost(int32, tag = "4")]
|
||||
pub track_number: i32,
|
||||
#[prost(string, tag = "5")]
|
||||
pub track_title: ::prost::alloc::string::String,
|
||||
#[prost(string, repeated, tag = "6")]
|
||||
pub other_tags: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
#[prost(bytes = "vec", tag = "7")]
|
||||
pub header: ::prost::alloc::vec::Vec<u8>,
|
||||
#[prost(bytes = "vec", repeated, tag = "8")]
|
||||
pub picture_block_headers: ::prost::alloc::vec::Vec<::prost::alloc::vec::Vec<u8>>,
|
||||
#[prost(message, repeated, tag = "9")]
|
||||
pub picture_data_ranges: ::prost::alloc::vec::Vec<PictureDataRange>,
|
||||
#[prost(uint64, tag = "10")]
|
||||
pub real_audio_start: u64,
|
||||
#[prost(uint64, tag = "11")]
|
||||
pub vorbis_comment_offset: u64,
|
||||
#[prost(uint64, tag = "12")]
|
||||
pub vorbis_comment_length: u64,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct PictureDataRange {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub offset: u64,
|
||||
#[prost(uint64, tag = "2")]
|
||||
pub length: u64,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct FileChunk {
|
||||
#[prost(bytes = "vec", tag = "1")]
|
||||
pub data: ::prost::alloc::vec::Vec<u8>,
|
||||
#[prost(uint64, tag = "2")]
|
||||
pub offset: u64,
|
||||
#[prost(uint64, tag = "3")]
|
||||
pub total_size: u64,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct ChangeEvent {
|
||||
/// "create", "modify", or "remove".
|
||||
#[prost(string, tag = "1")]
|
||||
pub kind: ::prost::alloc::string::String,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct GetClientStatusRequest {}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct ClientStatusResponse {
|
||||
#[prost(string, tag = "1")]
|
||||
pub origin_type: ::prost::alloc::string::String,
|
||||
#[prost(string, tag = "2")]
|
||||
pub mountpoint: ::prost::alloc::string::String,
|
||||
#[prost(uint64, tag = "3")]
|
||||
pub file_count: u64,
|
||||
#[prost(bool, tag = "4")]
|
||||
pub server_connected: bool,
|
||||
#[prost(string, tag = "5")]
|
||||
pub server_endpoint: ::prost::alloc::string::String,
|
||||
#[prost(int64, tag = "6")]
|
||||
pub last_reconcile_unix: i64,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct GetMusicMetadataRequest {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub inode: u64,
|
||||
}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct GetMusicMetadataResponse {
|
||||
#[prost(message, optional, tag = "1")]
|
||||
pub metadata: ::core::option::Option<MusicMetadata>,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct UpdateMusicMetadataRequest {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub inode: u64,
|
||||
#[prost(string, repeated, tag = "2")]
|
||||
pub artist: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
#[prost(string, optional, tag = "3")]
|
||||
pub album_artist: ::core::option::Option<::prost::alloc::string::String>,
|
||||
#[prost(string, tag = "4")]
|
||||
pub album: ::prost::alloc::string::String,
|
||||
#[prost(int32, tag = "5")]
|
||||
pub track_number: i32,
|
||||
#[prost(string, tag = "6")]
|
||||
pub track_title: ::prost::alloc::string::String,
|
||||
#[prost(string, repeated, tag = "7")]
|
||||
pub other_tags: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct UpdateMusicMetadataResponse {
|
||||
#[prost(message, optional, tag = "1")]
|
||||
pub metadata: ::core::option::Option<MusicMetadata>,
|
||||
}
|
||||
#[derive(Clone, Copy, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct ListFilesRequest {}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct ListFilesResponse {
|
||||
#[prost(message, repeated, tag = "1")]
|
||||
pub files: ::prost::alloc::vec::Vec<FileEntry>,
|
||||
}
|
||||
/// Human-readable tag fields only. Excludes the binary layout fields
|
||||
/// (header, picture_block_headers, etc.) that only make sense inside
|
||||
/// musicfs's FUSE read-assembly path.
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct FileMetadata {
|
||||
#[prost(string, repeated, tag = "1")]
|
||||
pub artist: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
#[prost(string, optional, tag = "2")]
|
||||
pub album_artist: ::core::option::Option<::prost::alloc::string::String>,
|
||||
#[prost(string, tag = "3")]
|
||||
pub album: ::prost::alloc::string::String,
|
||||
#[prost(int32, tag = "4")]
|
||||
pub track_number: i32,
|
||||
#[prost(string, tag = "5")]
|
||||
pub track_title: ::prost::alloc::string::String,
|
||||
#[prost(string, repeated, tag = "6")]
|
||||
pub other_tags: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
|
||||
pub struct FileEntry {
|
||||
#[prost(uint64, tag = "1")]
|
||||
pub inode: u64,
|
||||
/// Real on-disk path (where tora/librqbit wrote the file).
|
||||
#[prost(string, tag = "2")]
|
||||
pub original_path: ::prost::alloc::string::String,
|
||||
/// Virtual FUSE path ({artist}/{album}/{filename}); updated when metadata
|
||||
/// changes via `UpdateMusicMetadata`.
|
||||
#[prost(string, tag = "3")]
|
||||
pub local_path: ::prost::alloc::string::String,
|
||||
/// Filename component only.
|
||||
#[prost(string, tag = "4")]
|
||||
pub name: ::prost::alloc::string::String,
|
||||
/// Empty if the file has no parsed tags.
|
||||
#[prost(message, optional, tag = "5")]
|
||||
pub metadata: ::core::option::Option<FileMetadata>,
|
||||
}
|
||||
/// Encoded file descriptor set for the `musicfs` package
|
||||
pub const FILE_DESCRIPTOR_SET: &[u8] = &[
|
||||
0x0a, 0xee, 0x4e, 0x0a, 0x0d, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x70, 0x72, 0x6f,
|
||||
0x74, 0x6f, 0x12, 0x07, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x22, 0x40, 0x0a, 0x10, 0x52,
|
||||
0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12,
|
||||
0x2c, 0x0a, 0x07, 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b,
|
||||
0x32, 0x12, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x49, 0x6e, 0x6f, 0x64, 0x65,
|
||||
0x48, 0x61, 0x73, 0x68, 0x52, 0x07, 0x65, 0x6e, 0x74, 0x72, 0x69, 0x65, 0x73, 0x22, 0x5b, 0x0a,
|
||||
0x11, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e,
|
||||
0x73, 0x65, 0x12, 0x2c, 0x0a, 0x07, 0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x64, 0x18, 0x01, 0x20,
|
||||
0x03, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x49, 0x6e,
|
||||
0x6f, 0x64, 0x65, 0x48, 0x61, 0x73, 0x68, 0x52, 0x07, 0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x64,
|
||||
0x12, 0x18, 0x0a, 0x07, 0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x64, 0x18, 0x02, 0x20, 0x03, 0x28,
|
||||
0x04, 0x52, 0x07, 0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x64, 0x22, 0x35, 0x0a, 0x09, 0x49, 0x6e,
|
||||
0x6f, 0x64, 0x65, 0x48, 0x61, 0x73, 0x68, 0x12, 0x14, 0x0a, 0x05, 0x69, 0x6e, 0x6f, 0x64, 0x65,
|
||||
0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x12, 0x12, 0x0a,
|
||||
0x04, 0x68, 0x61, 0x73, 0x68, 0x18, 0x02, 0x20, 0x01, 0x28, 0x04, 0x52, 0x04, 0x68, 0x61, 0x73,
|
||||
0x68, 0x22, 0x2c, 0x0a, 0x12, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61,
|
||||
0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x16, 0x0a, 0x06, 0x69, 0x6e, 0x6f, 0x64, 0x65,
|
||||
0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x04, 0x52, 0x06, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x73, 0x22,
|
||||
0x14, 0x0a, 0x12, 0x47, 0x65, 0x74, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x52, 0x65,
|
||||
0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x4e, 0x0a, 0x0e, 0x47, 0x65, 0x74, 0x46, 0x69, 0x6c, 0x65,
|
||||
0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20,
|
||||
0x01, 0x28, 0x04, 0x52, 0x02, 0x69, 0x64, 0x12, 0x14, 0x0a, 0x05, 0x73, 0x74, 0x61, 0x72, 0x74,
|
||||
0x18, 0x02, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x73, 0x74, 0x61, 0x72, 0x74, 0x12, 0x16, 0x0a,
|
||||
0x06, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x18, 0x03, 0x20, 0x01, 0x28, 0x04, 0x52, 0x06, 0x6c,
|
||||
0x65, 0x6e, 0x67, 0x74, 0x68, 0x22, 0x18, 0x0a, 0x16, 0x53, 0x75, 0x62, 0x73, 0x63, 0x72, 0x69,
|
||||
0x62, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22,
|
||||
0xd1, 0x01, 0x0a, 0x0d, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x45, 0x6e, 0x74, 0x72,
|
||||
0x79, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x02, 0x69,
|
||||
0x64, 0x12, 0x19, 0x0a, 0x08, 0x72, 0x65, 0x6c, 0x5f, 0x70, 0x61, 0x74, 0x68, 0x18, 0x02, 0x20,
|
||||
0x01, 0x28, 0x09, 0x52, 0x07, 0x72, 0x65, 0x6c, 0x50, 0x61, 0x74, 0x68, 0x12, 0x12, 0x0a, 0x04,
|
||||
0x73, 0x69, 0x7a, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x04, 0x52, 0x04, 0x73, 0x69, 0x7a, 0x65,
|
||||
0x12, 0x14, 0x0a, 0x05, 0x6d, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x04, 0x52,
|
||||
0x05, 0x6d, 0x74, 0x69, 0x6d, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x63, 0x74, 0x69, 0x6d, 0x65, 0x18,
|
||||
0x05, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x63, 0x74, 0x69, 0x6d, 0x65, 0x12, 0x16, 0x0a, 0x06,
|
||||
0x63, 0x72, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x04, 0x52, 0x06, 0x63, 0x72,
|
||||
0x74, 0x69, 0x6d, 0x65, 0x12, 0x3d, 0x0a, 0x0e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x5f, 0x6d, 0x65,
|
||||
0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x6d,
|
||||
0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61,
|
||||
0x64, 0x61, 0x74, 0x61, 0x52, 0x0d, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64,
|
||||
0x61, 0x74, 0x61, 0x22, 0x82, 0x04, 0x0a, 0x0d, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x16, 0x0a, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18,
|
||||
0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x12, 0x26, 0x0a,
|
||||
0x0c, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18, 0x02, 0x20,
|
||||
0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0b, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x41, 0x72, 0x74, 0x69,
|
||||
0x73, 0x74, 0x88, 0x01, 0x01, 0x12, 0x14, 0x0a, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x18, 0x03,
|
||||
0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x12, 0x21, 0x0a, 0x0c, 0x74,
|
||||
0x72, 0x61, 0x63, 0x6b, 0x5f, 0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x04, 0x20, 0x01, 0x28,
|
||||
0x05, 0x52, 0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x12, 0x1f,
|
||||
0x0a, 0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x5f, 0x74, 0x69, 0x74, 0x6c, 0x65, 0x18, 0x05, 0x20,
|
||||
0x01, 0x28, 0x09, 0x52, 0x0a, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x54, 0x69, 0x74, 0x6c, 0x65, 0x12,
|
||||
0x1d, 0x0a, 0x0a, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x74, 0x61, 0x67, 0x73, 0x18, 0x06, 0x20,
|
||||
0x03, 0x28, 0x09, 0x52, 0x09, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x54, 0x61, 0x67, 0x73, 0x12, 0x16,
|
||||
0x0a, 0x06, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06,
|
||||
0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x12, 0x32, 0x0a, 0x15, 0x70, 0x69, 0x63, 0x74, 0x75, 0x72,
|
||||
0x65, 0x5f, 0x62, 0x6c, 0x6f, 0x63, 0x6b, 0x5f, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x73, 0x18,
|
||||
0x08, 0x20, 0x03, 0x28, 0x0c, 0x52, 0x13, 0x70, 0x69, 0x63, 0x74, 0x75, 0x72, 0x65, 0x42, 0x6c,
|
||||
0x6f, 0x63, 0x6b, 0x48, 0x65, 0x61, 0x64, 0x65, 0x72, 0x73, 0x12, 0x49, 0x0a, 0x13, 0x70, 0x69,
|
||||
0x63, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x72, 0x61, 0x6e, 0x67, 0x65,
|
||||
0x73, 0x18, 0x09, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66,
|
||||
0x73, 0x2e, 0x50, 0x69, 0x63, 0x74, 0x75, 0x72, 0x65, 0x44, 0x61, 0x74, 0x61, 0x52, 0x61, 0x6e,
|
||||
0x67, 0x65, 0x52, 0x11, 0x70, 0x69, 0x63, 0x74, 0x75, 0x72, 0x65, 0x44, 0x61, 0x74, 0x61, 0x52,
|
||||
0x61, 0x6e, 0x67, 0x65, 0x73, 0x12, 0x28, 0x0a, 0x10, 0x72, 0x65, 0x61, 0x6c, 0x5f, 0x61, 0x75,
|
||||
0x64, 0x69, 0x6f, 0x5f, 0x73, 0x74, 0x61, 0x72, 0x74, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x04, 0x52,
|
||||
0x0e, 0x72, 0x65, 0x61, 0x6c, 0x41, 0x75, 0x64, 0x69, 0x6f, 0x53, 0x74, 0x61, 0x72, 0x74, 0x12,
|
||||
0x32, 0x0a, 0x15, 0x76, 0x6f, 0x72, 0x62, 0x69, 0x73, 0x5f, 0x63, 0x6f, 0x6d, 0x6d, 0x65, 0x6e,
|
||||
0x74, 0x5f, 0x6f, 0x66, 0x66, 0x73, 0x65, 0x74, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x04, 0x52, 0x13,
|
||||
0x76, 0x6f, 0x72, 0x62, 0x69, 0x73, 0x43, 0x6f, 0x6d, 0x6d, 0x65, 0x6e, 0x74, 0x4f, 0x66, 0x66,
|
||||
0x73, 0x65, 0x74, 0x12, 0x32, 0x0a, 0x15, 0x76, 0x6f, 0x72, 0x62, 0x69, 0x73, 0x5f, 0x63, 0x6f,
|
||||
0x6d, 0x6d, 0x65, 0x6e, 0x74, 0x5f, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x18, 0x0c, 0x20, 0x01,
|
||||
0x28, 0x04, 0x52, 0x13, 0x76, 0x6f, 0x72, 0x62, 0x69, 0x73, 0x43, 0x6f, 0x6d, 0x6d, 0x65, 0x6e,
|
||||
0x74, 0x4c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x42, 0x0f, 0x0a, 0x0d, 0x5f, 0x61, 0x6c, 0x62, 0x75,
|
||||
0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x22, 0x42, 0x0a, 0x10, 0x50, 0x69, 0x63, 0x74,
|
||||
0x75, 0x72, 0x65, 0x44, 0x61, 0x74, 0x61, 0x52, 0x61, 0x6e, 0x67, 0x65, 0x12, 0x16, 0x0a, 0x06,
|
||||
0x6f, 0x66, 0x66, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x06, 0x6f, 0x66,
|
||||
0x66, 0x73, 0x65, 0x74, 0x12, 0x16, 0x0a, 0x06, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x18, 0x02,
|
||||
0x20, 0x01, 0x28, 0x04, 0x52, 0x06, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x22, 0x56, 0x0a, 0x09,
|
||||
0x46, 0x69, 0x6c, 0x65, 0x43, 0x68, 0x75, 0x6e, 0x6b, 0x12, 0x12, 0x0a, 0x04, 0x64, 0x61, 0x74,
|
||||
0x61, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x04, 0x64, 0x61, 0x74, 0x61, 0x12, 0x16, 0x0a,
|
||||
0x06, 0x6f, 0x66, 0x66, 0x73, 0x65, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x04, 0x52, 0x06, 0x6f,
|
||||
0x66, 0x66, 0x73, 0x65, 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x5f, 0x73,
|
||||
0x69, 0x7a, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x04, 0x52, 0x09, 0x74, 0x6f, 0x74, 0x61, 0x6c,
|
||||
0x53, 0x69, 0x7a, 0x65, 0x22, 0x21, 0x0a, 0x0b, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x45, 0x76,
|
||||
0x65, 0x6e, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6b, 0x69, 0x6e, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28,
|
||||
0x09, 0x52, 0x04, 0x6b, 0x69, 0x6e, 0x64, 0x22, 0x18, 0x0a, 0x16, 0x47, 0x65, 0x74, 0x43, 0x6c,
|
||||
0x69, 0x65, 0x6e, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73,
|
||||
0x74, 0x22, 0xfa, 0x01, 0x0a, 0x14, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x53, 0x74, 0x61, 0x74,
|
||||
0x75, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1f, 0x0a, 0x0b, 0x6f, 0x72,
|
||||
0x69, 0x67, 0x69, 0x6e, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
|
||||
0x0a, 0x6f, 0x72, 0x69, 0x67, 0x69, 0x6e, 0x54, 0x79, 0x70, 0x65, 0x12, 0x1e, 0x0a, 0x0a, 0x6d,
|
||||
0x6f, 0x75, 0x6e, 0x74, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52,
|
||||
0x0a, 0x6d, 0x6f, 0x75, 0x6e, 0x74, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x66,
|
||||
0x69, 0x6c, 0x65, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x04, 0x52,
|
||||
0x09, 0x66, 0x69, 0x6c, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x29, 0x0a, 0x10, 0x73, 0x65,
|
||||
0x72, 0x76, 0x65, 0x72, 0x5f, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x65, 0x64, 0x18, 0x04,
|
||||
0x20, 0x01, 0x28, 0x08, 0x52, 0x0f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x43, 0x6f, 0x6e, 0x6e,
|
||||
0x65, 0x63, 0x74, 0x65, 0x64, 0x12, 0x27, 0x0a, 0x0f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x5f,
|
||||
0x65, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e,
|
||||
0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x45, 0x6e, 0x64, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0x12, 0x2e,
|
||||
0x0a, 0x13, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x72, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65,
|
||||
0x5f, 0x75, 0x6e, 0x69, 0x78, 0x18, 0x06, 0x20, 0x01, 0x28, 0x03, 0x52, 0x11, 0x6c, 0x61, 0x73,
|
||||
0x74, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x55, 0x6e, 0x69, 0x78, 0x22, 0x2f,
|
||||
0x0a, 0x17, 0x47, 0x65, 0x74, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61,
|
||||
0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x14, 0x0a, 0x05, 0x69, 0x6e, 0x6f,
|
||||
0x64, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x22,
|
||||
0x4e, 0x0a, 0x18, 0x47, 0x65, 0x74, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64,
|
||||
0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x32, 0x0a, 0x08, 0x6d,
|
||||
0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e,
|
||||
0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x08, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x22,
|
||||
0xfc, 0x01, 0x0a, 0x1a, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d,
|
||||
0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x14,
|
||||
0x0a, 0x05, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x69,
|
||||
0x6e, 0x6f, 0x64, 0x65, 0x12, 0x16, 0x0a, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18, 0x02,
|
||||
0x20, 0x03, 0x28, 0x09, 0x52, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x12, 0x26, 0x0a, 0x0c,
|
||||
0x61, 0x6c, 0x62, 0x75, 0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18, 0x03, 0x20, 0x01,
|
||||
0x28, 0x09, 0x48, 0x00, 0x52, 0x0b, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x41, 0x72, 0x74, 0x69, 0x73,
|
||||
0x74, 0x88, 0x01, 0x01, 0x12, 0x14, 0x0a, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x18, 0x04, 0x20,
|
||||
0x01, 0x28, 0x09, 0x52, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x12, 0x21, 0x0a, 0x0c, 0x74, 0x72,
|
||||
0x61, 0x63, 0x6b, 0x5f, 0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x05, 0x20, 0x01, 0x28, 0x05,
|
||||
0x52, 0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x12, 0x1f, 0x0a,
|
||||
0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x5f, 0x74, 0x69, 0x74, 0x6c, 0x65, 0x18, 0x06, 0x20, 0x01,
|
||||
0x28, 0x09, 0x52, 0x0a, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x54, 0x69, 0x74, 0x6c, 0x65, 0x12, 0x1d,
|
||||
0x0a, 0x0a, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x74, 0x61, 0x67, 0x73, 0x18, 0x07, 0x20, 0x03,
|
||||
0x28, 0x09, 0x52, 0x09, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x54, 0x61, 0x67, 0x73, 0x42, 0x0f, 0x0a,
|
||||
0x0d, 0x5f, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x22, 0x51,
|
||||
0x0a, 0x1b, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x32, 0x0a,
|
||||
0x08, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32,
|
||||
0x16, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d,
|
||||
0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x08, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74,
|
||||
0x61, 0x22, 0x12, 0x0a, 0x10, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x69, 0x6c, 0x65, 0x73, 0x52, 0x65,
|
||||
0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x3d, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x69, 0x6c,
|
||||
0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x28, 0x0a, 0x05, 0x66, 0x69,
|
||||
0x6c, 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x6d, 0x75, 0x73, 0x69,
|
||||
0x63, 0x66, 0x73, 0x2e, 0x46, 0x69, 0x6c, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x05, 0x66,
|
||||
0x69, 0x6c, 0x65, 0x73, 0x22, 0xd8, 0x01, 0x0a, 0x0c, 0x46, 0x69, 0x6c, 0x65, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x16, 0x0a, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18,
|
||||
0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x06, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x12, 0x26, 0x0a,
|
||||
0x0c, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x18, 0x02, 0x20,
|
||||
0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0b, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x41, 0x72, 0x74, 0x69,
|
||||
0x73, 0x74, 0x88, 0x01, 0x01, 0x12, 0x14, 0x0a, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x18, 0x03,
|
||||
0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x12, 0x21, 0x0a, 0x0c, 0x74,
|
||||
0x72, 0x61, 0x63, 0x6b, 0x5f, 0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x04, 0x20, 0x01, 0x28,
|
||||
0x05, 0x52, 0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x4e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x12, 0x1f,
|
||||
0x0a, 0x0b, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x5f, 0x74, 0x69, 0x74, 0x6c, 0x65, 0x18, 0x05, 0x20,
|
||||
0x01, 0x28, 0x09, 0x52, 0x0a, 0x74, 0x72, 0x61, 0x63, 0x6b, 0x54, 0x69, 0x74, 0x6c, 0x65, 0x12,
|
||||
0x1d, 0x0a, 0x0a, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x74, 0x61, 0x67, 0x73, 0x18, 0x06, 0x20,
|
||||
0x03, 0x28, 0x09, 0x52, 0x09, 0x6f, 0x74, 0x68, 0x65, 0x72, 0x54, 0x61, 0x67, 0x73, 0x42, 0x0f,
|
||||
0x0a, 0x0d, 0x5f, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x5f, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74, 0x22,
|
||||
0xbe, 0x01, 0x0a, 0x09, 0x46, 0x69, 0x6c, 0x65, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x14, 0x0a,
|
||||
0x05, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x04, 0x52, 0x05, 0x69, 0x6e,
|
||||
0x6f, 0x64, 0x65, 0x12, 0x23, 0x0a, 0x0d, 0x6f, 0x72, 0x69, 0x67, 0x69, 0x6e, 0x61, 0x6c, 0x5f,
|
||||
0x70, 0x61, 0x74, 0x68, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x6f, 0x72, 0x69, 0x67,
|
||||
0x69, 0x6e, 0x61, 0x6c, 0x50, 0x61, 0x74, 0x68, 0x12, 0x1d, 0x0a, 0x0a, 0x6c, 0x6f, 0x63, 0x61,
|
||||
0x6c, 0x5f, 0x70, 0x61, 0x74, 0x68, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x6c, 0x6f,
|
||||
0x63, 0x61, 0x6c, 0x50, 0x61, 0x74, 0x68, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18,
|
||||
0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x36, 0x0a, 0x08, 0x6d,
|
||||
0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e,
|
||||
0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x46, 0x69, 0x6c, 0x65, 0x4d, 0x65, 0x74, 0x61,
|
||||
0x64, 0x61, 0x74, 0x61, 0x48, 0x00, 0x52, 0x08, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61,
|
||||
0x88, 0x01, 0x01, 0x42, 0x0b, 0x0a, 0x09, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61,
|
||||
0x32, 0xdf, 0x02, 0x0a, 0x07, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x46, 0x73, 0x12, 0x42, 0x0a, 0x09,
|
||||
0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x12, 0x19, 0x2e, 0x6d, 0x75, 0x73, 0x69,
|
||||
0x63, 0x66, 0x73, 0x2e, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x52, 0x65, 0x71,
|
||||
0x75, 0x65, 0x73, 0x74, 0x1a, 0x1a, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x52,
|
||||
0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65,
|
||||
0x12, 0x44, 0x0a, 0x0b, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12,
|
||||
0x1b, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x47, 0x65, 0x74, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x16, 0x2e, 0x6d,
|
||||
0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x45,
|
||||
0x6e, 0x74, 0x72, 0x79, 0x30, 0x01, 0x12, 0x44, 0x0a, 0x0b, 0x47, 0x65, 0x74, 0x4d, 0x61, 0x6e,
|
||||
0x69, 0x66, 0x65, 0x73, 0x74, 0x12, 0x1b, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e,
|
||||
0x47, 0x65, 0x74, 0x4d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65,
|
||||
0x73, 0x74, 0x1a, 0x16, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4d, 0x61, 0x6e,
|
||||
0x69, 0x66, 0x65, 0x73, 0x74, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x30, 0x01, 0x12, 0x38, 0x0a, 0x07,
|
||||
0x47, 0x65, 0x74, 0x46, 0x69, 0x6c, 0x65, 0x12, 0x17, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66,
|
||||
0x73, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x69, 0x6c, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74,
|
||||
0x1a, 0x12, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x46, 0x69, 0x6c, 0x65, 0x43,
|
||||
0x68, 0x75, 0x6e, 0x6b, 0x30, 0x01, 0x12, 0x4a, 0x0a, 0x0f, 0x53, 0x75, 0x62, 0x73, 0x63, 0x72,
|
||||
0x69, 0x62, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74, 0x73, 0x12, 0x1f, 0x2e, 0x6d, 0x75, 0x73, 0x69,
|
||||
0x63, 0x66, 0x73, 0x2e, 0x53, 0x75, 0x62, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x45, 0x76, 0x65,
|
||||
0x6e, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x14, 0x2e, 0x6d, 0x75, 0x73,
|
||||
0x69, 0x63, 0x66, 0x73, 0x2e, 0x43, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74,
|
||||
0x30, 0x01, 0x32, 0x61, 0x0a, 0x0c, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x53, 0x74, 0x61, 0x74,
|
||||
0x75, 0x73, 0x12, 0x51, 0x0a, 0x0f, 0x47, 0x65, 0x74, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x53,
|
||||
0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x1f, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e,
|
||||
0x47, 0x65, 0x74, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52,
|
||||
0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1d, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73,
|
||||
0x2e, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x65, 0x73,
|
||||
0x70, 0x6f, 0x6e, 0x73, 0x65, 0x32, 0x8e, 0x02, 0x0a, 0x0d, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74,
|
||||
0x43, 0x6f, 0x6e, 0x74, 0x72, 0x6f, 0x6c, 0x12, 0x57, 0x0a, 0x10, 0x47, 0x65, 0x74, 0x4d, 0x75,
|
||||
0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x20, 0x2e, 0x6d, 0x75,
|
||||
0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x47, 0x65, 0x74, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65,
|
||||
0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e,
|
||||
0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x47, 0x65, 0x74, 0x4d, 0x75, 0x73, 0x69, 0x63,
|
||||
0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65,
|
||||
0x12, 0x60, 0x0a, 0x13, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d,
|
||||
0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x12, 0x23, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66,
|
||||
0x73, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x24, 0x2e, 0x6d,
|
||||
0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x4d, 0x75, 0x73,
|
||||
0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e,
|
||||
0x73, 0x65, 0x12, 0x42, 0x0a, 0x09, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x69, 0x6c, 0x65, 0x73, 0x12,
|
||||
0x19, 0x2e, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x69,
|
||||
0x6c, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1a, 0x2e, 0x6d, 0x75, 0x73,
|
||||
0x69, 0x63, 0x66, 0x73, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x69, 0x6c, 0x65, 0x73, 0x52, 0x65,
|
||||
0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x4a, 0x80, 0x35, 0x0a, 0x07, 0x12, 0x05, 0x00, 0x00, 0xb4,
|
||||
0x01, 0x01, 0x0a, 0x08, 0x0a, 0x01, 0x0c, 0x12, 0x03, 0x00, 0x00, 0x12, 0x0a, 0x08, 0x0a, 0x01,
|
||||
0x02, 0x12, 0x03, 0x02, 0x00, 0x10, 0x0a, 0x62, 0x0a, 0x02, 0x06, 0x00, 0x12, 0x04, 0x06, 0x00,
|
||||
0x20, 0x01, 0x1a, 0x56, 0x20, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x46, 0x73, 0x20, 0x73, 0x65, 0x72,
|
||||
0x76, 0x69, 0x63, 0x65, 0x20, 0x65, 0x78, 0x70, 0x6f, 0x73, 0x65, 0x73, 0x20, 0x61, 0x20, 0x73,
|
||||
0x65, 0x72, 0x76, 0x65, 0x72, 0x2d, 0x73, 0x69, 0x64, 0x65, 0x20, 0x6d, 0x75, 0x73, 0x69, 0x63,
|
||||
0x20, 0x6c, 0x69, 0x62, 0x72, 0x61, 0x72, 0x79, 0x20, 0x74, 0x6f, 0x20, 0x61, 0x20, 0x72, 0x65,
|
||||
0x6d, 0x6f, 0x74, 0x65, 0x20, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x0a, 0x20, 0x46, 0x55,
|
||||
0x53, 0x45, 0x20, 0x6d, 0x6f, 0x75, 0x6e, 0x74, 0x2e, 0x0a, 0x0a, 0x0a, 0x0a, 0x03, 0x06, 0x00,
|
||||
0x01, 0x12, 0x03, 0x06, 0x08, 0x0f, 0x0a, 0xac, 0x02, 0x0a, 0x04, 0x06, 0x00, 0x02, 0x00, 0x12,
|
||||
0x03, 0x0b, 0x02, 0x3e, 0x1a, 0x9e, 0x02, 0x20, 0x48, 0x61, 0x73, 0x68, 0x2d, 0x62, 0x61, 0x73,
|
||||
0x65, 0x64, 0x20, 0x72, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x69, 0x61, 0x74, 0x69, 0x6f,
|
||||
0x6e, 0x2e, 0x20, 0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20, 0x73, 0x65, 0x6e, 0x64, 0x73, 0x20,
|
||||
0x74, 0x68, 0x65, 0x20, 0x28, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x2c, 0x20, 0x68, 0x61, 0x73, 0x68,
|
||||
0x29, 0x20, 0x70, 0x61, 0x69, 0x72, 0x73, 0x20, 0x69, 0x74, 0x0a, 0x20, 0x63, 0x75, 0x72, 0x72,
|
||||
0x65, 0x6e, 0x74, 0x6c, 0x79, 0x20, 0x68, 0x61, 0x73, 0x3b, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65,
|
||||
0x72, 0x20, 0x72, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x64, 0x73, 0x20, 0x77, 0x69, 0x74, 0x68, 0x20,
|
||||
0x74, 0x68, 0x65, 0x20, 0x73, 0x75, 0x62, 0x73, 0x65, 0x74, 0x20, 0x74, 0x68, 0x61, 0x74, 0x20,
|
||||
0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x64, 0x20, 0x28, 0x6e, 0x65, 0x77, 0x20, 0x6f, 0x72, 0x0a,
|
||||
0x20, 0x68, 0x61, 0x73, 0x68, 0x2d, 0x64, 0x69, 0x66, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x74, 0x29,
|
||||
0x20, 0x70, 0x6c, 0x75, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x69, 0x6e, 0x6f, 0x64, 0x65, 0x73,
|
||||
0x20, 0x74, 0x68, 0x65, 0x20, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20, 0x68, 0x61, 0x73, 0x20,
|
||||
0x62, 0x75, 0x74, 0x20, 0x74, 0x68, 0x65, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20, 0x6e,
|
||||
0x6f, 0x20, 0x6c, 0x6f, 0x6e, 0x67, 0x65, 0x72, 0x0a, 0x20, 0x68, 0x61, 0x73, 0x2e, 0x20, 0x43,
|
||||
0x68, 0x65, 0x61, 0x70, 0x20, 0x6f, 0x6e, 0x20, 0x74, 0x68, 0x65, 0x20, 0x77, 0x69, 0x72, 0x65,
|
||||
0x20, 0xe2, 0x80, 0x94, 0x20, 0x6e, 0x6f, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x20, 0x6d, 0x65, 0x74,
|
||||
0x61, 0x64, 0x61, 0x74, 0x61, 0x20, 0x63, 0x72, 0x6f, 0x73, 0x73, 0x65, 0x73, 0x20, 0x75, 0x6e,
|
||||
0x74, 0x69, 0x6c, 0x20, 0x74, 0x68, 0x65, 0x20, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20, 0x61,
|
||||
0x73, 0x6b, 0x73, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x00, 0x01, 0x12, 0x03,
|
||||
0x0b, 0x06, 0x0f, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x00, 0x02, 0x12, 0x03, 0x0b, 0x10,
|
||||
0x20, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x00, 0x03, 0x12, 0x03, 0x0b, 0x2b, 0x3c, 0x0a,
|
||||
0x94, 0x01, 0x0a, 0x04, 0x06, 0x00, 0x02, 0x01, 0x12, 0x03, 0x0f, 0x02, 0x45, 0x1a, 0x86, 0x01,
|
||||
0x20, 0x46, 0x65, 0x74, 0x63, 0x68, 0x20, 0x66, 0x75, 0x6c, 0x6c, 0x20, 0x6d, 0x65, 0x74, 0x61,
|
||||
0x64, 0x61, 0x74, 0x61, 0x20, 0x66, 0x6f, 0x72, 0x20, 0x61, 0x20, 0x73, 0x70, 0x65, 0x63, 0x69,
|
||||
0x66, 0x69, 0x63, 0x20, 0x73, 0x65, 0x74, 0x20, 0x6f, 0x66, 0x20, 0x69, 0x6e, 0x6f, 0x64, 0x65,
|
||||
0x73, 0x2e, 0x20, 0x43, 0x61, 0x6c, 0x6c, 0x65, 0x64, 0x20, 0x61, 0x66, 0x74, 0x65, 0x72, 0x20,
|
||||
0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x0a, 0x20, 0x72, 0x65, 0x74, 0x75, 0x72,
|
||||
0x6e, 0x73, 0x20, 0x61, 0x20, 0x6e, 0x6f, 0x6e, 0x2d, 0x65, 0x6d, 0x70, 0x74, 0x79, 0x20, 0x60,
|
||||
0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x64, 0x60, 0x20, 0x6c, 0x69, 0x73, 0x74, 0x2c, 0x20, 0x66,
|
||||
0x6f, 0x72, 0x20, 0x6a, 0x75, 0x73, 0x74, 0x20, 0x74, 0x68, 0x6f, 0x73, 0x65, 0x20, 0x69, 0x6e,
|
||||
0x6f, 0x64, 0x65, 0x73, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x01, 0x01, 0x12,
|
||||
0x03, 0x0f, 0x06, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x01, 0x02, 0x12, 0x03, 0x0f,
|
||||
0x12, 0x24, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x01, 0x06, 0x12, 0x03, 0x0f, 0x2f, 0x35,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x01, 0x03, 0x12, 0x03, 0x0f, 0x36, 0x43, 0x0a, 0xa8,
|
||||
0x01, 0x0a, 0x04, 0x06, 0x00, 0x02, 0x02, 0x12, 0x03, 0x14, 0x02, 0x45, 0x1a, 0x9a, 0x01, 0x20,
|
||||
0x53, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x20, 0x65, 0x76, 0x65, 0x72, 0x79, 0x20, 0x66, 0x69, 0x6c,
|
||||
0x65, 0x20, 0x65, 0x6e, 0x74, 0x72, 0x79, 0x20, 0x69, 0x6e, 0x20, 0x74, 0x68, 0x65, 0x20, 0x6c,
|
||||
0x69, 0x62, 0x72, 0x61, 0x72, 0x79, 0x2e, 0x20, 0x43, 0x6f, 0x6e, 0x76, 0x65, 0x6e, 0x69, 0x65,
|
||||
0x6e, 0x63, 0x65, 0x20, 0x52, 0x50, 0x43, 0x20, 0x66, 0x6f, 0x72, 0x20, 0x66, 0x69, 0x72, 0x73,
|
||||
0x74, 0x2d, 0x72, 0x75, 0x6e, 0x0a, 0x20, 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70,
|
||||
0x70, 0x69, 0x6e, 0x67, 0x20, 0x6f, 0x72, 0x20, 0x66, 0x75, 0x6c, 0x6c, 0x2d, 0x72, 0x65, 0x66,
|
||||
0x72, 0x65, 0x73, 0x68, 0x2e, 0x20, 0x54, 0x68, 0x65, 0x20, 0x64, 0x69, 0x66, 0x66, 0x2d, 0x62,
|
||||
0x61, 0x73, 0x65, 0x64, 0x20, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x63, 0x69, 0x6c, 0x65, 0x20, 0x70,
|
||||
0x61, 0x74, 0x68, 0x20, 0x69, 0x73, 0x20, 0x74, 0x68, 0x65, 0x0a, 0x20, 0x6e, 0x6f, 0x72, 0x6d,
|
||||
0x61, 0x6c, 0x20, 0x63, 0x61, 0x73, 0x65, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02,
|
||||
0x02, 0x01, 0x12, 0x03, 0x14, 0x06, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x02, 0x02,
|
||||
0x12, 0x03, 0x14, 0x12, 0x24, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x02, 0x06, 0x12, 0x03,
|
||||
0x14, 0x2f, 0x35, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x02, 0x03, 0x12, 0x03, 0x14, 0x36,
|
||||
0x43, 0x0a, 0xf8, 0x01, 0x0a, 0x04, 0x06, 0x00, 0x02, 0x03, 0x12, 0x03, 0x1a, 0x02, 0x39, 0x1a,
|
||||
0xea, 0x01, 0x20, 0x53, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x20, 0x74, 0x68, 0x65, 0x20, 0x62, 0x79,
|
||||
0x74, 0x65, 0x73, 0x20, 0x6f, 0x66, 0x20, 0x6f, 0x6e, 0x65, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x2e,
|
||||
0x20, 0x49, 0x66, 0x20, 0x60, 0x73, 0x74, 0x61, 0x72, 0x74, 0x60, 0x20, 0x61, 0x6e, 0x64, 0x20,
|
||||
0x60, 0x6c, 0x65, 0x6e, 0x67, 0x74, 0x68, 0x60, 0x20, 0x61, 0x72, 0x65, 0x20, 0x62, 0x6f, 0x74,
|
||||
0x68, 0x20, 0x7a, 0x65, 0x72, 0x6f, 0x20, 0x74, 0x68, 0x65, 0x0a, 0x20, 0x73, 0x65, 0x72, 0x76,
|
||||
0x65, 0x72, 0x20, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x65,
|
||||
0x6e, 0x74, 0x69, 0x72, 0x65, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x3b, 0x20, 0x6f, 0x74, 0x68, 0x65,
|
||||
0x72, 0x77, 0x69, 0x73, 0x65, 0x20, 0x6a, 0x75, 0x73, 0x74, 0x20, 0x74, 0x68, 0x65, 0x20, 0x72,
|
||||
0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x65, 0x64, 0x20, 0x72, 0x61, 0x6e, 0x67, 0x65, 0x2e, 0x20,
|
||||
0x54, 0x68, 0x65, 0x0a, 0x20, 0x66, 0x69, 0x72, 0x73, 0x74, 0x20, 0x63, 0x68, 0x75, 0x6e, 0x6b,
|
||||
0x20, 0x63, 0x61, 0x72, 0x72, 0x69, 0x65, 0x73, 0x20, 0x60, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x5f,
|
||||
0x73, 0x69, 0x7a, 0x65, 0x60, 0x20, 0x61, 0x6e, 0x64, 0x20, 0x60, 0x6f, 0x66, 0x66, 0x73, 0x65,
|
||||
0x74, 0x60, 0x3b, 0x20, 0x73, 0x75, 0x62, 0x73, 0x65, 0x71, 0x75, 0x65, 0x6e, 0x74, 0x20, 0x63,
|
||||
0x68, 0x75, 0x6e, 0x6b, 0x73, 0x20, 0x63, 0x6f, 0x6e, 0x74, 0x69, 0x6e, 0x75, 0x65, 0x0a, 0x20,
|
||||
0x66, 0x72, 0x6f, 0x6d, 0x20, 0x74, 0x68, 0x65, 0x72, 0x65, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x06, 0x00, 0x02, 0x03, 0x01, 0x12, 0x03, 0x1a, 0x06, 0x0d, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00,
|
||||
0x02, 0x03, 0x02, 0x12, 0x03, 0x1a, 0x0e, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x03,
|
||||
0x06, 0x12, 0x03, 0x1a, 0x27, 0x2d, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x03, 0x03, 0x12,
|
||||
0x03, 0x1a, 0x2e, 0x37, 0x0a, 0xc2, 0x01, 0x0a, 0x04, 0x06, 0x00, 0x02, 0x04, 0x12, 0x03, 0x1f,
|
||||
0x02, 0x4b, 0x1a, 0xb4, 0x01, 0x20, 0x4c, 0x6f, 0x6e, 0x67, 0x2d, 0x6c, 0x69, 0x76, 0x65, 0x64,
|
||||
0x20, 0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x20, 0x6f, 0x66, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x73,
|
||||
0x79, 0x73, 0x74, 0x65, 0x6d, 0x20, 0x63, 0x68, 0x61, 0x6e, 0x67, 0x65, 0x20, 0x65, 0x76, 0x65,
|
||||
0x6e, 0x74, 0x73, 0x2e, 0x20, 0x54, 0x68, 0x65, 0x20, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20,
|
||||
0x75, 0x73, 0x65, 0x73, 0x20, 0x74, 0x68, 0x65, 0x73, 0x65, 0x20, 0x61, 0x73, 0x0a, 0x20, 0x77,
|
||||
0x61, 0x6b, 0x65, 0x2d, 0x75, 0x70, 0x73, 0x20, 0x74, 0x6f, 0x20, 0x72, 0x65, 0x66, 0x65, 0x74,
|
||||
0x63, 0x68, 0x20, 0x74, 0x68, 0x65, 0x20, 0x6d, 0x61, 0x6e, 0x69, 0x66, 0x65, 0x73, 0x74, 0x3b,
|
||||
0x20, 0x63, 0x6f, 0x72, 0x72, 0x65, 0x63, 0x74, 0x6e, 0x65, 0x73, 0x73, 0x20, 0x61, 0x6c, 0x77,
|
||||
0x61, 0x79, 0x73, 0x20, 0x72, 0x65, 0x73, 0x74, 0x73, 0x20, 0x6f, 0x6e, 0x20, 0x74, 0x68, 0x65,
|
||||
0x20, 0x68, 0x61, 0x73, 0x68, 0x0a, 0x20, 0x64, 0x69, 0x66, 0x66, 0x2c, 0x20, 0x6e, 0x65, 0x76,
|
||||
0x65, 0x72, 0x20, 0x6f, 0x6e, 0x20, 0x74, 0x68, 0x65, 0x20, 0x65, 0x76, 0x65, 0x6e, 0x74, 0x20,
|
||||
0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02,
|
||||
0x04, 0x01, 0x12, 0x03, 0x1f, 0x06, 0x15, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x04, 0x02,
|
||||
0x12, 0x03, 0x1f, 0x16, 0x2c, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x04, 0x06, 0x12, 0x03,
|
||||
0x1f, 0x37, 0x3d, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x00, 0x02, 0x04, 0x03, 0x12, 0x03, 0x1f, 0x3e,
|
||||
0x49, 0x0a, 0x4f, 0x0a, 0x02, 0x06, 0x01, 0x12, 0x04, 0x23, 0x00, 0x25, 0x01, 0x1a, 0x43, 0x20,
|
||||
0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x2d, 0x73, 0x69, 0x64, 0x65, 0x20, 0x73, 0x74, 0x61, 0x74,
|
||||
0x75, 0x73, 0x20, 0x72, 0x65, 0x70, 0x6f, 0x72, 0x74, 0x69, 0x6e, 0x67, 0x20, 0x28, 0x73, 0x65,
|
||||
0x72, 0x76, 0x65, 0x64, 0x20, 0x62, 0x79, 0x20, 0x74, 0x68, 0x65, 0x20, 0x46, 0x55, 0x53, 0x45,
|
||||
0x20, 0x63, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20, 0x70, 0x72, 0x6f, 0x63, 0x65, 0x73, 0x73, 0x29,
|
||||
0x2e, 0x0a, 0x0a, 0x0a, 0x0a, 0x03, 0x06, 0x01, 0x01, 0x12, 0x03, 0x23, 0x08, 0x14, 0x0a, 0x0b,
|
||||
0x0a, 0x04, 0x06, 0x01, 0x02, 0x00, 0x12, 0x03, 0x24, 0x02, 0x4d, 0x0a, 0x0c, 0x0a, 0x05, 0x06,
|
||||
0x01, 0x02, 0x00, 0x01, 0x12, 0x03, 0x24, 0x06, 0x15, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x01, 0x02,
|
||||
0x00, 0x02, 0x12, 0x03, 0x24, 0x16, 0x2c, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x01, 0x02, 0x00, 0x03,
|
||||
0x12, 0x03, 0x24, 0x37, 0x4b, 0x0a, 0x0a, 0x0a, 0x02, 0x06, 0x02, 0x12, 0x04, 0x27, 0x00, 0x2e,
|
||||
0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x06, 0x02, 0x01, 0x12, 0x03, 0x27, 0x08, 0x15, 0x0a, 0x0b, 0x0a,
|
||||
0x04, 0x06, 0x02, 0x02, 0x00, 0x12, 0x03, 0x28, 0x02, 0x53, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02,
|
||||
0x02, 0x00, 0x01, 0x12, 0x03, 0x28, 0x06, 0x16, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x00,
|
||||
0x02, 0x12, 0x03, 0x28, 0x17, 0x2e, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x00, 0x03, 0x12,
|
||||
0x03, 0x28, 0x39, 0x51, 0x0a, 0x0b, 0x0a, 0x04, 0x06, 0x02, 0x02, 0x01, 0x12, 0x03, 0x29, 0x02,
|
||||
0x5c, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x01, 0x01, 0x12, 0x03, 0x29, 0x06, 0x19, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x01, 0x02, 0x12, 0x03, 0x29, 0x1a, 0x34, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x06, 0x02, 0x02, 0x01, 0x03, 0x12, 0x03, 0x29, 0x3f, 0x5a, 0x0a, 0xb4, 0x01, 0x0a, 0x04,
|
||||
0x06, 0x02, 0x02, 0x02, 0x12, 0x03, 0x2d, 0x02, 0x3e, 0x1a, 0xa6, 0x01, 0x20, 0x45, 0x76, 0x65,
|
||||
0x72, 0x79, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x20, 0x6d, 0x75, 0x73, 0x69, 0x63, 0x66, 0x73, 0x20,
|
||||
0x6b, 0x6e, 0x6f, 0x77, 0x73, 0x20, 0x61, 0x62, 0x6f, 0x75, 0x74, 0x2e, 0x20, 0x43, 0x61, 0x6c,
|
||||
0x6c, 0x65, 0x72, 0x20, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x73, 0x20, 0x62, 0x79, 0x20, 0x60,
|
||||
0x6f, 0x72, 0x69, 0x67, 0x69, 0x6e, 0x61, 0x6c, 0x5f, 0x70, 0x61, 0x74, 0x68, 0x60, 0x0a, 0x20,
|
||||
0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x74, 0x6f, 0x20, 0x66, 0x69, 0x6e, 0x64, 0x20, 0x74,
|
||||
0x68, 0x65, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x73, 0x20, 0x6f, 0x66, 0x20, 0x69, 0x6e, 0x74, 0x65,
|
||||
0x72, 0x65, 0x73, 0x74, 0x20, 0x28, 0x65, 0x2e, 0x67, 0x2e, 0x20, 0x61, 0x6c, 0x6c, 0x20, 0x66,
|
||||
0x69, 0x6c, 0x65, 0x73, 0x20, 0x75, 0x6e, 0x64, 0x65, 0x72, 0x20, 0x61, 0x20, 0x73, 0x69, 0x6e,
|
||||
0x67, 0x6c, 0x65, 0x0a, 0x20, 0x74, 0x6f, 0x72, 0x72, 0x65, 0x6e, 0x74, 0x27, 0x73, 0x20, 0x6f,
|
||||
0x75, 0x74, 0x70, 0x75, 0x74, 0x20, 0x64, 0x69, 0x72, 0x65, 0x63, 0x74, 0x6f, 0x72, 0x79, 0x29,
|
||||
0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x02, 0x01, 0x12, 0x03, 0x2d, 0x06, 0x0f,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x06, 0x02, 0x02, 0x02, 0x02, 0x12, 0x03, 0x2d, 0x10, 0x20, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x06, 0x02, 0x02, 0x02, 0x03, 0x12, 0x03, 0x2d, 0x2b, 0x3c, 0x0a, 0x0a, 0x0a, 0x02,
|
||||
0x04, 0x00, 0x12, 0x04, 0x30, 0x00, 0x32, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x00, 0x01, 0x12,
|
||||
0x03, 0x30, 0x08, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x00, 0x02, 0x00, 0x04, 0x12, 0x03, 0x31,
|
||||
0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x00, 0x02, 0x00, 0x12, 0x03, 0x31, 0x02, 0x21, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x00, 0x02, 0x00, 0x06, 0x12, 0x03, 0x31, 0x0b, 0x14, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x00, 0x02, 0x00, 0x01, 0x12, 0x03, 0x31, 0x15, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x00, 0x02, 0x00, 0x03, 0x12, 0x03, 0x31, 0x1f, 0x20, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x01, 0x12,
|
||||
0x04, 0x34, 0x00, 0x37, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x01, 0x01, 0x12, 0x03, 0x34, 0x08,
|
||||
0x19, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x01, 0x02, 0x00, 0x04, 0x12, 0x03, 0x35, 0x02, 0x0a, 0x0a,
|
||||
0x0b, 0x0a, 0x04, 0x04, 0x01, 0x02, 0x00, 0x12, 0x03, 0x35, 0x02, 0x21, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x01, 0x02, 0x00, 0x06, 0x12, 0x03, 0x35, 0x0b, 0x14, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x01,
|
||||
0x02, 0x00, 0x01, 0x12, 0x03, 0x35, 0x15, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x01, 0x02, 0x00,
|
||||
0x03, 0x12, 0x03, 0x35, 0x1f, 0x20, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x01, 0x02, 0x01, 0x04, 0x12,
|
||||
0x03, 0x36, 0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x01, 0x02, 0x01, 0x12, 0x03, 0x36, 0x02,
|
||||
0x1e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x01, 0x02, 0x01, 0x05, 0x12, 0x03, 0x36, 0x0b, 0x11, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x01, 0x02, 0x01, 0x01, 0x12, 0x03, 0x36, 0x12, 0x19, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x01, 0x02, 0x01, 0x03, 0x12, 0x03, 0x36, 0x1c, 0x1d, 0x0a, 0x0a, 0x0a, 0x02, 0x04,
|
||||
0x02, 0x12, 0x04, 0x39, 0x00, 0x3c, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x02, 0x01, 0x12, 0x03,
|
||||
0x39, 0x08, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x02, 0x02, 0x00, 0x05, 0x12, 0x03, 0x3a, 0x02,
|
||||
0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x02, 0x02, 0x00, 0x12, 0x03, 0x3a, 0x02, 0x13, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x02, 0x02, 0x00, 0x01, 0x12, 0x03, 0x3a, 0x09, 0x0e, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x02, 0x02, 0x00, 0x03, 0x12, 0x03, 0x3a, 0x11, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x02,
|
||||
0x02, 0x01, 0x05, 0x12, 0x03, 0x3b, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x02, 0x02, 0x01,
|
||||
0x12, 0x03, 0x3b, 0x02, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x02, 0x02, 0x01, 0x01, 0x12, 0x03,
|
||||
0x3b, 0x09, 0x0d, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x02, 0x02, 0x01, 0x03, 0x12, 0x03, 0x3b, 0x10,
|
||||
0x11, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x03, 0x12, 0x04, 0x3e, 0x00, 0x40, 0x01, 0x0a, 0x0a, 0x0a,
|
||||
0x03, 0x04, 0x03, 0x01, 0x12, 0x03, 0x3e, 0x08, 0x1a, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x03, 0x02,
|
||||
0x00, 0x04, 0x12, 0x03, 0x3f, 0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x03, 0x02, 0x00, 0x12,
|
||||
0x03, 0x3f, 0x02, 0x1d, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x03, 0x02, 0x00, 0x05, 0x12, 0x03, 0x3f,
|
||||
0x0b, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x03, 0x02, 0x00, 0x01, 0x12, 0x03, 0x3f, 0x12, 0x18,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x03, 0x02, 0x00, 0x03, 0x12, 0x03, 0x3f, 0x1b, 0x1c, 0x0a, 0x09,
|
||||
0x0a, 0x02, 0x04, 0x04, 0x12, 0x03, 0x42, 0x00, 0x1d, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x04, 0x01,
|
||||
0x12, 0x03, 0x42, 0x08, 0x1a, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x05, 0x12, 0x04, 0x44, 0x00, 0x49,
|
||||
0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x05, 0x01, 0x12, 0x03, 0x44, 0x08, 0x16, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x05, 0x02, 0x00, 0x05, 0x12, 0x03, 0x45, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04,
|
||||
0x05, 0x02, 0x00, 0x12, 0x03, 0x45, 0x02, 0x10, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x00,
|
||||
0x01, 0x12, 0x03, 0x45, 0x09, 0x0b, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x00, 0x03, 0x12,
|
||||
0x03, 0x45, 0x0e, 0x0f, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x01, 0x05, 0x12, 0x03, 0x47,
|
||||
0x02, 0x08, 0x0a, 0x45, 0x0a, 0x04, 0x04, 0x05, 0x02, 0x01, 0x12, 0x03, 0x47, 0x02, 0x13, 0x1a,
|
||||
0x38, 0x20, 0x57, 0x68, 0x65, 0x6e, 0x20, 0x62, 0x6f, 0x74, 0x68, 0x20, 0x61, 0x72, 0x65, 0x20,
|
||||
0x7a, 0x65, 0x72, 0x6f, 0x20, 0x74, 0x68, 0x65, 0x20, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x20,
|
||||
0x73, 0x74, 0x72, 0x65, 0x61, 0x6d, 0x73, 0x20, 0x74, 0x68, 0x65, 0x20, 0x65, 0x6e, 0x74, 0x69,
|
||||
0x72, 0x65, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02,
|
||||
0x01, 0x01, 0x12, 0x03, 0x47, 0x09, 0x0e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x01, 0x03,
|
||||
0x12, 0x03, 0x47, 0x11, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x02, 0x05, 0x12, 0x03,
|
||||
0x48, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x05, 0x02, 0x02, 0x12, 0x03, 0x48, 0x02, 0x14,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x05, 0x02, 0x02, 0x01, 0x12, 0x03, 0x48, 0x09, 0x0f, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x05, 0x02, 0x02, 0x03, 0x12, 0x03, 0x48, 0x12, 0x13, 0x0a, 0x09, 0x0a, 0x02,
|
||||
0x04, 0x06, 0x12, 0x03, 0x4b, 0x00, 0x21, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x06, 0x01, 0x12, 0x03,
|
||||
0x4b, 0x08, 0x1e, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x07, 0x12, 0x04, 0x4d, 0x00, 0x55, 0x01, 0x0a,
|
||||
0x0a, 0x0a, 0x03, 0x04, 0x07, 0x01, 0x12, 0x03, 0x4d, 0x08, 0x15, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x07, 0x02, 0x00, 0x05, 0x12, 0x03, 0x4e, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x07, 0x02,
|
||||
0x00, 0x12, 0x03, 0x4e, 0x02, 0x10, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x00, 0x01, 0x12,
|
||||
0x03, 0x4e, 0x09, 0x0b, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x00, 0x03, 0x12, 0x03, 0x4e,
|
||||
0x0e, 0x0f, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x01, 0x05, 0x12, 0x03, 0x4f, 0x02, 0x08,
|
||||
0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x07, 0x02, 0x01, 0x12, 0x03, 0x4f, 0x02, 0x16, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x07, 0x02, 0x01, 0x01, 0x12, 0x03, 0x4f, 0x09, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x07, 0x02, 0x01, 0x03, 0x12, 0x03, 0x4f, 0x14, 0x15, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02,
|
||||
0x02, 0x05, 0x12, 0x03, 0x50, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x07, 0x02, 0x02, 0x12,
|
||||
0x03, 0x50, 0x02, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x02, 0x01, 0x12, 0x03, 0x50,
|
||||
0x09, 0x0d, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x02, 0x03, 0x12, 0x03, 0x50, 0x10, 0x11,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x03, 0x05, 0x12, 0x03, 0x51, 0x02, 0x08, 0x0a, 0x0b,
|
||||
0x0a, 0x04, 0x04, 0x07, 0x02, 0x03, 0x12, 0x03, 0x51, 0x02, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x07, 0x02, 0x03, 0x01, 0x12, 0x03, 0x51, 0x09, 0x0e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02,
|
||||
0x03, 0x03, 0x12, 0x03, 0x51, 0x11, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x04, 0x05,
|
||||
0x12, 0x03, 0x52, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x07, 0x02, 0x04, 0x12, 0x03, 0x52,
|
||||
0x02, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x04, 0x01, 0x12, 0x03, 0x52, 0x09, 0x0e,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x04, 0x03, 0x12, 0x03, 0x52, 0x11, 0x12, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x07, 0x02, 0x05, 0x05, 0x12, 0x03, 0x53, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04,
|
||||
0x04, 0x07, 0x02, 0x05, 0x12, 0x03, 0x53, 0x02, 0x14, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02,
|
||||
0x05, 0x01, 0x12, 0x03, 0x53, 0x09, 0x0f, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x05, 0x03,
|
||||
0x12, 0x03, 0x53, 0x12, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x06, 0x06, 0x12, 0x03,
|
||||
0x54, 0x02, 0x0f, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x07, 0x02, 0x06, 0x12, 0x03, 0x54, 0x02, 0x23,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x07, 0x02, 0x06, 0x01, 0x12, 0x03, 0x54, 0x10, 0x1e, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x07, 0x02, 0x06, 0x03, 0x12, 0x03, 0x54, 0x21, 0x22, 0x0a, 0x0a, 0x0a, 0x02,
|
||||
0x04, 0x08, 0x12, 0x04, 0x57, 0x00, 0x64, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x08, 0x01, 0x12,
|
||||
0x03, 0x57, 0x08, 0x15, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x00, 0x04, 0x12, 0x03, 0x58,
|
||||
0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x00, 0x12, 0x03, 0x58, 0x02, 0x1d, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x00, 0x05, 0x12, 0x03, 0x58, 0x0b, 0x11, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x08, 0x02, 0x00, 0x01, 0x12, 0x03, 0x58, 0x12, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x08, 0x02, 0x00, 0x03, 0x12, 0x03, 0x58, 0x1b, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02,
|
||||
0x01, 0x04, 0x12, 0x03, 0x59, 0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x01, 0x12,
|
||||
0x03, 0x59, 0x02, 0x23, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x01, 0x05, 0x12, 0x03, 0x59,
|
||||
0x0b, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x01, 0x01, 0x12, 0x03, 0x59, 0x12, 0x1e,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x01, 0x03, 0x12, 0x03, 0x59, 0x21, 0x22, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x08, 0x02, 0x02, 0x05, 0x12, 0x03, 0x5a, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04,
|
||||
0x04, 0x08, 0x02, 0x02, 0x12, 0x03, 0x5a, 0x02, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02,
|
||||
0x02, 0x01, 0x12, 0x03, 0x5a, 0x09, 0x0e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x02, 0x03,
|
||||
0x12, 0x03, 0x5a, 0x11, 0x12, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x03, 0x05, 0x12, 0x03,
|
||||
0x5b, 0x02, 0x07, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x03, 0x12, 0x03, 0x5b, 0x02, 0x19,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x03, 0x01, 0x12, 0x03, 0x5b, 0x08, 0x14, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x08, 0x02, 0x03, 0x03, 0x12, 0x03, 0x5b, 0x17, 0x18, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x08, 0x02, 0x04, 0x05, 0x12, 0x03, 0x5c, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08,
|
||||
0x02, 0x04, 0x12, 0x03, 0x5c, 0x02, 0x19, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x04, 0x01,
|
||||
0x12, 0x03, 0x5c, 0x09, 0x14, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x04, 0x03, 0x12, 0x03,
|
||||
0x5c, 0x17, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x05, 0x04, 0x12, 0x03, 0x5d, 0x02,
|
||||
0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x05, 0x12, 0x03, 0x5d, 0x02, 0x21, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x08, 0x02, 0x05, 0x05, 0x12, 0x03, 0x5d, 0x0b, 0x11, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x08, 0x02, 0x05, 0x01, 0x12, 0x03, 0x5d, 0x12, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08,
|
||||
0x02, 0x05, 0x03, 0x12, 0x03, 0x5d, 0x1f, 0x20, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x06,
|
||||
0x05, 0x12, 0x03, 0x5e, 0x02, 0x07, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x06, 0x12, 0x03,
|
||||
0x5e, 0x02, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x06, 0x01, 0x12, 0x03, 0x5e, 0x08,
|
||||
0x0e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x06, 0x03, 0x12, 0x03, 0x5e, 0x11, 0x12, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x07, 0x04, 0x12, 0x03, 0x5f, 0x02, 0x0a, 0x0a, 0x0b, 0x0a,
|
||||
0x04, 0x04, 0x08, 0x02, 0x07, 0x12, 0x03, 0x5f, 0x02, 0x2b, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08,
|
||||
0x02, 0x07, 0x05, 0x12, 0x03, 0x5f, 0x0b, 0x10, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x07,
|
||||
0x01, 0x12, 0x03, 0x5f, 0x11, 0x26, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x07, 0x03, 0x12,
|
||||
0x03, 0x5f, 0x29, 0x2a, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x08, 0x04, 0x12, 0x03, 0x60,
|
||||
0x02, 0x0a, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x08, 0x12, 0x03, 0x60, 0x02, 0x34, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x08, 0x06, 0x12, 0x03, 0x60, 0x0b, 0x1b, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x08, 0x02, 0x08, 0x01, 0x12, 0x03, 0x60, 0x1c, 0x2f, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x08, 0x02, 0x08, 0x03, 0x12, 0x03, 0x60, 0x32, 0x33, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02,
|
||||
0x09, 0x05, 0x12, 0x03, 0x61, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x09, 0x12,
|
||||
0x03, 0x61, 0x02, 0x1f, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x09, 0x01, 0x12, 0x03, 0x61,
|
||||
0x09, 0x19, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x09, 0x03, 0x12, 0x03, 0x61, 0x1c, 0x1e,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x0a, 0x05, 0x12, 0x03, 0x62, 0x02, 0x08, 0x0a, 0x0b,
|
||||
0x0a, 0x04, 0x04, 0x08, 0x02, 0x0a, 0x12, 0x03, 0x62, 0x02, 0x24, 0x0a, 0x0c, 0x0a, 0x05, 0x04,
|
||||
0x08, 0x02, 0x0a, 0x01, 0x12, 0x03, 0x62, 0x09, 0x1e, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02,
|
||||
0x0a, 0x03, 0x12, 0x03, 0x62, 0x21, 0x23, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x0b, 0x05,
|
||||
0x12, 0x03, 0x63, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x08, 0x02, 0x0b, 0x12, 0x03, 0x63,
|
||||
0x02, 0x24, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x0b, 0x01, 0x12, 0x03, 0x63, 0x09, 0x1e,
|
||||
0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x08, 0x02, 0x0b, 0x03, 0x12, 0x03, 0x63, 0x21, 0x23, 0x0a, 0x0a,
|
||||
0x0a, 0x02, 0x04, 0x09, 0x12, 0x04, 0x66, 0x00, 0x69, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x09,
|
||||
0x01, 0x12, 0x03, 0x66, 0x08, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x09, 0x02, 0x00, 0x05, 0x12,
|
||||
0x03, 0x67, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x09, 0x02, 0x00, 0x12, 0x03, 0x67, 0x02,
|
||||
0x14, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x09, 0x02, 0x00, 0x01, 0x12, 0x03, 0x67, 0x09, 0x0f, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x09, 0x02, 0x00, 0x03, 0x12, 0x03, 0x67, 0x12, 0x13, 0x0a, 0x0c, 0x0a,
|
||||
0x05, 0x04, 0x09, 0x02, 0x01, 0x05, 0x12, 0x03, 0x68, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04,
|
||||
0x09, 0x02, 0x01, 0x12, 0x03, 0x68, 0x02, 0x14, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x09, 0x02, 0x01,
|
||||
0x01, 0x12, 0x03, 0x68, 0x09, 0x0f, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x09, 0x02, 0x01, 0x03, 0x12,
|
||||
0x03, 0x68, 0x12, 0x13, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x0a, 0x12, 0x04, 0x6b, 0x00, 0x6f, 0x01,
|
||||
0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x0a, 0x01, 0x12, 0x03, 0x6b, 0x08, 0x11, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x0a, 0x02, 0x00, 0x05, 0x12, 0x03, 0x6c, 0x02, 0x07, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0a,
|
||||
0x02, 0x00, 0x12, 0x03, 0x6c, 0x02, 0x11, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a, 0x02, 0x00, 0x01,
|
||||
0x12, 0x03, 0x6c, 0x08, 0x0c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a, 0x02, 0x00, 0x03, 0x12, 0x03,
|
||||
0x6c, 0x0f, 0x10, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a, 0x02, 0x01, 0x05, 0x12, 0x03, 0x6d, 0x02,
|
||||
0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0a, 0x02, 0x01, 0x12, 0x03, 0x6d, 0x02, 0x14, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x0a, 0x02, 0x01, 0x01, 0x12, 0x03, 0x6d, 0x09, 0x0f, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x0a, 0x02, 0x01, 0x03, 0x12, 0x03, 0x6d, 0x12, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a,
|
||||
0x02, 0x02, 0x05, 0x12, 0x03, 0x6e, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0a, 0x02, 0x02,
|
||||
0x12, 0x03, 0x6e, 0x02, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a, 0x02, 0x02, 0x01, 0x12, 0x03,
|
||||
0x6e, 0x09, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0a, 0x02, 0x02, 0x03, 0x12, 0x03, 0x6e, 0x16,
|
||||
0x17, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x0b, 0x12, 0x04, 0x71, 0x00, 0x74, 0x01, 0x0a, 0x0a, 0x0a,
|
||||
0x03, 0x04, 0x0b, 0x01, 0x12, 0x03, 0x71, 0x08, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0b, 0x02,
|
||||
0x00, 0x05, 0x12, 0x03, 0x73, 0x02, 0x08, 0x0a, 0x2f, 0x0a, 0x04, 0x04, 0x0b, 0x02, 0x00, 0x12,
|
||||
0x03, 0x73, 0x02, 0x12, 0x1a, 0x22, 0x20, 0x22, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x22, 0x2c,
|
||||
0x20, 0x22, 0x6d, 0x6f, 0x64, 0x69, 0x66, 0x79, 0x22, 0x2c, 0x20, 0x6f, 0x72, 0x20, 0x22, 0x72,
|
||||
0x65, 0x6d, 0x6f, 0x76, 0x65, 0x22, 0x2e, 0x0a, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0b, 0x02, 0x00,
|
||||
0x01, 0x12, 0x03, 0x73, 0x09, 0x0d, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0b, 0x02, 0x00, 0x03, 0x12,
|
||||
0x03, 0x73, 0x10, 0x11, 0x0a, 0x09, 0x0a, 0x02, 0x04, 0x0c, 0x12, 0x03, 0x76, 0x00, 0x21, 0x0a,
|
||||
0x0a, 0x0a, 0x03, 0x04, 0x0c, 0x01, 0x12, 0x03, 0x76, 0x08, 0x1e, 0x0a, 0x0a, 0x0a, 0x02, 0x04,
|
||||
0x0d, 0x12, 0x04, 0x78, 0x00, 0x7f, 0x01, 0x0a, 0x0a, 0x0a, 0x03, 0x04, 0x0d, 0x01, 0x12, 0x03,
|
||||
0x78, 0x08, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x00, 0x05, 0x12, 0x03, 0x79, 0x02,
|
||||
0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0d, 0x02, 0x00, 0x12, 0x03, 0x79, 0x02, 0x19, 0x0a, 0x0c,
|
||||
0x0a, 0x05, 0x04, 0x0d, 0x02, 0x00, 0x01, 0x12, 0x03, 0x79, 0x09, 0x14, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x0d, 0x02, 0x00, 0x03, 0x12, 0x03, 0x79, 0x17, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d,
|
||||
0x02, 0x01, 0x05, 0x12, 0x03, 0x7a, 0x02, 0x08, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0d, 0x02, 0x01,
|
||||
0x12, 0x03, 0x7a, 0x02, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x01, 0x01, 0x12, 0x03,
|
||||
0x7a, 0x09, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x01, 0x03, 0x12, 0x03, 0x7a, 0x16,
|
||||
0x17, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x02, 0x05, 0x12, 0x03, 0x7b, 0x02, 0x08, 0x0a,
|
||||
0x0b, 0x0a, 0x04, 0x04, 0x0d, 0x02, 0x02, 0x12, 0x03, 0x7b, 0x02, 0x18, 0x0a, 0x0c, 0x0a, 0x05,
|
||||
0x04, 0x0d, 0x02, 0x02, 0x01, 0x12, 0x03, 0x7b, 0x09, 0x13, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d,
|
||||
0x02, 0x02, 0x03, 0x12, 0x03, 0x7b, 0x16, 0x17, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x03,
|
||||
0x05, 0x12, 0x03, 0x7c, 0x02, 0x06, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0d, 0x02, 0x03, 0x12, 0x03,
|
||||
0x7c, 0x02, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x03, 0x01, 0x12, 0x03, 0x7c, 0x07,
|
||||
0x17, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x03, 0x03, 0x12, 0x03, 0x7c, 0x1a, 0x1b, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x04, 0x05, 0x12, 0x03, 0x7d, 0x02, 0x08, 0x0a, 0x0b, 0x0a,
|
||||
0x04, 0x04, 0x0d, 0x02, 0x04, 0x12, 0x03, 0x7d, 0x02, 0x1d, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d,
|
||||
0x02, 0x04, 0x01, 0x12, 0x03, 0x7d, 0x09, 0x18, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x04,
|
||||
0x03, 0x12, 0x03, 0x7d, 0x1b, 0x1c, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x05, 0x05, 0x12,
|
||||
0x03, 0x7e, 0x02, 0x07, 0x0a, 0x0b, 0x0a, 0x04, 0x04, 0x0d, 0x02, 0x05, 0x12, 0x03, 0x7e, 0x02,
|
||||
0x20, 0x0a, 0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x05, 0x01, 0x12, 0x03, 0x7e, 0x08, 0x1b, 0x0a,
|
||||
0x0c, 0x0a, 0x05, 0x04, 0x0d, 0x02, 0x05, 0x03, 0x12, 0x03, 0x7e, 0x1e, 0x1f, 0x0a, 0x0c, 0x0a,
|
||||
0x02, 0x04, 0x0e, 0x12, 0x06, 0x81, 0x01, 0x00, 0x83, 0x01, 0x01, 0x0a, 0x0b, 0x0a, 0x03, 0x04,
|
||||
0x0e, 0x01, 0x12, 0x04, 0x81, 0x01, 0x08, 0x1f, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x0e, 0x02, 0x00,
|
||||
0x05, 0x12, 0x04, 0x82, 0x01, 0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x0e, 0x02, 0x00, 0x12,
|
||||
0x04, 0x82, 0x01, 0x02, 0x13, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x0e, 0x02, 0x00, 0x01, 0x12, 0x04,
|
||||
0x82, 0x01, 0x09, 0x0e, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x0e, 0x02, 0x00, 0x03, 0x12, 0x04, 0x82,
|
||||
0x01, 0x11, 0x12, 0x0a, 0x0c, 0x0a, 0x02, 0x04, 0x0f, 0x12, 0x06, 0x85, 0x01, 0x00, 0x87, 0x01,
|
||||
0x01, 0x0a, 0x0b, 0x0a, 0x03, 0x04, 0x0f, 0x01, 0x12, 0x04, 0x85, 0x01, 0x08, 0x20, 0x0a, 0x0d,
|
||||
0x0a, 0x05, 0x04, 0x0f, 0x02, 0x00, 0x06, 0x12, 0x04, 0x86, 0x01, 0x02, 0x0f, 0x0a, 0x0c, 0x0a,
|
||||
0x04, 0x04, 0x0f, 0x02, 0x00, 0x12, 0x04, 0x86, 0x01, 0x02, 0x1d, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x0f, 0x02, 0x00, 0x01, 0x12, 0x04, 0x86, 0x01, 0x10, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x0f,
|
||||
0x02, 0x00, 0x03, 0x12, 0x04, 0x86, 0x01, 0x1b, 0x1c, 0x0a, 0x0c, 0x0a, 0x02, 0x04, 0x10, 0x12,
|
||||
0x06, 0x89, 0x01, 0x00, 0x91, 0x01, 0x01, 0x0a, 0x0b, 0x0a, 0x03, 0x04, 0x10, 0x01, 0x12, 0x04,
|
||||
0x89, 0x01, 0x08, 0x22, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x00, 0x05, 0x12, 0x04, 0x8a,
|
||||
0x01, 0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x10, 0x02, 0x00, 0x12, 0x04, 0x8a, 0x01, 0x02,
|
||||
0x13, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x00, 0x01, 0x12, 0x04, 0x8a, 0x01, 0x09, 0x0e,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x00, 0x03, 0x12, 0x04, 0x8a, 0x01, 0x11, 0x12, 0x0a,
|
||||
0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x01, 0x04, 0x12, 0x04, 0x8b, 0x01, 0x02, 0x0a, 0x0a, 0x0c,
|
||||
0x0a, 0x04, 0x04, 0x10, 0x02, 0x01, 0x12, 0x04, 0x8b, 0x01, 0x02, 0x1d, 0x0a, 0x0d, 0x0a, 0x05,
|
||||
0x04, 0x10, 0x02, 0x01, 0x05, 0x12, 0x04, 0x8b, 0x01, 0x0b, 0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x10, 0x02, 0x01, 0x01, 0x12, 0x04, 0x8b, 0x01, 0x12, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10,
|
||||
0x02, 0x01, 0x03, 0x12, 0x04, 0x8b, 0x01, 0x1b, 0x1c, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02,
|
||||
0x02, 0x04, 0x12, 0x04, 0x8c, 0x01, 0x02, 0x0a, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x10, 0x02, 0x02,
|
||||
0x12, 0x04, 0x8c, 0x01, 0x02, 0x23, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x02, 0x05, 0x12,
|
||||
0x04, 0x8c, 0x01, 0x0b, 0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x02, 0x01, 0x12, 0x04,
|
||||
0x8c, 0x01, 0x12, 0x1e, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x02, 0x03, 0x12, 0x04, 0x8c,
|
||||
0x01, 0x21, 0x22, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x03, 0x05, 0x12, 0x04, 0x8d, 0x01,
|
||||
0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x10, 0x02, 0x03, 0x12, 0x04, 0x8d, 0x01, 0x02, 0x13,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x03, 0x01, 0x12, 0x04, 0x8d, 0x01, 0x09, 0x0e, 0x0a,
|
||||
0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x03, 0x03, 0x12, 0x04, 0x8d, 0x01, 0x11, 0x12, 0x0a, 0x0d,
|
||||
0x0a, 0x05, 0x04, 0x10, 0x02, 0x04, 0x05, 0x12, 0x04, 0x8e, 0x01, 0x02, 0x07, 0x0a, 0x0c, 0x0a,
|
||||
0x04, 0x04, 0x10, 0x02, 0x04, 0x12, 0x04, 0x8e, 0x01, 0x02, 0x19, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x10, 0x02, 0x04, 0x01, 0x12, 0x04, 0x8e, 0x01, 0x08, 0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10,
|
||||
0x02, 0x04, 0x03, 0x12, 0x04, 0x8e, 0x01, 0x17, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02,
|
||||
0x05, 0x05, 0x12, 0x04, 0x8f, 0x01, 0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x10, 0x02, 0x05,
|
||||
0x12, 0x04, 0x8f, 0x01, 0x02, 0x19, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x05, 0x01, 0x12,
|
||||
0x04, 0x8f, 0x01, 0x09, 0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x05, 0x03, 0x12, 0x04,
|
||||
0x8f, 0x01, 0x17, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x06, 0x04, 0x12, 0x04, 0x90,
|
||||
0x01, 0x02, 0x0a, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x10, 0x02, 0x06, 0x12, 0x04, 0x90, 0x01, 0x02,
|
||||
0x21, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x06, 0x05, 0x12, 0x04, 0x90, 0x01, 0x0b, 0x11,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x06, 0x01, 0x12, 0x04, 0x90, 0x01, 0x12, 0x1c, 0x0a,
|
||||
0x0d, 0x0a, 0x05, 0x04, 0x10, 0x02, 0x06, 0x03, 0x12, 0x04, 0x90, 0x01, 0x1f, 0x20, 0x0a, 0x0c,
|
||||
0x0a, 0x02, 0x04, 0x11, 0x12, 0x06, 0x93, 0x01, 0x00, 0x95, 0x01, 0x01, 0x0a, 0x0b, 0x0a, 0x03,
|
||||
0x04, 0x11, 0x01, 0x12, 0x04, 0x93, 0x01, 0x08, 0x23, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x11, 0x02,
|
||||
0x00, 0x06, 0x12, 0x04, 0x94, 0x01, 0x02, 0x0f, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x11, 0x02, 0x00,
|
||||
0x12, 0x04, 0x94, 0x01, 0x02, 0x1d, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x11, 0x02, 0x00, 0x01, 0x12,
|
||||
0x04, 0x94, 0x01, 0x10, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x11, 0x02, 0x00, 0x03, 0x12, 0x04,
|
||||
0x94, 0x01, 0x1b, 0x1c, 0x0a, 0x0a, 0x0a, 0x02, 0x04, 0x12, 0x12, 0x04, 0x97, 0x01, 0x00, 0x1b,
|
||||
0x0a, 0x0b, 0x0a, 0x03, 0x04, 0x12, 0x01, 0x12, 0x04, 0x97, 0x01, 0x08, 0x18, 0x0a, 0x0c, 0x0a,
|
||||
0x02, 0x04, 0x13, 0x12, 0x06, 0x99, 0x01, 0x00, 0x9b, 0x01, 0x01, 0x0a, 0x0b, 0x0a, 0x03, 0x04,
|
||||
0x13, 0x01, 0x12, 0x04, 0x99, 0x01, 0x08, 0x19, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x13, 0x02, 0x00,
|
||||
0x04, 0x12, 0x04, 0x9a, 0x01, 0x02, 0x0a, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x13, 0x02, 0x00, 0x12,
|
||||
0x04, 0x9a, 0x01, 0x02, 0x1f, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x13, 0x02, 0x00, 0x06, 0x12, 0x04,
|
||||
0x9a, 0x01, 0x0b, 0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x13, 0x02, 0x00, 0x01, 0x12, 0x04, 0x9a,
|
||||
0x01, 0x15, 0x1a, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x13, 0x02, 0x00, 0x03, 0x12, 0x04, 0x9a, 0x01,
|
||||
0x1d, 0x1e, 0x0a, 0xb9, 0x01, 0x0a, 0x02, 0x04, 0x14, 0x12, 0x06, 0xa0, 0x01, 0x00, 0xa7, 0x01,
|
||||
0x01, 0x1a, 0xaa, 0x01, 0x20, 0x48, 0x75, 0x6d, 0x61, 0x6e, 0x2d, 0x72, 0x65, 0x61, 0x64, 0x61,
|
||||
0x62, 0x6c, 0x65, 0x20, 0x74, 0x61, 0x67, 0x20, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x20, 0x6f,
|
||||
0x6e, 0x6c, 0x79, 0x2e, 0x20, 0x45, 0x78, 0x63, 0x6c, 0x75, 0x64, 0x65, 0x73, 0x20, 0x74, 0x68,
|
||||
0x65, 0x20, 0x62, 0x69, 0x6e, 0x61, 0x72, 0x79, 0x20, 0x6c, 0x61, 0x79, 0x6f, 0x75, 0x74, 0x20,
|
||||
0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x0a, 0x20, 0x28, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x2c,
|
||||
0x20, 0x70, 0x69, 0x63, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x62, 0x6c, 0x6f, 0x63, 0x6b, 0x5f, 0x68,
|
||||
0x65, 0x61, 0x64, 0x65, 0x72, 0x73, 0x2c, 0x20, 0x65, 0x74, 0x63, 0x2e, 0x29, 0x20, 0x74, 0x68,
|
||||
0x61, 0x74, 0x20, 0x6f, 0x6e, 0x6c, 0x79, 0x20, 0x6d, 0x61, 0x6b, 0x65, 0x20, 0x73, 0x65, 0x6e,
|
||||
0x73, 0x65, 0x20, 0x69, 0x6e, 0x73, 0x69, 0x64, 0x65, 0x0a, 0x20, 0x6d, 0x75, 0x73, 0x69, 0x63,
|
||||
0x66, 0x73, 0x27, 0x73, 0x20, 0x46, 0x55, 0x53, 0x45, 0x20, 0x72, 0x65, 0x61, 0x64, 0x2d, 0x61,
|
||||
0x73, 0x73, 0x65, 0x6d, 0x62, 0x6c, 0x79, 0x20, 0x70, 0x61, 0x74, 0x68, 0x2e, 0x0a, 0x0a, 0x0b,
|
||||
0x0a, 0x03, 0x04, 0x14, 0x01, 0x12, 0x04, 0xa0, 0x01, 0x08, 0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x14, 0x02, 0x00, 0x04, 0x12, 0x04, 0xa1, 0x01, 0x02, 0x0a, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x14,
|
||||
0x02, 0x00, 0x12, 0x04, 0xa1, 0x01, 0x02, 0x1d, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x00,
|
||||
0x05, 0x12, 0x04, 0xa1, 0x01, 0x0b, 0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x00, 0x01,
|
||||
0x12, 0x04, 0xa1, 0x01, 0x12, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x00, 0x03, 0x12,
|
||||
0x04, 0xa1, 0x01, 0x1b, 0x1c, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x01, 0x04, 0x12, 0x04,
|
||||
0xa2, 0x01, 0x02, 0x0a, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x14, 0x02, 0x01, 0x12, 0x04, 0xa2, 0x01,
|
||||
0x02, 0x23, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x01, 0x05, 0x12, 0x04, 0xa2, 0x01, 0x0b,
|
||||
0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x01, 0x01, 0x12, 0x04, 0xa2, 0x01, 0x12, 0x1e,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x01, 0x03, 0x12, 0x04, 0xa2, 0x01, 0x21, 0x22, 0x0a,
|
||||
0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x02, 0x05, 0x12, 0x04, 0xa3, 0x01, 0x02, 0x08, 0x0a, 0x0c,
|
||||
0x0a, 0x04, 0x04, 0x14, 0x02, 0x02, 0x12, 0x04, 0xa3, 0x01, 0x02, 0x13, 0x0a, 0x0d, 0x0a, 0x05,
|
||||
0x04, 0x14, 0x02, 0x02, 0x01, 0x12, 0x04, 0xa3, 0x01, 0x09, 0x0e, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x14, 0x02, 0x02, 0x03, 0x12, 0x04, 0xa3, 0x01, 0x11, 0x12, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14,
|
||||
0x02, 0x03, 0x05, 0x12, 0x04, 0xa4, 0x01, 0x02, 0x07, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x14, 0x02,
|
||||
0x03, 0x12, 0x04, 0xa4, 0x01, 0x02, 0x19, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x03, 0x01,
|
||||
0x12, 0x04, 0xa4, 0x01, 0x08, 0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x03, 0x03, 0x12,
|
||||
0x04, 0xa4, 0x01, 0x17, 0x18, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x04, 0x05, 0x12, 0x04,
|
||||
0xa5, 0x01, 0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x14, 0x02, 0x04, 0x12, 0x04, 0xa5, 0x01,
|
||||
0x02, 0x19, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x04, 0x01, 0x12, 0x04, 0xa5, 0x01, 0x09,
|
||||
0x14, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x04, 0x03, 0x12, 0x04, 0xa5, 0x01, 0x17, 0x18,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x14, 0x02, 0x05, 0x04, 0x12, 0x04, 0xa6, 0x01, 0x02, 0x0a, 0x0a,
|
||||
0x0c, 0x0a, 0x04, 0x04, 0x14, 0x02, 0x05, 0x12, 0x04, 0xa6, 0x01, 0x02, 0x21, 0x0a, 0x0d, 0x0a,
|
||||
0x05, 0x04, 0x14, 0x02, 0x05, 0x05, 0x12, 0x04, 0xa6, 0x01, 0x0b, 0x11, 0x0a, 0x0d, 0x0a, 0x05,
|
||||
0x04, 0x14, 0x02, 0x05, 0x01, 0x12, 0x04, 0xa6, 0x01, 0x12, 0x1c, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x14, 0x02, 0x05, 0x03, 0x12, 0x04, 0xa6, 0x01, 0x1f, 0x20, 0x0a, 0x0c, 0x0a, 0x02, 0x04, 0x15,
|
||||
0x12, 0x06, 0xa9, 0x01, 0x00, 0xb4, 0x01, 0x01, 0x0a, 0x0b, 0x0a, 0x03, 0x04, 0x15, 0x01, 0x12,
|
||||
0x04, 0xa9, 0x01, 0x08, 0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x00, 0x05, 0x12, 0x04,
|
||||
0xaa, 0x01, 0x02, 0x08, 0x0a, 0x0c, 0x0a, 0x04, 0x04, 0x15, 0x02, 0x00, 0x12, 0x04, 0xaa, 0x01,
|
||||
0x02, 0x13, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x00, 0x01, 0x12, 0x04, 0xaa, 0x01, 0x09,
|
||||
0x0e, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x00, 0x03, 0x12, 0x04, 0xaa, 0x01, 0x11, 0x12,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x01, 0x05, 0x12, 0x04, 0xac, 0x01, 0x02, 0x08, 0x0a,
|
||||
0x47, 0x0a, 0x04, 0x04, 0x15, 0x02, 0x01, 0x12, 0x04, 0xac, 0x01, 0x02, 0x1b, 0x1a, 0x39, 0x20,
|
||||
0x52, 0x65, 0x61, 0x6c, 0x20, 0x6f, 0x6e, 0x2d, 0x64, 0x69, 0x73, 0x6b, 0x20, 0x70, 0x61, 0x74,
|
||||
0x68, 0x20, 0x28, 0x77, 0x68, 0x65, 0x72, 0x65, 0x20, 0x74, 0x6f, 0x72, 0x61, 0x2f, 0x6c, 0x69,
|
||||
0x62, 0x72, 0x71, 0x62, 0x69, 0x74, 0x20, 0x77, 0x72, 0x6f, 0x74, 0x65, 0x20, 0x74, 0x68, 0x65,
|
||||
0x20, 0x66, 0x69, 0x6c, 0x65, 0x29, 0x2e, 0x0a, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x01,
|
||||
0x01, 0x12, 0x04, 0xac, 0x01, 0x09, 0x16, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x01, 0x03,
|
||||
0x12, 0x04, 0xac, 0x01, 0x19, 0x1a, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x02, 0x05, 0x12,
|
||||
0x04, 0xaf, 0x01, 0x02, 0x08, 0x0a, 0x7a, 0x0a, 0x04, 0x04, 0x15, 0x02, 0x02, 0x12, 0x04, 0xaf,
|
||||
0x01, 0x02, 0x18, 0x1a, 0x6c, 0x20, 0x56, 0x69, 0x72, 0x74, 0x75, 0x61, 0x6c, 0x20, 0x46, 0x55,
|
||||
0x53, 0x45, 0x20, 0x70, 0x61, 0x74, 0x68, 0x20, 0x28, 0x7b, 0x61, 0x72, 0x74, 0x69, 0x73, 0x74,
|
||||
0x7d, 0x2f, 0x7b, 0x61, 0x6c, 0x62, 0x75, 0x6d, 0x7d, 0x2f, 0x7b, 0x66, 0x69, 0x6c, 0x65, 0x6e,
|
||||
0x61, 0x6d, 0x65, 0x7d, 0x29, 0x3b, 0x20, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x64, 0x20, 0x77,
|
||||
0x68, 0x65, 0x6e, 0x20, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x0a, 0x20, 0x63, 0x68,
|
||||
0x61, 0x6e, 0x67, 0x65, 0x73, 0x20, 0x76, 0x69, 0x61, 0x20, 0x60, 0x55, 0x70, 0x64, 0x61, 0x74,
|
||||
0x65, 0x4d, 0x75, 0x73, 0x69, 0x63, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x60, 0x2e,
|
||||
0x0a, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x02, 0x01, 0x12, 0x04, 0xaf, 0x01, 0x09, 0x13,
|
||||
0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x02, 0x03, 0x12, 0x04, 0xaf, 0x01, 0x16, 0x17, 0x0a,
|
||||
0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x03, 0x05, 0x12, 0x04, 0xb1, 0x01, 0x02, 0x08, 0x0a, 0x28,
|
||||
0x0a, 0x04, 0x04, 0x15, 0x02, 0x03, 0x12, 0x04, 0xb1, 0x01, 0x02, 0x12, 0x1a, 0x1a, 0x20, 0x46,
|
||||
0x69, 0x6c, 0x65, 0x6e, 0x61, 0x6d, 0x65, 0x20, 0x63, 0x6f, 0x6d, 0x70, 0x6f, 0x6e, 0x65, 0x6e,
|
||||
0x74, 0x20, 0x6f, 0x6e, 0x6c, 0x79, 0x2e, 0x0a, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x03,
|
||||
0x01, 0x12, 0x04, 0xb1, 0x01, 0x09, 0x0d, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x03, 0x03,
|
||||
0x12, 0x04, 0xb1, 0x01, 0x10, 0x11, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15, 0x02, 0x04, 0x04, 0x12,
|
||||
0x04, 0xb3, 0x01, 0x02, 0x0a, 0x0a, 0x35, 0x0a, 0x04, 0x04, 0x15, 0x02, 0x04, 0x12, 0x04, 0xb3,
|
||||
0x01, 0x02, 0x25, 0x1a, 0x27, 0x20, 0x45, 0x6d, 0x70, 0x74, 0x79, 0x20, 0x69, 0x66, 0x20, 0x74,
|
||||
0x68, 0x65, 0x20, 0x66, 0x69, 0x6c, 0x65, 0x20, 0x68, 0x61, 0x73, 0x20, 0x6e, 0x6f, 0x20, 0x70,
|
||||
0x61, 0x72, 0x73, 0x65, 0x64, 0x20, 0x74, 0x61, 0x67, 0x73, 0x2e, 0x0a, 0x0a, 0x0d, 0x0a, 0x05,
|
||||
0x04, 0x15, 0x02, 0x04, 0x06, 0x12, 0x04, 0xb3, 0x01, 0x0b, 0x17, 0x0a, 0x0d, 0x0a, 0x05, 0x04,
|
||||
0x15, 0x02, 0x04, 0x01, 0x12, 0x04, 0xb3, 0x01, 0x18, 0x20, 0x0a, 0x0d, 0x0a, 0x05, 0x04, 0x15,
|
||||
0x02, 0x04, 0x03, 0x12, 0x04, 0xb3, 0x01, 0x23, 0x24, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f,
|
||||
0x33,
|
||||
];
|
||||
include!("musicfs.tonic.rs");
|
||||
// @@protoc_insertion_point(module)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,15 @@
|
||||
pub mod musicfs {
|
||||
include!("generated/musicfs/musicfs.rs");
|
||||
}
|
||||
|
||||
pub use musicfs::{
|
||||
ChangeEvent, ClientStatusResponse, FileChunk, FileEntry, FileMetadata, GetClientStatusRequest,
|
||||
GetFileRequest, GetManifestRequest, GetMetadataRequest, GetMusicMetadataRequest,
|
||||
GetMusicMetadataResponse, InodeHash, ListFilesRequest, ListFilesResponse, ManifestEntry,
|
||||
MusicMetadata, PictureDataRange, ReconcileRequest, ReconcileResponse, SubscribeEventsRequest,
|
||||
UpdateMusicMetadataRequest, UpdateMusicMetadataResponse,
|
||||
client_control_server::{ClientControl, ClientControlServer},
|
||||
client_status_server::{ClientStatus, ClientStatusServer},
|
||||
music_fs_client::MusicFsClient,
|
||||
music_fs_server::{MusicFs, MusicFsServer},
|
||||
};
|
||||
@@ -0,0 +1,26 @@
|
||||
[package]
|
||||
name = "musicfs-server"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
default-run = "musicfs-server"
|
||||
|
||||
[[bin]]
|
||||
name = "musicfs-server"
|
||||
path = "src/bin/musicfs-server.rs"
|
||||
|
||||
[dependencies]
|
||||
musicfs-core.workspace = true
|
||||
musicfs-proto.workspace = true
|
||||
tonic.workspace = true
|
||||
tonic-health.workspace = true
|
||||
tonic-reflection.workspace = true
|
||||
tokio.workspace = true
|
||||
anyhow.workspace = true
|
||||
async-trait.workspace = true
|
||||
clap.workspace = true
|
||||
notify.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile.workspace = true
|
||||
@@ -0,0 +1,102 @@
|
||||
use std::{net::SocketAddr, path::PathBuf};
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use clap::Parser;
|
||||
use musicfs_core::logging::{LogConfig, init};
|
||||
use musicfs_server::server::{
|
||||
state::ServerState,
|
||||
transport::{self, TransportArgs},
|
||||
watcher::ServerWatcher,
|
||||
};
|
||||
use tokio::signal::unix::{SignalKind, signal};
|
||||
use tracing::{error, info};
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(version, about, long_about = None)]
|
||||
struct Args {
|
||||
/// Directory containing the music library to serve.
|
||||
#[arg(short, long, required = true)]
|
||||
source: PathBuf,
|
||||
|
||||
/// Address:port the transport should listen on (e.g. 0.0.0.0:50051).
|
||||
#[arg(short, long, default_value = "0.0.0.0:50051")]
|
||||
listen: SocketAddr,
|
||||
|
||||
/// Name of the transport implementation to use. Today only "grpc".
|
||||
#[arg(short, long, default_value = "grpc")]
|
||||
transport: String,
|
||||
|
||||
/// Directory for daily-rotated log files.
|
||||
#[arg(long, default_value = "./logs")]
|
||||
log_dir: PathBuf,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let args = Args::parse();
|
||||
|
||||
// Bind the guard for the whole process so the non-blocking file writer
|
||||
// flushes on exit. Initialized before anything else so even early
|
||||
// failures land in the log.
|
||||
let _guard = init(LogConfig {
|
||||
log_dir: args.log_dir.clone(),
|
||||
file_prefix: "musicfs-server".to_string(),
|
||||
max_files: 7,
|
||||
});
|
||||
|
||||
let source = args.source.clone();
|
||||
info!(
|
||||
source = %source.display(),
|
||||
transport = %args.transport,
|
||||
listen = %args.listen,
|
||||
log_dir = %args.log_dir.display(),
|
||||
"musicfs-server starting"
|
||||
);
|
||||
|
||||
if !source.is_dir() {
|
||||
error!(source = %source.display(), "source is not a readable directory");
|
||||
return Err(anyhow::anyhow!(
|
||||
"source is not a readable directory: {}",
|
||||
source.display()
|
||||
));
|
||||
}
|
||||
|
||||
let state = ServerState::new();
|
||||
state
|
||||
.replace_all(&source)
|
||||
.with_context(|| format!("initial scan of {}", source.display()))?;
|
||||
info!(entries = state.manifest().len(), "initial scan complete");
|
||||
|
||||
let (_watcher, events_rx) = ServerWatcher::spawn(source.clone(), state.clone());
|
||||
|
||||
let transport = transport::build(
|
||||
&args.transport,
|
||||
TransportArgs {
|
||||
listen: args.listen,
|
||||
},
|
||||
)
|
||||
.with_context(|| format!("building transport {:?}", args.transport))?;
|
||||
|
||||
info!(
|
||||
transport = %args.transport,
|
||||
listen = %args.listen,
|
||||
"transport ready"
|
||||
);
|
||||
|
||||
let server_task = tokio::spawn(async move {
|
||||
if let Err(e) = transport.run(state, events_rx).await {
|
||||
error!(error = %e, "transport ended with error");
|
||||
}
|
||||
});
|
||||
|
||||
let mut sigint = signal(SignalKind::interrupt()).expect("register SIGINT");
|
||||
let mut sigterm = signal(SignalKind::terminate()).expect("register SIGTERM");
|
||||
|
||||
tokio::select! {
|
||||
_ = sigint.recv() => info!("received SIGINT, shutting down"),
|
||||
_ = sigterm.recv() => info!("received SIGTERM, shutting down"),
|
||||
_ = server_task => info!("transport task exited"),
|
||||
}
|
||||
|
||||
return Ok(());
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
pub mod server;
|
||||
@@ -0,0 +1,93 @@
|
||||
use musicfs_core::compute_item_hash;
|
||||
use musicfs_core::music::metadata::MusicMetadata;
|
||||
|
||||
/// One row of the in-memory manifest: every field the client needs to
|
||||
/// reconstruct an `Item` whose hash matches the server's hash.
|
||||
///
|
||||
/// Field semantics:
|
||||
/// - `id` is the server filesystem inode. The client uses it as the FUSE
|
||||
/// inode, which keeps `compute_hash` inputs identical on both sides.
|
||||
/// - `rel_path` is the path relative to the server's `--source` root, using
|
||||
/// `/` as separator. The client stores it as `Item.original_path` and uses
|
||||
/// it as the byte-source locator.
|
||||
/// - `mtime` / `ctime` / `crtime` are seconds since the Unix epoch. These are
|
||||
/// the exact three time fields `compute_hash` consumes.
|
||||
/// - `size` is the real on-disk file size in bytes (the client's virtual size
|
||||
/// is derived from `music_metadata.virtual_size(size)` when present).
|
||||
/// - `music_metadata` is the fully-encoded metadata produced server-side via
|
||||
/// `parse_music_metadata_for_path`. The client never parses audio.
|
||||
///
|
||||
/// Wire format conversion lives in `transport/grpc.rs` (`From<ManifestEntry>`
|
||||
/// for the generated proto type).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ManifestEntry {
|
||||
pub id: u64,
|
||||
pub rel_path: String,
|
||||
pub size: u64,
|
||||
pub mtime: u64,
|
||||
pub ctime: u64,
|
||||
pub crtime: u64,
|
||||
pub music_metadata: Option<MusicMetadata>,
|
||||
}
|
||||
|
||||
impl ManifestEntry {
|
||||
pub fn hash(&self) -> u64 {
|
||||
return compute_item_hash(
|
||||
self.id,
|
||||
self.rel_path.as_bytes(),
|
||||
self.ctime,
|
||||
self.mtime,
|
||||
self.crtime,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn manifest_entry_clones_fields_into_copy() {
|
||||
let entry = ManifestEntry {
|
||||
id: 42,
|
||||
rel_path: "Artist/Album/track.flac".to_string(),
|
||||
size: 12345,
|
||||
mtime: 1_700_000_000,
|
||||
ctime: 1_699_999_000,
|
||||
crtime: 1_699_990_000,
|
||||
music_metadata: None,
|
||||
};
|
||||
|
||||
let cloned = entry.clone();
|
||||
assert_eq!(cloned.id, entry.id);
|
||||
assert_eq!(cloned.rel_path, entry.rel_path);
|
||||
assert_eq!(cloned.size, entry.size);
|
||||
assert_eq!(cloned.mtime, entry.mtime);
|
||||
assert_eq!(cloned.ctime, entry.ctime);
|
||||
assert_eq!(cloned.crtime, entry.crtime);
|
||||
assert!(cloned.music_metadata.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn manifest_entry_carries_music_metadata_when_present() {
|
||||
let mm = MusicMetadata {
|
||||
artist: vec!["Test Artist".to_string()],
|
||||
album: "Test Album".to_string(),
|
||||
track_title: "Test Title".to_string(),
|
||||
track_number: 3,
|
||||
..MusicMetadata::default()
|
||||
};
|
||||
let entry = ManifestEntry {
|
||||
id: 1,
|
||||
rel_path: "a/b.flac".to_string(),
|
||||
size: 100,
|
||||
mtime: 0,
|
||||
ctime: 0,
|
||||
crtime: 0,
|
||||
music_metadata: Some(mm.clone()),
|
||||
};
|
||||
|
||||
assert_eq!(entry.music_metadata.as_ref().unwrap().album, "Test Album");
|
||||
assert_eq!(entry.music_metadata.as_ref().unwrap().track_number, 3);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
pub mod manifest;
|
||||
pub mod range;
|
||||
pub mod state;
|
||||
pub mod transport;
|
||||
pub mod watcher;
|
||||
@@ -0,0 +1,175 @@
|
||||
/// Parsed `Range: bytes=...` request header.
|
||||
///
|
||||
/// Supported forms (RFC 7233):
|
||||
/// - `bytes=a-b` → `StartEnd(a, b)` (inclusive)
|
||||
/// - `bytes=a-` → `Start(a)`
|
||||
/// - `bytes=-N` → `Suffix(N)` (last N bytes)
|
||||
///
|
||||
/// Returns `None` if the header is missing, uses a unit other than `bytes`,
|
||||
/// or fails to parse. Multiple ranges (`bytes=a-b,c-d`) are not supported —
|
||||
/// the first is used and the rest ignored.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum ByteRange {
|
||||
StartEnd(u64, u64),
|
||||
Start(u64),
|
||||
Suffix(u64),
|
||||
}
|
||||
|
||||
pub fn parse_range_header(header: &str) -> Option<ByteRange> {
|
||||
let header = header.trim();
|
||||
let rest = header.strip_prefix("bytes=")?;
|
||||
let first = rest.split(',').next()?.trim();
|
||||
let (left, right) = first.split_once('-')?;
|
||||
let left = left.trim();
|
||||
let right = right.trim();
|
||||
|
||||
return match (left.is_empty(), right.is_empty()) {
|
||||
(false, false) => Some(ByteRange::StartEnd(left.parse().ok()?, right.parse().ok()?)),
|
||||
(false, true) => Some(ByteRange::Start(left.parse().ok()?)),
|
||||
(true, false) => Some(ByteRange::Suffix(right.parse().ok()?)),
|
||||
(true, true) => None,
|
||||
};
|
||||
}
|
||||
|
||||
/// Resolve a parsed range against a real file size, yielding an absolute
|
||||
/// `(start, length)` pair suitable for `seek + read`.
|
||||
///
|
||||
/// Returns `None` if the resolved range is unsatisfiable (e.g. start past
|
||||
/// end of file). The returned `start` is clamped to `[0, size]` and `length`
|
||||
/// is clamped to not exceed `size - start`.
|
||||
pub fn resolve_range(range: ByteRange, size: u64) -> Option<(u64, u64)> {
|
||||
let (start, end_inclusive) = match range {
|
||||
ByteRange::StartEnd(a, b) => {
|
||||
if a >= size || a > b {
|
||||
return None;
|
||||
}
|
||||
(a, b.min(size - 1))
|
||||
}
|
||||
ByteRange::Start(a) => {
|
||||
if a >= size {
|
||||
return None;
|
||||
}
|
||||
(a, size - 1)
|
||||
}
|
||||
ByteRange::Suffix(n) => {
|
||||
if n == 0 || size == 0 {
|
||||
return None;
|
||||
}
|
||||
// A suffix larger than the file clamps to the whole file.
|
||||
let start = size.saturating_sub(n);
|
||||
(start, size - 1)
|
||||
}
|
||||
};
|
||||
return Some((start, end_inclusive - start + 1));
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_start_end() {
|
||||
assert_eq!(
|
||||
parse_range_header("bytes=0-499"),
|
||||
Some(ByteRange::StartEnd(0, 499))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_range_header("bytes=500-999"),
|
||||
Some(ByteRange::StartEnd(500, 999))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_start_only() {
|
||||
assert_eq!(
|
||||
parse_range_header("bytes=9500-"),
|
||||
Some(ByteRange::Start(9500))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_suffix() {
|
||||
assert_eq!(
|
||||
parse_range_header("bytes=-500"),
|
||||
Some(ByteRange::Suffix(500))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_ignores_additional_ranges_after_first() {
|
||||
assert_eq!(
|
||||
parse_range_header("bytes=0-499,1000-1499"),
|
||||
Some(ByteRange::StartEnd(0, 499))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_returns_none_for_non_bytes_unit() {
|
||||
assert_eq!(parse_range_header("items=0-4"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_returns_none_for_malformed() {
|
||||
assert_eq!(parse_range_header("not a range"), None);
|
||||
assert_eq!(parse_range_header("bytes="), None);
|
||||
assert_eq!(parse_range_header("bytes=-"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_start_end_in_bounds() {
|
||||
assert_eq!(
|
||||
resolve_range(ByteRange::StartEnd(0, 499), 1000),
|
||||
Some((0, 500))
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_range(ByteRange::StartEnd(100, 199), 1000),
|
||||
Some((100, 100))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_start_end_clamps_end_to_size() {
|
||||
assert_eq!(
|
||||
resolve_range(ByteRange::StartEnd(900, 2000), 1000),
|
||||
Some((900, 100))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_start_end_unsatisfiable_when_start_past_size() {
|
||||
assert_eq!(resolve_range(ByteRange::StartEnd(1500, 2000), 1000), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_start_only() {
|
||||
assert_eq!(resolve_range(ByteRange::Start(900), 1000), Some((900, 100)));
|
||||
assert_eq!(resolve_range(ByteRange::Start(0), 1000), Some((0, 1000)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_start_unsatisfiable_when_past_size() {
|
||||
assert_eq!(resolve_range(ByteRange::Start(1500), 1000), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_suffix() {
|
||||
assert_eq!(
|
||||
resolve_range(ByteRange::Suffix(500), 1000),
|
||||
Some((500, 500))
|
||||
);
|
||||
assert_eq!(resolve_range(ByteRange::Suffix(1), 1000), Some((999, 1)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_suffix_larger_than_size_clamps_to_full_file() {
|
||||
assert_eq!(
|
||||
resolve_range(ByteRange::Suffix(2000), 1000),
|
||||
Some((0, 1000))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_suffix_zero_unsatisfiable() {
|
||||
assert_eq!(resolve_range(ByteRange::Suffix(0), 1000), None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,260 @@
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
fs, io,
|
||||
path::{Path, PathBuf},
|
||||
sync::{Arc, Mutex},
|
||||
time::{SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
|
||||
use crate::server::manifest::ManifestEntry;
|
||||
use musicfs_core::FileAttrs;
|
||||
use musicfs_core::music::parse::parse_music_metadata_for_path;
|
||||
use tracing::info;
|
||||
|
||||
/// Server-side entry for one file. Built once on startup from a directory
|
||||
/// scan and refreshed by the watcher on inotify events.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct FileEntry {
|
||||
pub abs_path: PathBuf,
|
||||
pub rel_path: String,
|
||||
pub attrs: FileAttrs,
|
||||
pub music_metadata: Option<musicfs_core::music::metadata::MusicMetadata>,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct ServerState {
|
||||
inner: Arc<Mutex<HashMap<u64, FileEntry>>>,
|
||||
}
|
||||
|
||||
impl ServerState {
|
||||
pub fn new() -> Self {
|
||||
return ServerState {
|
||||
inner: Arc::new(Mutex::new(HashMap::new())),
|
||||
};
|
||||
}
|
||||
|
||||
/// Replace the entire map with a fresh scan of `source`. Used on startup
|
||||
/// and on watcher-driven reconciliations.
|
||||
pub fn replace_all(&self, source: &Path) -> io::Result<bool> {
|
||||
let entries = scan_directory(source)?;
|
||||
let new_map: HashMap<u64, FileEntry> = entries.into_iter().collect();
|
||||
let mut map = self.inner.lock().unwrap();
|
||||
if *map == new_map {
|
||||
return Ok(false);
|
||||
}
|
||||
let count = new_map.len();
|
||||
*map = new_map;
|
||||
info!(count, "server state: scan complete (changed)");
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
/// Snapshot the current state into a manifest. Order is by inode ascending
|
||||
/// so two scans of an unchanged library serialize identically.
|
||||
pub fn manifest(&self) -> Vec<ManifestEntry> {
|
||||
let map = self.inner.lock().unwrap();
|
||||
let mut entries: Vec<ManifestEntry> = map
|
||||
.iter()
|
||||
.map(|(id, file)| ManifestEntry {
|
||||
id: *id,
|
||||
rel_path: file.rel_path.clone(),
|
||||
size: file.attrs.size,
|
||||
mtime: secs(file.attrs.mtime),
|
||||
ctime: secs(file.attrs.ctime),
|
||||
crtime: secs(file.attrs.crtime),
|
||||
music_metadata: file.music_metadata.clone(),
|
||||
})
|
||||
.collect();
|
||||
entries.sort_by_key(|e| e.id);
|
||||
return entries;
|
||||
}
|
||||
|
||||
pub fn lookup(&self, id: u64) -> Option<FileEntry> {
|
||||
return self.inner.lock().unwrap().get(&id).cloned();
|
||||
}
|
||||
}
|
||||
|
||||
fn secs(time: SystemTime) -> u64 {
|
||||
return time
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0);
|
||||
}
|
||||
|
||||
fn scan_directory(source: &Path) -> io::Result<Vec<(u64, FileEntry)>> {
|
||||
let mut out = Vec::new();
|
||||
walk(source, source, &mut out)?;
|
||||
return Ok(out);
|
||||
}
|
||||
|
||||
fn walk(source: &Path, dir: &Path, out: &mut Vec<(u64, FileEntry)>) -> io::Result<()> {
|
||||
for entry in fs::read_dir(dir)? {
|
||||
let entry = entry?;
|
||||
let path = entry.path();
|
||||
let file_type = entry.file_type()?;
|
||||
if file_type.is_dir() {
|
||||
walk(source, &path, out)?;
|
||||
continue;
|
||||
}
|
||||
if !file_type.is_file() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let metadata = entry.metadata()?;
|
||||
let rel_path = path
|
||||
.strip_prefix(source)
|
||||
.map(|p| p.to_path_buf())
|
||||
.unwrap_or_else(|_| path.clone())
|
||||
.to_string_lossy()
|
||||
.replace('\\', "/");
|
||||
|
||||
let inode = metadata.ino();
|
||||
let music_metadata = parse_music_metadata_for_path(&path);
|
||||
let file_entry = FileEntry {
|
||||
abs_path: path,
|
||||
rel_path,
|
||||
attrs: FileAttrs::from(&metadata),
|
||||
music_metadata,
|
||||
};
|
||||
out.push((inode, file_entry));
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::Write;
|
||||
|
||||
#[test]
|
||||
fn replace_all_loads_files_into_state() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
|
||||
let mut f = fs::File::create(source.join("a.txt")).unwrap();
|
||||
f.write_all(b"hello").unwrap();
|
||||
drop(f);
|
||||
let mut f = fs::File::create(source.join("b.txt")).unwrap();
|
||||
f.write_all(b"world!").unwrap();
|
||||
drop(f);
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
|
||||
let manifest = state.manifest();
|
||||
assert_eq!(manifest.len(), 2);
|
||||
assert!(manifest.iter().any(|e| e.rel_path == "a.txt"));
|
||||
assert!(manifest.iter().any(|e| e.rel_path == "b.txt"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn manifest_entries_sorted_by_id_ascending() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
// Create files in any order; inode order is determined by the FS.
|
||||
for name in ["z.txt", "a.txt", "m.txt"] {
|
||||
fs::write(source.join(name), b"x").unwrap();
|
||||
}
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
|
||||
let ids: Vec<u64> = state.manifest().iter().map(|e| e.id).collect();
|
||||
let mut sorted = ids.clone();
|
||||
sorted.sort();
|
||||
assert_eq!(ids, sorted);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_clears_existing_entries() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"x").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
assert_eq!(state.manifest().len(), 1);
|
||||
|
||||
fs::remove_file(source.join("a.txt")).unwrap();
|
||||
state.replace_all(source).unwrap();
|
||||
assert_eq!(state.manifest().len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_true_on_first_scan() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"x").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
assert!(state.replace_all(source).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_false_when_unchanged() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"hello").unwrap();
|
||||
fs::write(source.join("b.txt"), b"world").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
assert!(!state.replace_all(source).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_true_after_file_added() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"x").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
|
||||
fs::write(source.join("b.txt"), b"y").unwrap();
|
||||
assert!(state.replace_all(source).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_true_after_file_removed() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"x").unwrap();
|
||||
fs::write(source.join("b.txt"), b"y").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
|
||||
fs::remove_file(source.join("a.txt")).unwrap();
|
||||
assert!(state.replace_all(source).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_true_after_file_renamed() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("old.txt"), b"x").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
assert!(!state.replace_all(source).unwrap());
|
||||
|
||||
fs::rename(source.join("old.txt"), source.join("new.txt")).unwrap();
|
||||
assert!(state.replace_all(source).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replace_all_returns_true_after_content_modified() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let source = tmp.path();
|
||||
fs::write(source.join("a.txt"), b"original").unwrap();
|
||||
|
||||
let state = ServerState::new();
|
||||
state.replace_all(source).unwrap();
|
||||
assert!(!state.replace_all(source).unwrap());
|
||||
|
||||
fs::write(source.join("a.txt"), b"modified content").unwrap();
|
||||
assert!(state.replace_all(source).unwrap());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
use std::{
|
||||
fs,
|
||||
io::{Read, Seek, SeekFrom},
|
||||
path::PathBuf,
|
||||
pin::Pin,
|
||||
};
|
||||
|
||||
use anyhow::Result;
|
||||
use async_trait::async_trait;
|
||||
use tokio::sync::broadcast::Receiver;
|
||||
use tonic::{Request, Response, Status, transport::Server};
|
||||
|
||||
use crate::server::manifest::ManifestEntry as DomainManifestEntry;
|
||||
use crate::server::state::ServerState;
|
||||
use crate::server::transport::{MusicTransport, TransportArgs};
|
||||
use crate::server::watcher::{ChangeEvent, ChangeKind};
|
||||
use musicfs_core::music::metadata::MusicMetadata;
|
||||
use musicfs_proto as proto_types;
|
||||
use musicfs_proto::MusicFs as MusicFsTrait;
|
||||
use musicfs_proto::{
|
||||
ChangeEvent as ProtoChangeEvent, FileChunk, GetFileRequest, GetManifestRequest,
|
||||
GetMetadataRequest, InodeHash, ManifestEntry, MusicFsServer,
|
||||
MusicMetadata as ProtoMusicMetadata, PictureDataRange, ReconcileRequest, ReconcileResponse,
|
||||
SubscribeEventsRequest,
|
||||
};
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
pub struct GrpcTransport {
|
||||
args: TransportArgs,
|
||||
}
|
||||
|
||||
impl GrpcTransport {
|
||||
pub fn new(args: TransportArgs) -> Self {
|
||||
return GrpcTransport { args };
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl MusicTransport for GrpcTransport {
|
||||
async fn run(self: Box<Self>, state: ServerState, events: Receiver<ChangeEvent>) -> Result<()> {
|
||||
let service = MusicFsService {
|
||||
state,
|
||||
events: tokio::sync::Mutex::new(events),
|
||||
};
|
||||
let listen = self.args.listen;
|
||||
info!(%listen, "musicfs-server gRPC listening");
|
||||
|
||||
// Standard gRPC Health Checking Protocol (grpc.health.v1.Health).
|
||||
let (reporter, health_service) = tonic_health::server::health_reporter();
|
||||
reporter
|
||||
.set_serving::<MusicFsServer<MusicFsService>>()
|
||||
.await;
|
||||
|
||||
// v1alpha covers older clients; v1 is the current standard.
|
||||
let reflection_v1 = tonic_reflection::server::Builder::configure()
|
||||
.register_encoded_file_descriptor_set(musicfs_proto::musicfs::FILE_DESCRIPTOR_SET)
|
||||
.register_encoded_file_descriptor_set(tonic_health::pb::FILE_DESCRIPTOR_SET)
|
||||
.build_v1()?;
|
||||
let reflection_v1alpha = tonic_reflection::server::Builder::configure()
|
||||
.register_encoded_file_descriptor_set(musicfs_proto::musicfs::FILE_DESCRIPTOR_SET)
|
||||
.register_encoded_file_descriptor_set(tonic_health::pb::FILE_DESCRIPTOR_SET)
|
||||
.build_v1alpha()?;
|
||||
|
||||
Server::builder()
|
||||
.add_service(health_service)
|
||||
.add_service(reflection_v1)
|
||||
.add_service(reflection_v1alpha)
|
||||
.add_service(MusicFsServer::new(service))
|
||||
.serve(listen)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
|
||||
struct MusicFsService {
|
||||
state: ServerState,
|
||||
events: tokio::sync::Mutex<Receiver<ChangeEvent>>,
|
||||
}
|
||||
|
||||
type BoxStream<T> = Pin<Box<dyn tokio_stream::Stream<Item = Result<T, Status>> + Send>>;
|
||||
|
||||
#[tonic::async_trait]
|
||||
impl MusicFsTrait for MusicFsService {
|
||||
type GetManifestStream = BoxStream<ManifestEntry>;
|
||||
type GetMetadataStream = BoxStream<ManifestEntry>;
|
||||
type GetFileStream = BoxStream<FileChunk>;
|
||||
type SubscribeEventsStream = BoxStream<ProtoChangeEvent>;
|
||||
|
||||
async fn reconcile(
|
||||
&self,
|
||||
request: Request<ReconcileRequest>,
|
||||
) -> Result<Response<ReconcileResponse>, Status> {
|
||||
let client_entries: std::collections::HashMap<u64, u64> = request
|
||||
.into_inner()
|
||||
.entries
|
||||
.into_iter()
|
||||
.map(|ih| (ih.inode, ih.hash))
|
||||
.collect();
|
||||
debug!(client_entries = client_entries.len(), "reconcile");
|
||||
|
||||
let server_manifest = self.state.manifest();
|
||||
let mut changed: Vec<InodeHash> = Vec::new();
|
||||
let mut deleted: Vec<u64> = Vec::new();
|
||||
|
||||
for entry in &server_manifest {
|
||||
let entry_hash = entry.hash();
|
||||
match client_entries.get(&entry.id) {
|
||||
Some(client_hash) if *client_hash == entry_hash => {}
|
||||
_ => {
|
||||
changed.push(InodeHash {
|
||||
inode: entry.id,
|
||||
hash: entry_hash,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
for client_inode in client_entries.keys() {
|
||||
if !server_manifest.iter().any(|e| &e.id == client_inode) {
|
||||
deleted.push(*client_inode);
|
||||
}
|
||||
}
|
||||
|
||||
debug!(
|
||||
changed = changed.len(),
|
||||
deleted = deleted.len(),
|
||||
"reconcile result"
|
||||
);
|
||||
return Ok(Response::new(ReconcileResponse { changed, deleted }));
|
||||
}
|
||||
|
||||
async fn get_metadata(
|
||||
&self,
|
||||
request: Request<GetMetadataRequest>,
|
||||
) -> Result<Response<Self::GetMetadataStream>, Status> {
|
||||
let wanted: std::collections::HashSet<u64> =
|
||||
request.into_inner().inodes.into_iter().collect();
|
||||
debug!(wanted = wanted.len(), "get_metadata");
|
||||
let entries: Vec<DomainManifestEntry> = self
|
||||
.state
|
||||
.manifest()
|
||||
.into_iter()
|
||||
.filter(|e| wanted.contains(&e.id))
|
||||
.collect();
|
||||
let stream = tokio_stream::iter(
|
||||
entries
|
||||
.into_iter()
|
||||
.map(|e| Ok::<ManifestEntry, Status>(e.into())),
|
||||
);
|
||||
return Ok(Response::new(Box::pin(stream)));
|
||||
}
|
||||
|
||||
async fn get_manifest(
|
||||
&self,
|
||||
_request: Request<GetManifestRequest>,
|
||||
) -> Result<Response<Self::GetManifestStream>, Status> {
|
||||
debug!("get_manifest");
|
||||
let entries: Vec<DomainManifestEntry> = self.state.manifest();
|
||||
let stream = tokio_stream::iter(
|
||||
entries
|
||||
.into_iter()
|
||||
.map(|e| Ok::<ManifestEntry, Status>(e.into())),
|
||||
);
|
||||
return Ok(Response::new(Box::pin(stream)));
|
||||
}
|
||||
|
||||
async fn get_file(
|
||||
&self,
|
||||
request: Request<GetFileRequest>,
|
||||
) -> Result<Response<Self::GetFileStream>, Status> {
|
||||
let req = request.into_inner();
|
||||
let id = req.id;
|
||||
debug!(%id, "get_file");
|
||||
let entry = match self.state.lookup(id) {
|
||||
Some(e) => e,
|
||||
None => {
|
||||
debug!(%id, "get_file: not found");
|
||||
return Err(Status::not_found(format!("no file with id {id}")));
|
||||
}
|
||||
};
|
||||
let abs_path: PathBuf = entry.abs_path.clone();
|
||||
let total_size = entry.attrs.size;
|
||||
let start = if req.start == 0 && req.length == 0 {
|
||||
0u64
|
||||
} else {
|
||||
req.start
|
||||
};
|
||||
let length = if req.start == 0 && req.length == 0 {
|
||||
total_size
|
||||
} else if req.length == 0 {
|
||||
total_size.saturating_sub(start)
|
||||
} else {
|
||||
req.length
|
||||
};
|
||||
|
||||
let chunk =
|
||||
tokio::task::spawn_blocking(move || read_range_blocking(&abs_path, start, length))
|
||||
.await
|
||||
.map_err(|e| {
|
||||
warn!(%id, error = %e, "get_file: join blocking read failed");
|
||||
Status::internal(format!("join blocking read: {e}"))
|
||||
})?
|
||||
.map_err(|e| {
|
||||
warn!(%id, error = %e, "get_file: read range failed");
|
||||
Status::internal(format!("read file range: {e}"))
|
||||
})?;
|
||||
|
||||
// Split into <=1MB messages: gRPC's default max is 4MB and music files
|
||||
// routinely exceed it.
|
||||
const CHUNK_SIZE: usize = 1024 * 1024;
|
||||
let chunk_stream = tokio_stream::iter(
|
||||
chunk
|
||||
.chunks(CHUNK_SIZE)
|
||||
.enumerate()
|
||||
.map(|(i, c)| {
|
||||
Ok::<FileChunk, Status>(FileChunk {
|
||||
data: c.to_vec(),
|
||||
offset: start + (i * CHUNK_SIZE) as u64,
|
||||
total_size,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>(),
|
||||
);
|
||||
return Ok(Response::new(Box::pin(chunk_stream)));
|
||||
}
|
||||
|
||||
async fn subscribe_events(
|
||||
&self,
|
||||
_request: Request<SubscribeEventsRequest>,
|
||||
) -> Result<Response<Self::SubscribeEventsStream>, Status> {
|
||||
debug!("subscribe_events");
|
||||
// Take an owned receiver (the guard is dropped at the end of this
|
||||
// statement) so it can move into the 'static spawned task, and so each
|
||||
// subscriber gets its own receiver rather than contending on one.
|
||||
let mut rx = self.events.lock().await.resubscribe();
|
||||
let (tx, rx_stream) =
|
||||
tokio::sync::mpsc::unbounded_channel::<Result<ProtoChangeEvent, Status>>();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
match rx.recv().await {
|
||||
Ok(event) => {
|
||||
let proto = ProtoChangeEvent {
|
||||
kind: kind_to_string(event.kind),
|
||||
};
|
||||
if tx.send(Ok(proto)).is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => {
|
||||
warn!(lagged = n, "subscribe_events: lagged; continuing");
|
||||
continue;
|
||||
}
|
||||
Err(tokio::sync::broadcast::error::RecvError::Closed) => {
|
||||
debug!("subscribe_events: broadcast closed; ending stream");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
let stream = tokio_stream::wrappers::UnboundedReceiverStream::new(rx_stream);
|
||||
return Ok(Response::new(Box::pin(stream)));
|
||||
}
|
||||
}
|
||||
|
||||
fn read_range_blocking(
|
||||
path: &std::path::Path,
|
||||
start: u64,
|
||||
length: u64,
|
||||
) -> std::io::Result<Vec<u8>> {
|
||||
let mut file = fs::File::open(path)?;
|
||||
file.seek(SeekFrom::Start(start))?;
|
||||
let mut buf = vec![0u8; length as usize];
|
||||
let mut filled = 0usize;
|
||||
while filled < buf.len() {
|
||||
let n = file.read(&mut buf[filled..])?;
|
||||
if n == 0 {
|
||||
buf.truncate(filled);
|
||||
break;
|
||||
}
|
||||
filled += n;
|
||||
}
|
||||
return Ok(buf);
|
||||
}
|
||||
|
||||
fn kind_to_string(kind: ChangeKind) -> String {
|
||||
return match kind {
|
||||
ChangeKind::Create => "create".to_string(),
|
||||
ChangeKind::Modify => "modify".to_string(),
|
||||
ChangeKind::Remove => "remove".to_string(),
|
||||
};
|
||||
}
|
||||
|
||||
impl From<DomainManifestEntry> for ManifestEntry {
|
||||
fn from(entry: DomainManifestEntry) -> Self {
|
||||
return ManifestEntry {
|
||||
id: entry.id,
|
||||
rel_path: entry.rel_path,
|
||||
size: entry.size,
|
||||
mtime: entry.mtime,
|
||||
ctime: entry.ctime,
|
||||
crtime: entry.crtime,
|
||||
music_metadata: entry.music_metadata.map(music_metadata_to_proto),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
fn music_metadata_to_proto(mm: MusicMetadata) -> ProtoMusicMetadata {
|
||||
ProtoMusicMetadata {
|
||||
artist: mm.artist,
|
||||
album_artist: mm.album_artist,
|
||||
album: mm.album,
|
||||
track_number: mm.track_number,
|
||||
track_title: mm.track_title,
|
||||
other_tags: mm.other_tags,
|
||||
header: mm.header,
|
||||
picture_block_headers: mm.picture_block_headers,
|
||||
picture_data_ranges: mm
|
||||
.picture_data_ranges
|
||||
.into_iter()
|
||||
.map(|(offset, length)| PictureDataRange { offset, length })
|
||||
.collect(),
|
||||
real_audio_start: mm.real_audio_start,
|
||||
vorbis_comment_offset: mm.vorbis_comment_offset,
|
||||
vorbis_comment_length: mm.vorbis_comment_length,
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(unused_imports)]
|
||||
use proto_types as _proto_types_anchor;
|
||||
@@ -0,0 +1,39 @@
|
||||
pub mod grpc;
|
||||
|
||||
use std::net::SocketAddr;
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
use async_trait::async_trait;
|
||||
use tokio::sync::broadcast::Receiver;
|
||||
|
||||
use crate::server::state::ServerState;
|
||||
use crate::server::watcher::ChangeEvent;
|
||||
|
||||
/// Transport-agnostic musicfs server. A single implementation is wired in
|
||||
/// today (`grpc`); the factory in [`build`] is the seam where additional
|
||||
/// transports (HTTP/3, raw QUIC, in-process for tests, ...) get plugged in
|
||||
/// without touching the binary.
|
||||
///
|
||||
/// `run` consumes `self` — a transport serves exactly once. It receives the
|
||||
/// shared [`ServerState`] and a fresh broadcast receiver for change events.
|
||||
#[async_trait]
|
||||
pub trait MusicTransport: Send + Sync + 'static {
|
||||
async fn run(self: Box<Self>, state: ServerState, events: Receiver<ChangeEvent>) -> Result<()>;
|
||||
}
|
||||
|
||||
/// Arguments every transport understands. Transports may extend this with
|
||||
/// their own configuration via their constructors.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct TransportArgs {
|
||||
pub listen: SocketAddr,
|
||||
}
|
||||
|
||||
/// Construct the named transport. The list of accepted names is intentionally
|
||||
/// discoverable (a single match arm) so adding a transport means adding an
|
||||
/// arm here plus a module under `transport/`.
|
||||
pub fn build(name: &str, args: TransportArgs) -> Result<Box<dyn MusicTransport>> {
|
||||
return match name {
|
||||
"grpc" => Ok(Box::new(grpc::GrpcTransport::new(args))),
|
||||
other => Err(anyhow!("unknown transport: {other}")),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
use std::{
|
||||
path::PathBuf,
|
||||
sync::{Arc, Mutex},
|
||||
thread,
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use notify::{EventKind, RecursiveMode, Watcher};
|
||||
use tokio::sync::broadcast;
|
||||
|
||||
use crate::server::state::ServerState;
|
||||
use tracing::{error, info};
|
||||
|
||||
const POLL_INTERVAL: Duration = Duration::from_secs(15);
|
||||
|
||||
/// A change observed by the watcher. Pushed onto the broadcast channel for
|
||||
/// `/events` subscribers. The client treats these as wake-ups: correctness
|
||||
/// always rests on the subsequent `/manifest` hash diff.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ChangeEvent {
|
||||
pub kind: ChangeKind,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub enum ChangeKind {
|
||||
Create,
|
||||
Modify,
|
||||
Remove,
|
||||
}
|
||||
|
||||
/// Server-side file watcher. Owns a background std thread driving `notify`
|
||||
/// (inotify on Linux). On any filesystem event under `source` it:
|
||||
/// 1. Rebuilds the shared [`ServerState`] from a fresh scan.
|
||||
/// 2. Broadcasts a [`ChangeEvent`] so connected `/events` clients wake up.
|
||||
///
|
||||
/// The state rebuild is full-scan rather than incremental. This matches the
|
||||
/// existing `LocalOriginFileWatcher` pattern, keeps the watcher simple, and
|
||||
/// is cheap on a LAN-scale library. Hash-diff reconciliation on the client
|
||||
/// absorbs any over-reporting.
|
||||
pub struct ServerWatcher {
|
||||
_events_tx: broadcast::Sender<ChangeEvent>,
|
||||
_worker: Arc<Mutex<Option<thread::JoinHandle<()>>>>,
|
||||
}
|
||||
|
||||
impl ServerWatcher {
|
||||
/// Spawn the watcher. Returns the broadcast receiver that `/events`
|
||||
/// handlers subscribe to.
|
||||
pub fn spawn(source: PathBuf, state: ServerState) -> (Self, broadcast::Receiver<ChangeEvent>) {
|
||||
let (events_tx, events_rx) = broadcast::channel(64);
|
||||
let worker_tx = events_tx.clone();
|
||||
let handle = thread::spawn(move || {
|
||||
run_watcher_loop(source, state, worker_tx);
|
||||
});
|
||||
let watcher = ServerWatcher {
|
||||
_events_tx: events_tx,
|
||||
_worker: Arc::new(Mutex::new(Some(handle))),
|
||||
};
|
||||
return (watcher, events_rx);
|
||||
}
|
||||
}
|
||||
|
||||
fn run_watcher_loop(
|
||||
source: PathBuf,
|
||||
state: ServerState,
|
||||
events_tx: broadcast::Sender<ChangeEvent>,
|
||||
) {
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
let mut watcher = match notify::recommended_watcher(tx) {
|
||||
Ok(w) => w,
|
||||
Err(e) => {
|
||||
error!(error = %e, "server watcher: failed to create inotify watcher");
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
if let Err(e) = watcher.watch(&source, RecursiveMode::Recursive) {
|
||||
error!(source = %source.display(), error = %e, "server watcher: failed to watch source");
|
||||
return;
|
||||
}
|
||||
|
||||
info!(source = %source.display(), "server watcher: scanning for changes");
|
||||
loop {
|
||||
match rx.recv_timeout(POLL_INTERVAL) {
|
||||
Ok(res) => match res {
|
||||
Ok(event) => match event.kind {
|
||||
EventKind::Create(_) | EventKind::Modify(_) | EventKind::Remove(_) => {
|
||||
let kind = match event.kind {
|
||||
EventKind::Create(_) => ChangeKind::Create,
|
||||
EventKind::Modify(_) => ChangeKind::Modify,
|
||||
EventKind::Remove(_) => ChangeKind::Remove,
|
||||
_ => continue,
|
||||
};
|
||||
match state.replace_all(&source) {
|
||||
Ok(true) => {
|
||||
let _ = events_tx.send(ChangeEvent { kind });
|
||||
}
|
||||
Ok(false) => {}
|
||||
Err(e) => {
|
||||
error!(error = %e, "server watcher: state refresh failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
},
|
||||
Err(e) => error!(error = %e, "server watcher: inotify error"),
|
||||
},
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => match state.replace_all(&source) {
|
||||
Ok(true) => {
|
||||
let _ = events_tx.send(ChangeEvent {
|
||||
kind: ChangeKind::Modify,
|
||||
});
|
||||
}
|
||||
Ok(false) => {}
|
||||
Err(e) => error!(error = %e, "server watcher: poll rescan failed"),
|
||||
},
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
CREATE TABLE items (
|
||||
inode BIGINT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
original_path TEXT NOT NULL,
|
||||
local_path TEXT NOT NULL,
|
||||
file_type TEXT NOT NULL CHECK (file_type IN ('directory', 'file')),
|
||||
hash BIGINT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE music_metadata (
|
||||
inode BIGINT PRIMARY KEY REFERENCES items(inode) ON DELETE CASCADE,
|
||||
track_title TEXT NOT NULL,
|
||||
album TEXT NOT NULL,
|
||||
track_number INTEGER NOT NULL,
|
||||
header BYTEA NOT NULL,
|
||||
real_audio_start BIGINT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE music_metadata_artists (
|
||||
inode BIGINT NOT NULL REFERENCES music_metadata(inode) ON DELETE CASCADE,
|
||||
artist TEXT NOT NULL,
|
||||
PRIMARY KEY (inode, artist)
|
||||
);
|
||||
|
||||
CREATE TABLE music_metadata_other_tags (
|
||||
inode BIGINT NOT NULL REFERENCES music_metadata(inode) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
tag TEXT NOT NULL,
|
||||
PRIMARY KEY (inode, position)
|
||||
);
|
||||
|
||||
CREATE TABLE music_metadata_pictures (
|
||||
inode BIGINT NOT NULL REFERENCES music_metadata(inode) ON DELETE CASCADE,
|
||||
position INTEGER NOT NULL,
|
||||
block_header BYTEA NOT NULL,
|
||||
data_offset BIGINT NOT NULL,
|
||||
data_length BIGINT NOT NULL,
|
||||
PRIMARY KEY (inode, position)
|
||||
);
|
||||
|
||||
-- Lazy byte cache for NetworkOrigin. Rows exist only for files the user has
|
||||
-- actually read; reconcile invalidates a row when its hash changes.
|
||||
CREATE TABLE cached_file_bytes (
|
||||
inode BIGINT PRIMARY KEY REFERENCES items(inode) ON DELETE CASCADE,
|
||||
data BYTEA NOT NULL,
|
||||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
+135
-11
@@ -1,5 +1,33 @@
|
||||
{
|
||||
"nodes": {
|
||||
"bun2nix": {
|
||||
"inputs": {
|
||||
"flake-parts": "flake-parts_2",
|
||||
"nixpkgs": [
|
||||
"pinix",
|
||||
"nixpkgs"
|
||||
],
|
||||
"systems": [
|
||||
"pinix",
|
||||
"systems"
|
||||
],
|
||||
"treefmt-nix": "treefmt-nix"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1778446047,
|
||||
"narHash": "sha256-oQvcadh2BCkrog+SGrG6YffKJrveYpjj3TdQJWaKhaM=",
|
||||
"owner": "nix-community",
|
||||
"repo": "bun2nix",
|
||||
"rev": "f2bc12af1a6369648aac41041ceeaa0b866599c6",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-community",
|
||||
"ref": "2.1.0",
|
||||
"repo": "bun2nix",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"cachix": {
|
||||
"inputs": {
|
||||
"devenv": [
|
||||
@@ -55,11 +83,11 @@
|
||||
"devenv": {
|
||||
"locked": {
|
||||
"dir": "src/modules",
|
||||
"lastModified": 1781800860,
|
||||
"narHash": "sha256-LrEo0eC5ckMvjpBRCuk5q5/vjItKlxnb4n/clHNRZlk=",
|
||||
"lastModified": 1782331842,
|
||||
"narHash": "sha256-7CJ2EqNVPMq0ly39aaP6dGgdO627MqUtM/+Dm+QwNdU=",
|
||||
"owner": "cachix",
|
||||
"repo": "devenv",
|
||||
"rev": "d59d872d80876d9eeb3e214d3b088bc4a14a9c4f",
|
||||
"rev": "885e1c9d62cfa12232802de77b36aaded1ca609b",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
@@ -141,6 +169,28 @@
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"flake-parts_2": {
|
||||
"inputs": {
|
||||
"nixpkgs-lib": [
|
||||
"pinix",
|
||||
"bun2nix",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1777988971,
|
||||
"narHash": "sha256-qIoWPDs+0/8JecyYgE3gpKQxW/4bLW/gp45vow9ioCQ=",
|
||||
"owner": "hercules-ci",
|
||||
"repo": "flake-parts",
|
||||
"rev": "0678d8986be1661af6bb555f3489f2fdfc31f6ff",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "hercules-ci",
|
||||
"repo": "flake-parts",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"git-hooks": {
|
||||
"inputs": {
|
||||
"flake-compat": [
|
||||
@@ -292,11 +342,11 @@
|
||||
"nixpkgs-src": {
|
||||
"flake": false,
|
||||
"locked": {
|
||||
"lastModified": 1781454065,
|
||||
"narHash": "sha256-d2xfDjnfRuf/xYGdu9VVRHiav/2w5hDL/5cw2TuVAXw=",
|
||||
"lastModified": 1781607440,
|
||||
"narHash": "sha256-rxO+uc/KFbSJp+pgyXRuAX6QlG9hJdnt0BXpEQRXY+U=",
|
||||
"owner": "NixOS",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "9eac87a12312b8f60dd52e1c6e1a265f6fc7f5fc",
|
||||
"rev": "3e41b24abd260e8f71dbe2f5737d24122f972158",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
@@ -311,11 +361,11 @@
|
||||
"nixpkgs-src": "nixpkgs-src"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1781620901,
|
||||
"narHash": "sha256-UF6scQlG+6lRkZBUpn/3KNavhOo5G8kDWhjVHcno8uc=",
|
||||
"lastModified": 1782132010,
|
||||
"narHash": "sha256-ZnAVHdVrotp80iIMm5CSR1fdxPlw7Uwmwxb+O/wsgZ8=",
|
||||
"owner": "cachix",
|
||||
"repo": "devenv-nixpkgs",
|
||||
"rev": "2df109b343d3c68efd752e32a444a1d9b9f89afa",
|
||||
"rev": "12866ae2dddbc0ab8b329915f8072bb9c75bde89",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
@@ -326,6 +376,22 @@
|
||||
}
|
||||
},
|
||||
"nixpkgs_3": {
|
||||
"locked": {
|
||||
"lastModified": 1773646010,
|
||||
"narHash": "sha256-iYrs97hS7p5u4lQzuNWzuALGIOdkPXvjz7bviiBjUu8=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "5b2c2d84341b2afb5647081c1386a80d7a8d8605",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nixos",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"nixpkgs_4": {
|
||||
"locked": {
|
||||
"lastModified": 1770107345,
|
||||
"narHash": "sha256-tbS0Ebx2PiA1FRW8mt8oejR0qMXmziJmPaU1d4kYY9g=",
|
||||
@@ -341,6 +407,26 @@
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"pinix": {
|
||||
"inputs": {
|
||||
"bun2nix": "bun2nix",
|
||||
"nixpkgs": "nixpkgs_3",
|
||||
"systems": "systems"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1782724139,
|
||||
"narHash": "sha256-Dn7GnkFBiQ5h4S0wthikh68QK4LfipDhE7ODIUAyVcE=",
|
||||
"owner": "lukasl-dev",
|
||||
"repo": "pi.nix",
|
||||
"rev": "8e64e43042900a764adbe22943fc2f798c71ef0b",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "lukasl-dev",
|
||||
"repo": "pi.nix",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"pre-commit-hooks": {
|
||||
"inputs": {
|
||||
"flake-compat": [
|
||||
@@ -373,12 +459,50 @@
|
||||
"devenv": "devenv",
|
||||
"git-hooks": "git-hooks_2",
|
||||
"nixpkgs": "nixpkgs_2",
|
||||
"treefmt-nix": "treefmt-nix"
|
||||
"pinix": "pinix",
|
||||
"treefmt-nix": "treefmt-nix_2"
|
||||
}
|
||||
},
|
||||
"systems": {
|
||||
"locked": {
|
||||
"lastModified": 1681028828,
|
||||
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nix-systems",
|
||||
"repo": "default",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"treefmt-nix": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs_3"
|
||||
"nixpkgs": [
|
||||
"pinix",
|
||||
"bun2nix",
|
||||
"nixpkgs"
|
||||
]
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1775636079,
|
||||
"narHash": "sha256-pc20NRoMdiar8oPQceQT47UUZMBTiMdUuWrYu2obUP0=",
|
||||
"owner": "numtide",
|
||||
"repo": "treefmt-nix",
|
||||
"rev": "790751ff7fd3801feeaf96d7dc416a8d581265ba",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "numtide",
|
||||
"repo": "treefmt-nix",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"treefmt-nix_2": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs_4"
|
||||
},
|
||||
"locked": {
|
||||
"lastModified": 1780220602,
|
||||
|
||||
+105
-1
@@ -6,6 +6,33 @@
|
||||
...
|
||||
}:
|
||||
|
||||
let
|
||||
# devenv's `languages.rust.import` only builds single root-package crates
|
||||
# (it hardcodes `cargoNix.rootCrate.build`). musicfs is a virtual workspace with
|
||||
# no root package, so we drive crate2nix directly and select the `musicfs-client`
|
||||
# member out of `workspaceMembers`. src = ./. gives crate2nix the whole workspace,
|
||||
# so the member's path-dependencies (musicfs-core, musicfs-proto) resolve.
|
||||
crate2nixTools = pkgs.callPackage "${inputs.crate2nix}/tools.nix" { };
|
||||
cargoNix =
|
||||
pkgs.callPackage
|
||||
(crate2nixTools.generatedCargoNix {
|
||||
name = "musicfs";
|
||||
src = ./.;
|
||||
})
|
||||
{
|
||||
# Build with the toolchain configured for this dev environment,
|
||||
# matching what languages.rust.import does internally.
|
||||
buildRustCrateForPkgs =
|
||||
_:
|
||||
pkgs.buildRustCrate.override {
|
||||
rustc = config.languages.rust.toolchainPackage;
|
||||
cargo = config.languages.rust.toolchainPackage;
|
||||
};
|
||||
};
|
||||
|
||||
# musicfs-client's binary is named `musicfs` (see [[bin]] in its Cargo.toml).
|
||||
musicfs = cargoNix.workspaceMembers."musicfs-client".build;
|
||||
in
|
||||
{
|
||||
languages.rust.enable = true;
|
||||
git-hooks.hooks = {
|
||||
@@ -27,9 +54,86 @@
|
||||
packages = with pkgs; [
|
||||
git
|
||||
just
|
||||
flac
|
||||
ffmpeg
|
||||
id3v2
|
||||
mpv
|
||||
protobuf
|
||||
buf
|
||||
grpcurl
|
||||
|
||||
opencode
|
||||
inputs.pinix.packages.${pkgs.system}.default
|
||||
];
|
||||
|
||||
processes.musicfs = {
|
||||
after = [ "devenv:processes:postgres" ];
|
||||
exec = lib.mkDefault ''
|
||||
${musicfs}/bin/musicfs \
|
||||
--source /home/fujin/Music \
|
||||
--mountpoint /tmp/rust-fuse \
|
||||
--database "postgresql://fujin@localhost/musicfs?host=$PGHOST"
|
||||
'';
|
||||
# Probes the standard gRPC Health Checking Protocol exposed on the
|
||||
# musicfs-client status server (default --listen 127.0.0.1:50052).
|
||||
# SERVING implies the FUSE mount succeeded and, for NetworkOrigin, the
|
||||
# upstream musicfs-server is reachable (health.rs toggles not_serving on
|
||||
# upstream loss). Inherited by profiles.* since they only override `exec`.
|
||||
ready.exec = "grpcurl -plaintext -max-time 2 127.0.0.1:50052 grpc.health.v1.Health/Check | grep -q SERVING";
|
||||
};
|
||||
|
||||
profiles.local.module = {
|
||||
processes.musicfs = {
|
||||
after = [ "devenv:processes:postgres" ];
|
||||
exec = ''
|
||||
${musicfs}/bin/musicfs \
|
||||
--source /home/fujin/Music \
|
||||
--mountpoint /tmp/rust-fuse \
|
||||
--database "postgresql://fujin@localhost/musicfs?host=$PGHOST"
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
profiles.remote.module = {
|
||||
processes.musicfs = {
|
||||
after = [ "devenv:processes:postgres" ];
|
||||
exec = ''
|
||||
${musicfs}/bin/musicfs \
|
||||
--source http://10.185.226.145:50051 \
|
||||
--mountpoint /tmp/rust-fuse \
|
||||
--database "postgresql://fujin@localhost/musicfs?host=$PGHOST"
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
profiles.e2e.module = {
|
||||
processes.musicfs = {
|
||||
after = [ "devenv:processes:postgres" ];
|
||||
exec = ''
|
||||
${musicfs}/bin/musicfs \
|
||||
--source http://127.0.0.1:50061 \
|
||||
--mountpoint ./target/e2e/mnt \
|
||||
--database "postgresql://fujin@localhost/musicfs_e2e?host=$PGHOST" \
|
||||
--log-dir ./logs/e2e
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
services.postgres = {
|
||||
enable = true;
|
||||
initialDatabases = [
|
||||
{
|
||||
name = "musicfs";
|
||||
schema = ./db/schema.sql;
|
||||
}
|
||||
{
|
||||
name = "musicfs_e2e";
|
||||
schema = ./db/schema.sql;
|
||||
}
|
||||
];
|
||||
};
|
||||
|
||||
outputs = {
|
||||
rust-app = config.languages.rust.import ./. { };
|
||||
musicfs = musicfs;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -13,3 +13,5 @@ inputs:
|
||||
url: github:cachix/devenv-nixpkgs/rolling
|
||||
treefmt-nix:
|
||||
url: github:numtide/treefmt-nix
|
||||
pinix:
|
||||
url: github:lukasl-dev/pi.nix
|
||||
|
||||
@@ -1,199 +0,0 @@
|
||||
# Search API Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
MusicFS provides two search interfaces:
|
||||
1. **FUSE Virtual Directory** - `/.search/query/` for file manager integration
|
||||
2. **gRPC API** - `Search` and `SearchStream` RPCs for programmatic access (planned)
|
||||
|
||||
---
|
||||
|
||||
## FUSE Search Interface
|
||||
|
||||
### Endpoint: `/.search/{query}/`
|
||||
|
||||
Browse search results as symlinks in a virtual directory.
|
||||
|
||||
### Happy Path
|
||||
|
||||
1. User navigates to `/.search/metallica/`
|
||||
2. FUSE returns directory listing of symlinks
|
||||
3. Each symlink points to absolute path: `/mnt/music/Metallica/Album/Track.flac`
|
||||
4. User can open symlink directly in media player
|
||||
|
||||
**Example:**
|
||||
```bash
|
||||
$ ls -la /mnt/musicfs/.search/metallica/
|
||||
001. Metallica - Enter Sandman.flac -> /mnt/musicfs/Metallica/Black Album/Enter Sandman.flac
|
||||
002. Metallica - Battery.flac -> /mnt/musicfs/Metallica/Master of Puppets/Battery.flac
|
||||
```
|
||||
|
||||
### Error Cases
|
||||
|
||||
| Scenario | Behavior | FUSE Error |
|
||||
|----------|----------|------------|
|
||||
| Empty query | Empty directory | (none) |
|
||||
| No results | Empty directory | (none) |
|
||||
| Query too long (>256 chars) | Truncated | (none) |
|
||||
| Invalid UTF-8 in query | EINVAL | `libc::EINVAL` |
|
||||
| Index corrupted | ENOENT | `libc::ENOENT` |
|
||||
| Index writer shutdown | EIO | `libc::EIO` |
|
||||
|
||||
### Cache Behavior
|
||||
|
||||
- Results cached for 5 minutes (TTL)
|
||||
- Maximum 1000 cached queries (LRU eviction)
|
||||
- Cache miss triggers tantivy query
|
||||
|
||||
---
|
||||
|
||||
## gRPC Search API
|
||||
|
||||
> **Note:** gRPC API is planned for implementation. See architecture docs for design.
|
||||
|
||||
### `Search(SearchRequest) -> SearchResponse`
|
||||
|
||||
Single request/response search.
|
||||
|
||||
#### Request Schema
|
||||
|
||||
```protobuf
|
||||
message SearchRequest {
|
||||
string query = 1; // Required: tantivy query string
|
||||
optional uint32 limit = 2; // Default: 100, max: 10000
|
||||
optional uint32 offset = 3; // Default: 0, for pagination
|
||||
optional string origin_id = 4; // Filter by origin (optional)
|
||||
}
|
||||
```
|
||||
|
||||
#### Response Schema
|
||||
|
||||
```protobuf
|
||||
message SearchResponse {
|
||||
repeated SearchResult results = 1;
|
||||
uint64 total_matches = 2; // Approximate total
|
||||
uint32 query_time_ms = 3; // Query execution time
|
||||
}
|
||||
|
||||
message SearchResult {
|
||||
int64 file_id = 1;
|
||||
string virtual_path = 2;
|
||||
optional string artist = 3;
|
||||
optional string album = 4;
|
||||
optional string title = 5;
|
||||
float score = 6; // Relevance score
|
||||
map<string, string> highlights = 7; // Matched fragments
|
||||
}
|
||||
```
|
||||
|
||||
### Error Cases
|
||||
|
||||
| Scenario | gRPC Status | Details |
|
||||
|----------|-------------|---------|
|
||||
| Empty query | `INVALID_ARGUMENT` | "Query cannot be empty" |
|
||||
| Malformed query syntax | `INVALID_ARGUMENT` | tantivy parse error message |
|
||||
| limit > 10000 | `INVALID_ARGUMENT` | "Limit exceeds maximum (10000)" |
|
||||
| Index unavailable | `UNAVAILABLE` | "Search index not ready" |
|
||||
| Index corrupted | `INTERNAL` | "Search index corrupted" |
|
||||
| Timeout (>5s) | `DEADLINE_EXCEEDED` | Client-specified deadline |
|
||||
|
||||
---
|
||||
|
||||
## Query Syntax
|
||||
|
||||
MusicFS uses tantivy query syntax with custom fuzzy support.
|
||||
|
||||
### Supported Operators
|
||||
|
||||
| Operator | Example | Description |
|
||||
|----------|---------|-------------|
|
||||
| Term | `metallica` | Match in any default field |
|
||||
| Field | `artist:metallica` | Match specific field |
|
||||
| Phrase | `"enter sandman"` | Exact phrase match |
|
||||
| Fuzzy | `metalica~1` | 1-character edit distance |
|
||||
| Boolean | `metallica AND 1991` | Combine conditions |
|
||||
| Range | `year:[1980 TO 1989]` | Numeric range |
|
||||
|
||||
### Searchable Fields
|
||||
|
||||
| Field | Type | Notes |
|
||||
|-------|------|-------|
|
||||
| `artist` | TEXT | Full-text searchable, default field |
|
||||
| `album` | TEXT | Full-text searchable, default field |
|
||||
| `album_artist` | TEXT | Full-text searchable, default field |
|
||||
| `title` | TEXT | Full-text searchable, default field |
|
||||
| `genre` | TEXT | Full-text searchable, default field |
|
||||
| `composer` | TEXT | Full-text searchable, default field |
|
||||
| `year` | u64 | Range queries only |
|
||||
|
||||
### Fuzzy Query Implementation
|
||||
|
||||
Fuzzy queries use the `term~N` syntax where N is the maximum edit distance (0-2).
|
||||
|
||||
When a fuzzy query is detected:
|
||||
1. Query is parsed to extract term and distance
|
||||
2. `FuzzyTermQuery` is created for each default field
|
||||
3. Results are combined with `BooleanQuery` (OR semantics)
|
||||
|
||||
Example: `metalica~1` matches "Metallica" (edit distance 1).
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
| Metric | Target | Notes |
|
||||
|--------|--------|-------|
|
||||
| Query latency (1M tracks) | <500ms | tantivy optimized |
|
||||
| Index throughput | >1000 files/sec | Batch commits recommended |
|
||||
| Memory per 1M tracks | <500MB | mmap-based index |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Index Schema
|
||||
|
||||
```rust
|
||||
pub struct SearchSchema {
|
||||
file_id: Field, // INDEXED | STORED - for deletion
|
||||
virtual_path: Field, // STORED - symlink target
|
||||
artist: Field, // TEXT | STORED
|
||||
album: Field, // TEXT | STORED
|
||||
album_artist: Field, // TEXT | STORED
|
||||
title: Field, // TEXT | STORED
|
||||
genre: Field, // TEXT | STORED
|
||||
composer: Field, // TEXT | STORED
|
||||
year: Field, // INDEXED | STORED
|
||||
duration_ms: Field, // STORED
|
||||
bitrate: Field, // STORED
|
||||
sample_rate: Field, // STORED
|
||||
}
|
||||
```
|
||||
|
||||
### Writer Pattern
|
||||
|
||||
Uses `Arc<RwLock<IndexWriter>>` per tantivy best practices:
|
||||
- `add_document()` and `delete_term()` require READ lock
|
||||
- `commit()` requires WRITE lock
|
||||
- Single writer, multiple concurrent indexers
|
||||
|
||||
### Event Integration
|
||||
|
||||
The `Indexer` subscribes to `EventBus` for:
|
||||
- `FileAdded` - Index new file via `MetadataLookup`
|
||||
- `FileRemoved` - Remove from index by file_id
|
||||
- `FileModified` - Update index entry
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
| Test | Type | Validates |
|
||||
|------|------|-----------|
|
||||
| `test_search_basic` | Unit | Basic search returns results |
|
||||
| `test_search_fuzzy` | Unit | Typo tolerance (FR-14.3) |
|
||||
| `test_search_genre` | Unit | Field-specific search |
|
||||
| `test_index_persistence` | Unit | Index survives restart |
|
||||
| `test_remove_file` | Unit | Deletion works correctly |
|
||||
| `test_index_batch` | Unit | Batch indexing via Indexer |
|
||||
| `test_search_ops_*` | Unit | FUSE SearchOps integration |
|
||||
@@ -1,315 +0,0 @@
|
||||
# Smart Features API Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
MusicFS Week 9 introduces three intelligent features:
|
||||
1. **Smart Collections** - Dynamic playlists based on queries, time ranges, and listening patterns
|
||||
2. **Artwork Extraction & Caching** - Extract and serve album art in multiple sizes
|
||||
3. **Predictive Prefetching** - Learn listening patterns to preload likely-next tracks
|
||||
|
||||
---
|
||||
|
||||
## Smart Collections
|
||||
|
||||
### CollectionStore
|
||||
|
||||
Manages persistent smart collections using SQLite.
|
||||
|
||||
```rust
|
||||
pub struct CollectionStore {
|
||||
db: rusqlite::Connection,
|
||||
}
|
||||
|
||||
pub struct Collection {
|
||||
pub id: i64,
|
||||
pub name: String,
|
||||
pub query: CollectionQuery,
|
||||
pub created_at: SystemTime,
|
||||
pub updated_at: SystemTime,
|
||||
}
|
||||
```
|
||||
|
||||
### CollectionQuery Types
|
||||
|
||||
| Query Type | Description | Example |
|
||||
|------------|-------------|---------|
|
||||
| `Match(String)` | tantivy search query | `"artist:Metallica"` |
|
||||
| `DateRange { start, end }` | Files added within range | Last 30 days |
|
||||
| `RecentlyAdded(days)` | Files added in last N days | `RecentlyAdded(7)` |
|
||||
| `RecentlyPlayed(days)` | Files played in last N days | `RecentlyPlayed(30)` |
|
||||
| `MostPlayed(limit)` | Top N most played tracks | `MostPlayed(100)` |
|
||||
| `Genre(String)` | All tracks matching genre | `"Progressive Rock"` |
|
||||
| `Compound(Vec)` | AND combination of queries | Multiple conditions |
|
||||
|
||||
### API
|
||||
|
||||
```rust
|
||||
impl CollectionStore {
|
||||
fn create(&self, name: &str, query: CollectionQuery) -> Result<i64, CollectionError>;
|
||||
fn get(&self, id: i64) -> Result<Option<Collection>, CollectionError>;
|
||||
fn list(&self) -> Result<Vec<Collection>, CollectionError>;
|
||||
fn update(&self, id: i64, name: &str, query: CollectionQuery) -> Result<(), CollectionError>;
|
||||
fn delete(&self, id: i64) -> Result<(), CollectionError>;
|
||||
fn evaluate(&self, id: i64, index: &SearchIndex, patterns: &PatternStore) -> Result<Vec<FileId>, CollectionError>;
|
||||
}
|
||||
```
|
||||
|
||||
### FUSE Integration (Planned)
|
||||
|
||||
Collections will appear as virtual directories under `/.collections/`:
|
||||
|
||||
```bash
|
||||
$ ls /mnt/musicfs/.collections/
|
||||
Recent Additions/
|
||||
Most Played/
|
||||
80s Metal/
|
||||
|
||||
$ ls /mnt/musicfs/.collections/Most\ Played/
|
||||
001. Track1.flac -> /mnt/musicfs/Artist/Album/Track1.flac
|
||||
002. Track2.flac -> /mnt/musicfs/Artist/Album/Track2.flac
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Artwork Extraction & Caching
|
||||
|
||||
### ArtworkExtractor
|
||||
|
||||
Extracts embedded artwork from audio files.
|
||||
|
||||
```rust
|
||||
pub struct Artwork {
|
||||
pub data: Vec<u8>,
|
||||
pub mime_type: String,
|
||||
pub art_type: ArtType,
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
}
|
||||
|
||||
pub enum ArtType {
|
||||
Front,
|
||||
Back,
|
||||
Other,
|
||||
}
|
||||
|
||||
pub enum ArtSize {
|
||||
Thumbnail, // 150x150 max
|
||||
Medium, // 300x300 max
|
||||
Full, // Original size
|
||||
}
|
||||
```
|
||||
|
||||
### API
|
||||
|
||||
```rust
|
||||
impl ArtworkExtractor {
|
||||
fn extract(&self, path: &Path) -> Result<Vec<Artwork>, ArtworkError>;
|
||||
fn extract_first(&self, path: &Path) -> Result<Option<Artwork>, ArtworkError>;
|
||||
fn resize(data: &[u8], size: ArtSize) -> Result<Vec<u8>, ArtworkError>;
|
||||
}
|
||||
```
|
||||
|
||||
### ArtworkCache
|
||||
|
||||
Caches artwork in CAS (Content-Addressable Storage).
|
||||
|
||||
```rust
|
||||
impl ArtworkCache {
|
||||
async fn store(&self, file_id: i64, artwork: &Artwork) -> Result<ChunkHash, ArtworkError>;
|
||||
async fn get(&self, file_id: i64, art_type: &str, size: ArtSize) -> Result<Option<Vec<u8>>, ArtworkError>;
|
||||
async fn has(&self, file_id: i64, art_type: &str) -> Result<bool, ArtworkError>;
|
||||
}
|
||||
```
|
||||
|
||||
### Size Specifications
|
||||
|
||||
| Size | Max Dimension | Use Case |
|
||||
|------|---------------|----------|
|
||||
| Thumbnail | 150px | List views, grids |
|
||||
| Medium | 300px | Detail panels |
|
||||
| Full | Original | High-res display |
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
1. Original artwork stored in CAS with content hash
|
||||
2. SQLite maps `(file_id, art_type)` → `chunk_hash`
|
||||
3. Resizing performed on-demand, not cached (saves storage)
|
||||
4. Max input size: 10MB (reject larger images)
|
||||
|
||||
---
|
||||
|
||||
## Predictive Prefetching
|
||||
|
||||
### Access Patterns (PatternStore)
|
||||
|
||||
Tracks file access history to predict next tracks.
|
||||
|
||||
```rust
|
||||
pub struct AccessPattern {
|
||||
pub file_id: FileId,
|
||||
pub timestamp: SystemTime,
|
||||
pub context: AccessContext,
|
||||
pub hour_of_day: u8,
|
||||
}
|
||||
|
||||
pub struct AccessContext {
|
||||
pub album_id: Option<i64>,
|
||||
pub track_number: Option<u32>,
|
||||
pub artist: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern Learning
|
||||
|
||||
| Pattern Type | Description | Use Case |
|
||||
|--------------|-------------|----------|
|
||||
| Sequential | A → B → C transitions | Album playback |
|
||||
| Time-based | Hour-of-day preferences | Morning playlist |
|
||||
| Frequency | Most played tracks | Popular content |
|
||||
|
||||
### API
|
||||
|
||||
```rust
|
||||
impl PatternStore {
|
||||
fn record(&self, file_id: FileId, context: AccessContext) -> Result<(), PatternError>;
|
||||
fn predict_next(&self, current: FileId, limit: usize) -> Vec<FileId>;
|
||||
fn predict_for_time(&self, hour: u8, limit: usize) -> Vec<FileId>;
|
||||
fn recently_played(&self, days: u32) -> Result<Vec<FileId>, PatternError>;
|
||||
fn most_played(&self, limit: u32) -> Result<Vec<FileId>, PatternError>;
|
||||
}
|
||||
```
|
||||
|
||||
### PrefetchEngine
|
||||
|
||||
Background engine that listens for file access events and prefetches predicted content.
|
||||
|
||||
```rust
|
||||
pub struct PrefetchConfig {
|
||||
pub lookahead: usize, // How many tracks to prefetch (default: 3)
|
||||
pub max_concurrent: usize, // Concurrent prefetch limit (default: 2)
|
||||
pub cooldown: Duration, // Delay between prefetch bursts (default: 100ms)
|
||||
pub enabled: bool, // Master switch
|
||||
}
|
||||
```
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ EventBus │────▶│ PrefetchEngine │────▶│ ContentFetcher │
|
||||
│ (FileAccessed) │ │ (predictions) │ │ (CAS storage) │
|
||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ PatternStore │
|
||||
│ (SQLite DB) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### FUSE Interface
|
||||
|
||||
Virtual directory `/.prefetch/` exposes prefetch status and hints:
|
||||
|
||||
```bash
|
||||
$ cat /mnt/musicfs/.prefetch/status
|
||||
MusicFS Prefetch Status
|
||||
=======================
|
||||
running: true
|
||||
in_flight: 2
|
||||
most_played: [42, 57, 103, 89, 12]
|
||||
|
||||
$ ls /mnt/musicfs/.prefetch/
|
||||
status
|
||||
hint_0042
|
||||
hint_0057
|
||||
hint_0103
|
||||
|
||||
$ cat /mnt/musicfs/.prefetch/hint_0042
|
||||
57
|
||||
103
|
||||
89
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Targets
|
||||
|
||||
| Feature | Metric | Target |
|
||||
|---------|--------|--------|
|
||||
| Collection evaluation | Latency | <100ms for 100k files |
|
||||
| Artwork extraction | Throughput | >10 files/sec |
|
||||
| Artwork resize | Latency | <50ms per image |
|
||||
| Pattern prediction | Latency | <10ms |
|
||||
| Prefetch hit rate | Accuracy | >70% for sequential play |
|
||||
|
||||
---
|
||||
|
||||
## Error Handling
|
||||
|
||||
### CollectionError
|
||||
|
||||
| Error | Description |
|
||||
|-------|-------------|
|
||||
| `Database(rusqlite::Error)` | SQLite operation failed |
|
||||
| `NotFound` | Collection ID doesn't exist |
|
||||
| `InvalidQuery` | Query failed to serialize |
|
||||
| `Search(SearchError)` | tantivy query failed |
|
||||
| `Pattern(PatternError)` | Pattern lookup failed |
|
||||
|
||||
### ArtworkError
|
||||
|
||||
| Error | Description |
|
||||
|-------|-------------|
|
||||
| `Database(rusqlite::Error)` | Cache DB operation failed |
|
||||
| `Cas(CasError)` | CAS storage operation failed |
|
||||
| `InvalidHash` | Stored hash is malformed |
|
||||
| `NotFound` | Artwork not in cache |
|
||||
| `ImageTooLarge(usize)` | Input exceeds 10MB limit |
|
||||
| `InvalidImage` | Cannot decode image data |
|
||||
| `ResizeFailed` | Image resize operation failed |
|
||||
|
||||
### PatternError
|
||||
|
||||
| Error | Description |
|
||||
|-------|-------------|
|
||||
| `Database(rusqlite::Error)` | SQLite operation failed |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Default Settings
|
||||
|
||||
```toml
|
||||
[prefetch]
|
||||
enabled = true
|
||||
lookahead = 3
|
||||
max_concurrent = 2
|
||||
cooldown_ms = 100
|
||||
|
||||
[artwork]
|
||||
max_input_size_mb = 10
|
||||
thumbnail_size = 150
|
||||
medium_size = 300
|
||||
|
||||
[patterns]
|
||||
max_history_days = 30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
| Test | Type | Validates |
|
||||
|------|------|-----------|
|
||||
| `test_collection_crud` | Unit | Create, read, update, delete |
|
||||
| `test_collection_evaluate_match` | Unit | Match query evaluation |
|
||||
| `test_collection_persistence` | Unit | Collections survive restart |
|
||||
| `test_artwork_extract_flac` | Unit | FLAC artwork extraction |
|
||||
| `test_artwork_cache_store_get` | Unit | Cache round-trip |
|
||||
| `test_artwork_resize` | Unit | Resize produces valid output |
|
||||
| `test_pattern_prediction` | Unit | Sequential pattern learning |
|
||||
| `test_pattern_persistence` | Unit | Patterns survive restart |
|
||||
| `test_prefetch_config_defaults` | Unit | Default config values |
|
||||
| `test_prefetch_ops_*` | Unit | FUSE PrefetchOps integration |
|
||||
Vendored
-96
@@ -1,96 +0,0 @@
|
||||
# [Project Name]: Design Doc
|
||||
|
||||
**Authors:** [Author Name(s)]
|
||||
**Status:** [Draft / In-Review / Approved / Obsolete]
|
||||
**Last Updated:** YYYY-MM-DD
|
||||
**Reviewers:** [List of reviewers, usually @usernames]
|
||||
**Approvers:** [List of final decision makers]
|
||||
**Document Link:** [Link to this file or rendered version]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
A high-level summary (1–3 paragraphs) of what the project is, what problem it solves, and the proposed solution. This should be readable by a non-expert.
|
||||
|
||||
## 2. Background
|
||||
Context for why this project exists.
|
||||
- What is the current state?
|
||||
- What are the pain points?
|
||||
- Are there existing systems that this will replace or interact with?
|
||||
- Include links to relevant PRDs (Product Requirement Documents) or previous design docs.
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
Clarity on scope is critical to prevent scope creep.
|
||||
|
||||
### 3.1. Goals
|
||||
* **Primary Goal:** The most important outcome.
|
||||
* Metric-driven goals (e.g., "Reduce latency by 20%").
|
||||
* Functional requirements (e.g., "Allow users to edit comments").
|
||||
|
||||
### 3.2. Non-Goals
|
||||
* Features that might seem related but are explicitly out of scope.
|
||||
* Future improvements that are deferred.
|
||||
|
||||
## 4. Proposed Design
|
||||
The "meat" of the document. Start with the high-level architecture and zoom in.
|
||||
|
||||
### 4.1. High-Level Architecture
|
||||
Provide a high-level diagram or description of how the system fits together.
|
||||
> *Tip: Use Mermaid.js or link to an embedded image.*
|
||||
|
||||
### 4.2. Detailed Design
|
||||
Go into specific components, APIs, and data models.
|
||||
* **API Definitions:** Describe new endpoints, Protobuf definitions, or CLI commands.
|
||||
* **Data Schema:** Database tables, key-value structures, or file formats.
|
||||
* **Workflows:** Step-by-step logic for complex operations (e.g., auth flow).
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
Google design docs place heavy emphasis on these "standard" reviews.
|
||||
|
||||
### 5.1. Security & Privacy
|
||||
- How is data encrypted?
|
||||
- What are the access control lists (ACLs)?
|
||||
- Does this handle PII (Personally Identifiable Information)?
|
||||
|
||||
### 5.2. Observability (Monitoring & Logging)
|
||||
- What metrics will be exported (e.g., RPC error rates, latency)?
|
||||
- What logging is required for debugging?
|
||||
- What are the "Golden Signals" for the dashboard?
|
||||
|
||||
### 5.3. Scalability & Performance
|
||||
- What are the expected QPS (Queries Per Second)?
|
||||
- How does the system scale (Horizontal vs. Vertical)?
|
||||
- What are the resource requirements (CPU, RAM, Storage)?
|
||||
|
||||
### 5.4. Testing Plan
|
||||
- Unit tests, integration tests, and end-to-end tests.
|
||||
- Strategy for load testing or "chaos" testing.
|
||||
|
||||
## 6. Alternatives Considered
|
||||
*A BlueDoc is not just about the chosen path, but why others were rejected.*
|
||||
* **Alternative A:** Briefly describe it and why it was rejected (e.g., "Too complex," "High latency").
|
||||
* **Alternative B:** Why "Doing Nothing" is not an option.
|
||||
|
||||
## 7. Implementation Plan
|
||||
- **Phase 1:** Minimum Viable Product (MVP).
|
||||
- **Phase 2:** Feature parity or migrations.
|
||||
- **Rollout/Rollback:** How will the feature be toggled? (e.g., feature flags).
|
||||
|
||||
## 8. Glossary / References
|
||||
- Links to external libraries.
|
||||
- Definitions for project-specific acronyms.
|
||||
|
||||
---
|
||||
|
||||
### Markdown Style Tips (Google Conventions):
|
||||
1. **Line Length:** Google's internal style guide suggests a soft limit of **80 characters** per line for source Markdown to make it easier to review in code-diff tools.
|
||||
2. **Headings:** Use `#` for title, `##` for sections, and `###` for subsections.
|
||||
3. **TOC:** If your environment supports it, use `[TOC]` at the top to generate a Table of Contents.
|
||||
4. **Diagrams:** Use **Mermaid** blocks if using GitHub/GitLab, otherwise link to a stable SVG/PNG.
|
||||
```mermaid
|
||||
graph TD;
|
||||
A-->B;
|
||||
A-->C;
|
||||
B-->D;
|
||||
C-->D;
|
||||
```
|
||||
Vendored
-56
@@ -1,56 +0,0 @@
|
||||
# [Project Name]: Design One-Pager (GreenDoc)
|
||||
|
||||
**Author(s):** [Name]
|
||||
**Status:** [Draft / Approved / Shipped]
|
||||
**Last Updated:** YYYY-MM-DD
|
||||
**Estimated Effort:** [e.g., 2 weeks, 1 sprint]
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
A 2–3 sentence overview of the change. What are you doing and why?
|
||||
|
||||
## 2. Problem Statement
|
||||
Describe the specific pain point or "broken" state this project addresses.
|
||||
* *Example: Currently, users cannot filter their search history by date, leading to high latency in manual lookup.*
|
||||
|
||||
## 3. Proposed Solution
|
||||
Explain the high-level logic of the fix or feature.
|
||||
* What is the specific code change or configuration update?
|
||||
* How does it interact with existing systems?
|
||||
* *Note: Use a single simple diagram if the logic is non-trivial.*
|
||||
|
||||
## 4. Risks & Trade-offs
|
||||
Even small changes have risks. Address them upfront.
|
||||
* **Performance:** Will this increase memory usage?
|
||||
* **Complexity:** Does this add a new dependency?
|
||||
* **Backwards Compatibility:** Will this break existing clients?
|
||||
* **Alternatives:** Why did you choose this over a "quicker" or "better" fix?
|
||||
|
||||
## 5. Success Criteria (Metrics)
|
||||
How will you know this worked?
|
||||
* [ ] Primary metric (e.g., "Feature usage > 5%")
|
||||
* [ ] Guardrail metric (e.g., "Latency does not increase by > 10ms")
|
||||
|
||||
## 6. Implementation & Rollout
|
||||
A brief bulleted list of the steps to ship.
|
||||
1. Feature flag implementation.
|
||||
2. Canary to 1% of users.
|
||||
3. Full rollout.
|
||||
|
||||
---
|
||||
|
||||
### Comparison: BlueDoc vs. GreenDoc
|
||||
|
||||
| Feature | BlueDoc (Standard) | GreenDoc (One-Pager) |
|
||||
| :--- | :--- | :--- |
|
||||
| **Scope** | Major systems, new services. | Small features, optimizations, bug fixes. |
|
||||
| **Length** | 5–20+ pages. | 1–2 pages. |
|
||||
| **Review** | Cross-functional committees (SRE, Security). | Peer-level or Team Lead review. |
|
||||
| **Focus** | Long-term scalability and architecture. | Immediate impact and implementation. |
|
||||
|
||||
### Markdown Tips for GreenDocs:
|
||||
* **Be Brutally Concise:** If a section requires more than three paragraphs, consider upgrading to a **BlueDoc**.
|
||||
* **Checklists:** Use `[ ]` to show remaining work or requirements.
|
||||
* **Inline Links:** Link directly to the relevant code files or bug tracker (Jira/Buganizer) to keep the doc self-contained.
|
||||
|
||||
@@ -1,118 +0,0 @@
|
||||
# beetfs - Reverse Engineered Documentation
|
||||
|
||||
> **Status**: Archived project (2010-2013), Python 2, fuse-python API
|
||||
> **Fork**: git@github.com:LichHunter/beetfs.git
|
||||
> **Original**: https://github.com/jbaiter/beetfs
|
||||
|
||||
## Overview
|
||||
|
||||
beetfs is a FUSE filesystem that presents audio files with **metadata from a database** while **passing through audio data unchanged** from original files. This enables transparent metadata modification without touching the underlying files.
|
||||
|
||||
### The Core Concept
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ APPLICATION (VLC, Jellyfin, etc.) │
|
||||
│ │
|
||||
│ read("/mount/Artist/Album/track.flac") │
|
||||
└─────────────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ beetfs (FUSE Layer) │
|
||||
│ ┌────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ FileHandler │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ if offset < header_boundary: │ │ │
|
||||
│ │ │ return MODIFIED_HEADER (from beets database) │ │ │
|
||||
│ │ │ else: │ │ │
|
||||
│ │ │ return ORIGINAL_AUDIO (from real file on disk) │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
│ │
|
||||
┌───────────┘ └───────────┐
|
||||
▼ ▼
|
||||
┌───────────────────┐ ┌───────────────────┐
|
||||
│ Beets Database │ │ Original File │
|
||||
│ (SQLite - tags) │ │ (untouched) │
|
||||
│ │ │ │
|
||||
│ title: "Fixed" │ │ [FLAC header] │
|
||||
│ artist: "Corr" │ │ [Audio frames] │
|
||||
│ album: "Right" │ │ │
|
||||
└───────────────────┘ └───────────────────┘
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| **Metadata Overlay** | Returns tags from database, not from file |
|
||||
| **Audio Passthrough** | Original audio data served unchanged |
|
||||
| **Write Interception** | Tag edits saved to database, not to file |
|
||||
| **Virtual Organization** | Presents files in template-based directory structure |
|
||||
| **Format Support** | FLAC (full), MP3 (partial - read-only) |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
beetfs/
|
||||
├── beetsplug/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ └── beetFs.py # ALL code (~1144 lines)
|
||||
├── README.rst # Original readme
|
||||
└── COPYING # GPLv3 license
|
||||
```
|
||||
|
||||
## Quick Architecture Summary
|
||||
|
||||
| Component | Lines | Purpose |
|
||||
|-----------|-------|---------|
|
||||
| `beetFs` (plugin) | 188-191 | Beets plugin hook |
|
||||
| `mount()` | 119-183 | CLI entry point, builds virtual tree |
|
||||
| `FSNode` | 390-436 | Virtual directory tree node |
|
||||
| `FileHandler` | 439-565 | **CORE**: Metadata interpolation |
|
||||
| `InterpolatedFLAC` | 274-388 | FLAC header generation |
|
||||
| `InterpolatedID3` | 200-271 | ID3 tag generation (incomplete) |
|
||||
| `beetFileSystem` | 622-1144 | FUSE operations implementation |
|
||||
| `Stat` | 568-619 | File stat structure |
|
||||
|
||||
## Documentation Index
|
||||
|
||||
1. **[Architecture Overview](./architecture.md)** - System design and component interaction
|
||||
2. **[Components Deep Dive](./components.md)** - Detailed component analysis
|
||||
3. **[Data Flow](./data-flow.md)** - Read/write operation flows
|
||||
4. **[Performance Analysis](./analysis.md)** - Latency, memory footprint, I/O patterns
|
||||
5. **[Drawbacks & Limitations](./drawbacks.md)** - Known issues and missing features
|
||||
6. **[Modernization Guide](./modernization.md)** - Notes for updating to Python 3
|
||||
|
||||
## Critical Issues Summary
|
||||
|
||||
| Issue | Severity | Impact |
|
||||
|-------|----------|--------|
|
||||
| Full file loaded into RAM | 🔴 Critical | OOM on large libraries |
|
||||
| MP3 support disabled | 🔴 Critical | Only FLAC works |
|
||||
| Python 2 only | 🔴 Critical | EOL, security risk |
|
||||
| Single-threaded | 🟡 Major | Poor concurrency |
|
||||
| 4 of 17 metadata fields | 🟡 Major | Limited functionality |
|
||||
|
||||
See [drawbacks.md](./drawbacks.md) for complete list (27 identified issues).
|
||||
|
||||
## Dependencies (Original)
|
||||
|
||||
```
|
||||
beets >= 1.0
|
||||
fuse-python (Python 2 FUSE bindings)
|
||||
mutagen (audio metadata library)
|
||||
```
|
||||
|
||||
## Usage (Original)
|
||||
|
||||
```bash
|
||||
# As beets plugin
|
||||
beet mount /path/to/mountpoint
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
GPLv3 - See COPYING file
|
||||
@@ -1,263 +0,0 @@
|
||||
# beetfs Performance Analysis
|
||||
|
||||
## Executive Summary
|
||||
|
||||
beetfs has significant performance limitations due to its 2010-era design assumptions. The primary issues are **full file loading into RAM** and **blocking I/O on file open**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Latency Analysis
|
||||
|
||||
### Operation Latencies
|
||||
|
||||
| Operation | Time Complexity | Typical Latency | Notes |
|
||||
|-----------|-----------------|-----------------|-------|
|
||||
| **File Open** | O(file_size) | 50ms - 1s+ | Reads entire file into memory |
|
||||
| **File Read** | O(1) | <1ms | Pure memory slice |
|
||||
| **File Write** | O(file_size) | 100ms - 2s+ | Reconstructs + DB write |
|
||||
| **Directory List** | O(n) | <10ms | In-memory tree traversal |
|
||||
| **getattr** | O(depth) | <1ms | Tree navigation + stat |
|
||||
|
||||
### File Open Breakdown
|
||||
|
||||
The file open operation is the critical bottleneck:
|
||||
|
||||
```
|
||||
Time breakdown for opening 50MB FLAC file:
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ 1. open() syscall │ ~1ms │
|
||||
│ 2. file_object.read() - load entire file │ ~100-200ms │
|
||||
│ 3. InterpolatedFLAC() - parse FLAC │ ~20-50ms │
|
||||
│ 4. Inject DB metadata │ ~1ms │
|
||||
│ 5. get_header() - generate new header │ ~10-20ms │
|
||||
│ 6. Seek to audio offset │ ~1ms │
|
||||
│ 7. Read audio into music_data │ ~100-200ms │
|
||||
├────────────────────────────────────────────────────────────┤
|
||||
│ TOTAL │ ~230-470ms │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Code Evidence** (lines 461-483):
|
||||
```python
|
||||
# Step 2-5: Load and parse entire file
|
||||
self.inf = InterpolatedFLAC(self.file_object.read()) # FULL FILE READ
|
||||
self.inf["title"] = self.item.title
|
||||
# ...
|
||||
self.header = self.inf.get_header(self.real_path)
|
||||
|
||||
# Step 6-7: Cache all audio data
|
||||
self.file_object.seek(self.music_offset)
|
||||
self.music_data = self.file_object.read() # ANOTHER FULL READ
|
||||
```
|
||||
|
||||
### Read Operation (Post-Open)
|
||||
|
||||
After file is opened, reads are fast:
|
||||
|
||||
```python
|
||||
def read(self, size, offset):
|
||||
if offset < self.bound:
|
||||
return self.header[offset:offset+size] # Memory slice: O(1)
|
||||
else:
|
||||
return self.music_data[offset - len(self.header):...] # Memory slice: O(1)
|
||||
```
|
||||
|
||||
### Write Operation
|
||||
|
||||
Writes to header area trigger expensive reconstruction:
|
||||
|
||||
```
|
||||
Time breakdown for tag write:
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ 1. Reconstruct filedata in memory │ ~10-50ms │
|
||||
│ 2. Parse as InterpolatedFLAC │ ~20-50ms │
|
||||
│ 3. Extract tag values │ ~1ms │
|
||||
│ 4. lib.store() + lib.save() (SQLite) │ ~10-50ms │
|
||||
│ 5. Regenerate header │ ~10-20ms │
|
||||
├────────────────────────────────────────────────────────────┤
|
||||
│ TOTAL │ ~50-170ms │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Memory Footprint
|
||||
|
||||
### Per-File Memory Usage
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ FileHandler Memory Layout │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.music_data (bytes) │ │
|
||||
│ │ Size: file_size - original_header_size │ │
|
||||
│ │ Typical: 95-99% of file size │ │
|
||||
│ │ Example: 48.5 MB for 50 MB file │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.header (bytes) │ │
|
||||
│ │ Size: Generated FLAC header with DB metadata │ │
|
||||
│ │ Typical: 4 KB - 64 KB (depends on metadata + padding) │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.inf (InterpolatedFLAC) │ │
|
||||
│ │ Size: Parsed metadata blocks + internal state │ │
|
||||
│ │ Typical: 10 KB - 100 KB │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Other attributes │ │
|
||||
│ │ path, real_path, item reference, format, etc. │ │
|
||||
│ │ Typical: ~1 KB │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ TOTAL per file: ~1.0x - 1.1x original file size │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Memory Scaling
|
||||
|
||||
| Scenario | Files Open | Avg File Size | RAM Usage |
|
||||
|----------|------------|---------------|-----------|
|
||||
| Single track playback | 1 | 30 MB | ~32 MB |
|
||||
| Album playback (gapless) | 2-3 | 30 MB | ~65-100 MB |
|
||||
| Album fully opened | 10 | 30 MB | ~320 MB |
|
||||
| Jellyfin library scan | 50-100 | 30 MB | **1.6 - 3.2 GB** |
|
||||
| Full library scan | 1000 | 30 MB | **32 GB** (OOM) |
|
||||
|
||||
### Global Memory
|
||||
|
||||
```python
|
||||
# Directory tree structure
|
||||
directory_structure = FSNode({}, {})
|
||||
# Memory: O(number_of_items)
|
||||
# Typical: 1-10 MB for libraries with 10,000-100,000 tracks
|
||||
|
||||
# Open file handles
|
||||
self.files = {} # Dict[str, FileHandler]
|
||||
# Memory: Sum of all FileHandler instances
|
||||
# Unbounded - grows with concurrent opens
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. I/O Patterns
|
||||
|
||||
### Current (Inefficient)
|
||||
|
||||
```
|
||||
File Open:
|
||||
Disk → [Read ALL] → RAM (music_data)
|
||||
→ RAM (inf object)
|
||||
→ RAM (header)
|
||||
|
||||
File Read:
|
||||
RAM (header or music_data) → Application
|
||||
|
||||
Total I/O: 1x-2x file size on open, 0 on read
|
||||
```
|
||||
|
||||
### Optimal (Not Implemented)
|
||||
|
||||
```
|
||||
File Open:
|
||||
Disk → [Read header only] → RAM (small)
|
||||
|
||||
File Read:
|
||||
If header region:
|
||||
RAM (header) → Application
|
||||
If audio region:
|
||||
Disk → [Seek + Read chunk] → Application
|
||||
|
||||
Total I/O: ~64KB on open, on-demand reads
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Concurrency
|
||||
|
||||
### Current Model
|
||||
|
||||
```python
|
||||
server.multithreaded = 0 # Single-threaded
|
||||
```
|
||||
|
||||
**Implications:**
|
||||
- All FUSE operations serialized
|
||||
- One slow file open blocks everything
|
||||
- No benefit from multi-core CPUs
|
||||
|
||||
### Impact on Use Cases
|
||||
|
||||
| Use Case | Impact |
|
||||
|----------|--------|
|
||||
| Single player (VLC) | Acceptable - one file at a time |
|
||||
| Media server scan | Severe - sequential processing |
|
||||
| Multiple clients | Severe - requests queue up |
|
||||
| Concurrent reads | Moderate - reads are fast once open |
|
||||
|
||||
---
|
||||
|
||||
## 5. Benchmarks (Theoretical)
|
||||
|
||||
Based on code analysis, not actual measurements:
|
||||
|
||||
### File Open Time vs Size
|
||||
|
||||
```
|
||||
File Size Open Time (HDD) Open Time (SSD)
|
||||
────────────────────────────────────────────────
|
||||
10 MB 50-100 ms 20-50 ms
|
||||
30 MB 150-300 ms 50-100 ms
|
||||
50 MB 250-500 ms 100-200 ms
|
||||
100 MB 500-1000 ms 200-400 ms
|
||||
200 MB 1000-2000 ms 400-800 ms
|
||||
```
|
||||
|
||||
### Memory vs Concurrent Opens
|
||||
|
||||
```
|
||||
Open Files RAM Usage (30MB avg)
|
||||
─────────────────────────────────────
|
||||
1 ~32 MB
|
||||
5 ~160 MB
|
||||
10 ~320 MB
|
||||
25 ~800 MB
|
||||
50 ~1.6 GB
|
||||
100 ~3.2 GB
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Comparison with Alternatives
|
||||
|
||||
| Metric | beetfs | Direct File | NFS | FUSE passthrough |
|
||||
|--------|--------|-------------|-----|------------------|
|
||||
| Open latency | 200-500ms | <10ms | 10-50ms | <10ms |
|
||||
| Read latency | <1ms | <1ms | 1-10ms | <1ms |
|
||||
| Memory/file | ~1x size | ~0 | ~0 | ~0 |
|
||||
| Metadata source | Database | File | File | File |
|
||||
| Modify original | No | Yes | Yes | Yes |
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendations
|
||||
|
||||
### For Current Usage
|
||||
|
||||
1. **Limit concurrent opens** - Don't scan full library
|
||||
2. **Use SSDs** - Reduces open latency by 2-3x
|
||||
3. **Increase RAM** - Expect 1x file size per open
|
||||
4. **Avoid large files** - 24-bit/192kHz FLACs are problematic
|
||||
|
||||
### For Modernization
|
||||
|
||||
1. **Implement lazy loading** - Read audio on demand
|
||||
2. **Add file handle caching** - Keep headers, release audio
|
||||
3. **Enable multi-threading** - Parallelize opens
|
||||
4. **Add memory limits** - Evict old FileHandlers
|
||||
@@ -1,403 +0,0 @@
|
||||
# beetfs Benchmark Plan
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Benchmark suite to measure beetfs FUSE filesystem performance across mount time, metadata operations, file I/O, and memory usage. Focus on realistic music library workloads.
|
||||
|
||||
## Critical Performance Findings (Pre-Benchmark)
|
||||
|
||||
### Architecture Bottlenecks Identified
|
||||
|
||||
| Bottleneck | Location | Impact |
|
||||
|------------|----------|--------|
|
||||
| **Full file load into RAM** | `FileHandler.__init__` line 481 | 50-100MB per open FLAC |
|
||||
| **Mount-time bulk load** | `mount()` line 143 | O(N) for N library items |
|
||||
| **GIL serialization** | Python 2.7 | Single-core limit for metadata ops |
|
||||
| **Per-file DB lookup** | `getattr()`, `access()` | SQLite query per stat call |
|
||||
|
||||
### Expected Performance Characteristics
|
||||
|
||||
| Operation | Expected Performance | Bottleneck |
|
||||
|-----------|---------------------|------------|
|
||||
| Mount (10K items) | 5-30 seconds | `lib.items()` + FSNode construction |
|
||||
| readdir | Fast (in-memory dict) | None |
|
||||
| getattr (file) | Slow (~1ms) | DB lookup + real file stat |
|
||||
| open (first) | Very slow | Full file read into RAM |
|
||||
| read | Fast | Memory-to-memory copy |
|
||||
| Memory (10 open files) | 500MB-1GB | FileHandler caches entire files |
|
||||
|
||||
---
|
||||
|
||||
## Benchmark Tools
|
||||
|
||||
### Primary Tools
|
||||
|
||||
| Tool | Purpose | Install |
|
||||
|------|---------|---------|
|
||||
| **fio** | I/O throughput, IOPS, latency | `nix-shell -p fio` |
|
||||
| **mdtest** | Metadata operations (stat, readdir) | `nix-shell -p ior` |
|
||||
| **hyperfine** | Mount time, command timing | `nix-shell -p hyperfine` |
|
||||
| **time** | Basic timing | builtin |
|
||||
| **/usr/bin/time -v** | Memory usage (maxrss) | builtin |
|
||||
|
||||
### Measurement Scripts
|
||||
|
||||
All benchmarks use synthetic FLAC files (5-10MB) to avoid I/O variance from real storage.
|
||||
|
||||
---
|
||||
|
||||
## Benchmark Categories
|
||||
|
||||
### 1. Mount Time Scaling
|
||||
|
||||
**Goal**: Measure how mount time scales with library size.
|
||||
|
||||
**Method**:
|
||||
```bash
|
||||
# Create libraries with N items: 100, 1K, 10K, 50K, 100K
|
||||
hyperfine --warmup 1 --runs 5 \
|
||||
'beet mount /mnt/beetfs && sleep 1 && fusermount -u /mnt/beetfs'
|
||||
```
|
||||
|
||||
**Metrics**:
|
||||
- Time to mount (seconds)
|
||||
- Memory usage at mount completion (RSS)
|
||||
|
||||
**Expected scaling**: O(N) - linear with library size
|
||||
|
||||
**Test matrix**:
|
||||
| Library Size | Expected Mount Time | Expected Memory |
|
||||
|--------------|--------------------:|----------------:|
|
||||
| 100 items | <1s | ~50MB |
|
||||
| 1,000 items | 1-3s | ~60MB |
|
||||
| 10,000 items | 5-15s | ~100MB |
|
||||
| 50,000 items | 30-60s | ~300MB |
|
||||
| 100,000 items | 60-120s | ~500MB |
|
||||
|
||||
---
|
||||
|
||||
### 2. Metadata Operations (stat/readdir)
|
||||
|
||||
**Goal**: Measure getattr and readdir performance - critical for music players that scan libraries.
|
||||
|
||||
#### 2a. Single stat latency
|
||||
|
||||
```bash
|
||||
# Measure single stat call latency
|
||||
hyperfine --warmup 10 --runs 100 \
|
||||
'stat /mnt/beetfs/Artist/Album/01-Track.flac'
|
||||
```
|
||||
|
||||
**Target**: <5ms average, <20ms p99
|
||||
|
||||
#### 2b. Bulk stat (library scan simulation)
|
||||
|
||||
```bash
|
||||
# Stat all files in library
|
||||
hyperfine --warmup 1 --runs 5 \
|
||||
'find /mnt/beetfs -type f -exec stat {} + > /dev/null'
|
||||
```
|
||||
|
||||
**Metrics**:
|
||||
- Total time for N files
|
||||
- stat operations per second
|
||||
- p50, p95, p99 latency
|
||||
|
||||
**Target**: >500 stat/s (Python FUSE baseline)
|
||||
|
||||
#### 2c. Directory listing
|
||||
|
||||
```bash
|
||||
# List directory with N entries
|
||||
hyperfine --warmup 3 --runs 10 \
|
||||
'ls /mnt/beetfs/Artist/Album/'
|
||||
```
|
||||
|
||||
**Test matrix**:
|
||||
| Directory entries | Target time |
|
||||
|------------------:|------------:|
|
||||
| 10 | <50ms |
|
||||
| 100 | <100ms |
|
||||
| 1,000 | <500ms |
|
||||
|
||||
---
|
||||
|
||||
### 3. File Open Performance
|
||||
|
||||
**Goal**: Measure file open latency - the critical bottleneck due to full file load.
|
||||
|
||||
#### 3a. First open (cold)
|
||||
|
||||
```bash
|
||||
# Clear any caches, then open file
|
||||
echo 3 > /proc/sys/vm/drop_caches
|
||||
hyperfine --warmup 0 --runs 10 \
|
||||
'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null'
|
||||
```
|
||||
|
||||
**Test matrix**:
|
||||
| File size | Expected open time |
|
||||
|----------:|-------------------:|
|
||||
| 5MB | 50-200ms |
|
||||
| 20MB | 200-500ms |
|
||||
| 50MB | 500ms-1s |
|
||||
| 100MB | 1-2s |
|
||||
|
||||
#### 3b. Cached open (warm)
|
||||
|
||||
```bash
|
||||
# File already opened once
|
||||
hyperfine --warmup 5 --runs 50 \
|
||||
'head -c 1 /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null'
|
||||
```
|
||||
|
||||
**Target**: <10ms (should hit FileHandler cache)
|
||||
|
||||
---
|
||||
|
||||
### 4. Read Throughput
|
||||
|
||||
**Goal**: Measure sequential and random read performance.
|
||||
|
||||
#### 4a. Sequential read
|
||||
|
||||
```bash
|
||||
fio --name=seq_read \
|
||||
--filename=/mnt/beetfs/Artist/Album/01-Track.flac \
|
||||
--rw=read --bs=1M --direct=0 \
|
||||
--ioengine=sync --numjobs=1 \
|
||||
--runtime=30 --time_based
|
||||
```
|
||||
|
||||
**Metrics**: MB/s throughput
|
||||
|
||||
**Target**: >100 MB/s (memory-backed after first read)
|
||||
|
||||
#### 4b. Random read (simulates seeking in audio player)
|
||||
|
||||
```bash
|
||||
fio --name=rand_read \
|
||||
--filename=/mnt/beetfs/Artist/Album/01-Track.flac \
|
||||
--rw=randread --bs=64k --direct=0 \
|
||||
--ioengine=sync --numjobs=1 \
|
||||
--runtime=30 --time_based
|
||||
```
|
||||
|
||||
**Metrics**: IOPS, latency histogram
|
||||
|
||||
---
|
||||
|
||||
### 5. Memory Usage
|
||||
|
||||
**Goal**: Measure memory consumption under load.
|
||||
|
||||
#### 5a. Idle memory (mounted, no activity)
|
||||
|
||||
```bash
|
||||
# Mount and measure RSS
|
||||
beet mount /mnt/beetfs &
|
||||
sleep 5
|
||||
ps -o rss= -p $(pgrep -f beetfs)
|
||||
```
|
||||
|
||||
#### 5b. Memory per open file
|
||||
|
||||
```bash
|
||||
# Open N files, measure memory growth
|
||||
for i in 1 5 10 20; do
|
||||
# Open $i files simultaneously
|
||||
cat /mnt/beetfs/Artist/Album/0{1..$i}*.flac > /dev/null &
|
||||
ps -o rss= -p $(pgrep -f beetfs)
|
||||
done
|
||||
```
|
||||
|
||||
**Expected**: ~file_size × open_files (FileHandler caches entire file)
|
||||
|
||||
#### 5c. Memory leak detection
|
||||
|
||||
```bash
|
||||
# Repeatedly open/close files, check for memory growth
|
||||
for i in {1..100}; do
|
||||
cat /mnt/beetfs/Artist/Album/01-Track.flac > /dev/null
|
||||
done
|
||||
# Compare RSS before and after
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Concurrent Access
|
||||
|
||||
**Goal**: Measure performance under parallel access (multiple processes).
|
||||
|
||||
```bash
|
||||
# Parallel stat operations
|
||||
hyperfine --warmup 1 --runs 5 \
|
||||
'seq 1 100 | xargs -P 4 -I {} stat /mnt/beetfs/Artist/Album/0{}-Track.flac'
|
||||
```
|
||||
|
||||
**Metrics**:
|
||||
- Throughput scaling with parallelism (1, 2, 4, 8 workers)
|
||||
- Latency degradation
|
||||
|
||||
**Expected**: Limited scaling due to Python GIL
|
||||
|
||||
---
|
||||
|
||||
### 7. Realistic Workloads
|
||||
|
||||
#### 7a. Music player library scan
|
||||
|
||||
Simulates: Rhythmbox/Clementine scanning library at startup
|
||||
|
||||
```bash
|
||||
# Recursive stat + readdir
|
||||
time find /mnt/beetfs -type f -name "*.flac" -exec stat {} + | wc -l
|
||||
```
|
||||
|
||||
#### 7b. Album playback
|
||||
|
||||
Simulates: Playing 12-track album sequentially
|
||||
|
||||
```bash
|
||||
# Open each file, read 1MB (simulate buffering), close
|
||||
for f in /mnt/beetfs/Artist/Album/*.flac; do
|
||||
dd if="$f" of=/dev/null bs=1M count=1 2>/dev/null
|
||||
done
|
||||
```
|
||||
|
||||
#### 7c. Metadata edit
|
||||
|
||||
Simulates: Editing tags in Picard/Kid3
|
||||
|
||||
```bash
|
||||
# Open file, write to header region, close
|
||||
# (Requires write support to be functional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Baseline Comparisons
|
||||
|
||||
### Reference Filesystems
|
||||
|
||||
| Filesystem | Purpose |
|
||||
|------------|---------|
|
||||
| **ext4 (local)** | Best-case baseline |
|
||||
| **fuse-passthrough** | FUSE overhead baseline |
|
||||
| **sshfs** | Network FUSE comparison |
|
||||
|
||||
### Comparison Method
|
||||
|
||||
Run identical benchmarks on:
|
||||
1. Real music files on ext4
|
||||
2. Same files via FUSE passthrough
|
||||
3. Same files via beetfs
|
||||
|
||||
Calculate overhead: `(beetfs_time - ext4_time) / ext4_time × 100%`
|
||||
|
||||
---
|
||||
|
||||
## Test Environment
|
||||
|
||||
### Hardware Requirements
|
||||
|
||||
- CPU: 4+ cores (to test GIL impact)
|
||||
- RAM: 8+ GB (for large library tests)
|
||||
- Storage: SSD recommended (reduces I/O variance)
|
||||
|
||||
### Software Requirements
|
||||
|
||||
```nix
|
||||
# Add to flake.nix devShell
|
||||
buildInputs = [
|
||||
fio
|
||||
hyperfine
|
||||
# ior # includes mdtest
|
||||
];
|
||||
```
|
||||
|
||||
### Cache Control
|
||||
|
||||
```bash
|
||||
# Clear all caches before cold benchmarks
|
||||
sync
|
||||
echo 3 > /proc/sys/vm/drop_caches
|
||||
|
||||
# Disable kernel FUSE caching for accurate measurements
|
||||
mount -o entry_timeout=0,attr_timeout=0,negative_timeout=0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Minimum Viable Performance
|
||||
|
||||
| Metric | Minimum | Target | Excellent |
|
||||
|--------|--------:|-------:|----------:|
|
||||
| Mount time (10K items) | <60s | <15s | <5s |
|
||||
| stat latency (avg) | <20ms | <5ms | <1ms |
|
||||
| stat throughput | >100/s | >500/s | >2000/s |
|
||||
| File open (50MB, cold) | <5s | <1s | <200ms |
|
||||
| Read throughput | >50 MB/s | >200 MB/s | >500 MB/s |
|
||||
| Memory (idle, 10K items) | <500MB | <100MB | <50MB |
|
||||
| Memory per open file | <2× file size | <1.5× | <1.1× |
|
||||
|
||||
### Regression Detection
|
||||
|
||||
Any benchmark result >20% worse than baseline triggers investigation.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Test Data Generation
|
||||
|
||||
Use existing test infrastructure from `tests/conftest.py`:
|
||||
- `create_synthetic_flac()` - generates valid FLAC files
|
||||
- `BeetFSTestCase` - creates isolated beets library
|
||||
|
||||
### Benchmark Script Structure
|
||||
|
||||
```
|
||||
beetfs/
|
||||
├── benchmarks/
|
||||
│ ├── run_all.sh # Master script
|
||||
│ ├── bench_mount.sh # Mount time tests
|
||||
│ ├── bench_metadata.sh # stat/readdir tests
|
||||
│ ├── bench_io.sh # Read/write throughput
|
||||
│ ├── bench_memory.sh # Memory profiling
|
||||
│ └── results/ # Output directory
|
||||
│ ├── mount_scaling.csv
|
||||
│ ├── stat_latency.csv
|
||||
│ └── ...
|
||||
```
|
||||
|
||||
### Output Format
|
||||
|
||||
```csv
|
||||
# Example: mount_scaling.csv
|
||||
library_size,mount_time_ms,memory_rss_kb,timestamp
|
||||
100,450,52000,2024-01-15T10:30:00
|
||||
1000,2100,61000,2024-01-15T10:31:00
|
||||
10000,12500,98000,2024-01-15T10:33:00
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Python 2.7 GIL**: Cannot achieve true parallelism - expect flat scaling beyond 1 core
|
||||
2. **FileHandler memory**: Each open file = full file in RAM - will OOM with many large files
|
||||
3. **No lazy loading**: All library items loaded at mount - slow for large libraries
|
||||
4. **SQLite single-writer**: Concurrent writes will serialize
|
||||
|
||||
## Optimization Opportunities (Post-Benchmark)
|
||||
|
||||
Based on benchmark results, consider:
|
||||
|
||||
1. **Lazy FSNode construction** - Build tree on first access, not mount
|
||||
2. **Memory-mapped file access** - mmap instead of full read
|
||||
3. **LRU cache for FileHandler** - Evict old files instead of holding all
|
||||
4. **Metadata caching** - Cache getattr results, invalidate on DB change
|
||||
5. **Batch DB queries** - Prefetch metadata for directory listings
|
||||
@@ -1,101 +0,0 @@
|
||||
# beetfs Benchmark Results
|
||||
|
||||
**Date**: 2026-05-12
|
||||
**Status**: ❌ ALL BENCHMARKS BLOCKED BY BUGS
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Benchmarks cannot complete due to critical bugs in beetfs. The implementation is non-functional for any library with content.
|
||||
|
||||
## Results
|
||||
|
||||
| Benchmark | Status | Mean | Error |
|
||||
|-----------|--------|------|-------|
|
||||
| mount_time | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| readdir | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| stat_latency | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| enoent_lookup | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| file_open | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| read_throughput | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| memory_usage | ❌ FAIL | N/A | Directory tree building bug |
|
||||
|
||||
## Blocking Bugs
|
||||
|
||||
### Bug #1: Nested Methods (Lines 758-1144)
|
||||
|
||||
All FUSE operations (`readdir`, `open`, `read`, `write`, etc.) are indented inside the `access()` method, making them local functions instead of class methods.
|
||||
|
||||
**Impact**: Even if mount succeeds, all file operations return `ENOSYS (Function not implemented)`.
|
||||
|
||||
**Fix Required**: Dedent lines 758-1144 by 8 spaces.
|
||||
|
||||
### Bug #2: Directory Tree Building (Lines 403-414)
|
||||
|
||||
`FSNode.adddir()` calls `getnode()` which assumes parent directories already exist. When building the tree for a new library, parent directories haven't been created yet.
|
||||
|
||||
**Error**:
|
||||
```
|
||||
KeyError: u'Bench Artist'
|
||||
File "beetFs.py", line 403, in getnode
|
||||
return self.getnode(elements, root=root.dirs[topdir])
|
||||
```
|
||||
|
||||
**Impact**: Mount crashes when library contains any tracks.
|
||||
|
||||
**Fix Required**: `adddir()` must create parent directories recursively before adding child.
|
||||
|
||||
### Bug #3: Empty Library Only
|
||||
|
||||
The only working configuration is mounting with an empty beets library:
|
||||
- `test_mount_empty_library`: ✅ PASS
|
||||
- Any library with tracks: ❌ CRASH
|
||||
|
||||
## Test Environment
|
||||
|
||||
- **Python**: 2.7.15
|
||||
- **OS**: Linux (NixOS)
|
||||
- **Test data**: 10 synthetic FLAC files (5 MB each)
|
||||
- **Beets**: 1.4.9
|
||||
|
||||
## Benchmark Configuration
|
||||
|
||||
```python
|
||||
num_tracks = 10
|
||||
track_size_mb = 5
|
||||
mount_runs = 3
|
||||
stat_runs = 20
|
||||
readdir_runs = 10
|
||||
```
|
||||
|
||||
## Raw Results
|
||||
|
||||
See `benchmarks/results/benchmark_results.json` for full JSON output.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Fix Bug #2** (directory tree building) - allows mount with content
|
||||
2. **Fix Bug #1** (nested methods) - allows FUSE operations to work
|
||||
3. **Re-run benchmarks** - get actual performance numbers
|
||||
|
||||
## Conclusion
|
||||
|
||||
**beetfs is currently non-functional** for real-world use. Both bugs must be fixed before performance can be measured. The test infrastructure and benchmark suite are ready; only the implementation needs repair.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: E2E Test Results (For Reference)
|
||||
|
||||
From the e2e test suite (74 tests):
|
||||
|
||||
| Category | Passed | Failed | Errors |
|
||||
|----------|--------|--------|--------|
|
||||
| Smoke tests | 4 | 3 | 0 |
|
||||
| Nested bug detection | 3 (confirmed bug) | 10 | 0 |
|
||||
| Readdir | 0 | 10 | 0 |
|
||||
| Stat | 0 | 8 | 0 |
|
||||
| Read | 0 | 11 | 0 |
|
||||
| Write | 0 | 7 | 0 |
|
||||
| Error handling | 0 | 7 | 3 |
|
||||
| **Total** | **12** | **56** | **3** |
|
||||
|
||||
The 12 passing tests are infrastructure tests and tests that verify the bugs exist.
|
||||
@@ -1,550 +0,0 @@
|
||||
# beetfs Components Deep Dive
|
||||
|
||||
## Component Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFs.py │
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐│
|
||||
│ │ PLUGIN LAYER ││
|
||||
│ │ beetFs (BeetsPlugin) beetFs_command (Subcommand) ││
|
||||
│ │ mount() template_mapping() ││
|
||||
│ └─────────────────────────────────────────────────────────────────────┘│
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐│
|
||||
│ │ VIRTUAL FILESYSTEM ││
|
||||
│ │ FSNode beetFileSystem (fuse.Fuse) ││
|
||||
│ │ Stat ││
|
||||
│ └─────────────────────────────────────────────────────────────────────┘│
|
||||
│ ┌─────────────────────────────────────────────────────────────────────┐│
|
||||
│ │ METADATA INTERPOLATION ││
|
||||
│ │ FileHandler InterpolatedFLAC ││
|
||||
│ │ InterpolatedID3 ││
|
||||
│ └─────────────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Plugin Layer
|
||||
|
||||
### 1.1 beetFs (BeetsPlugin)
|
||||
|
||||
**Location**: Lines 188-191
|
||||
|
||||
```python
|
||||
class beetFs(BeetsPlugin):
|
||||
""" The beets plugin hook."""
|
||||
def commands(self):
|
||||
return [beetFs_command]
|
||||
```
|
||||
|
||||
**Purpose**: Registers beetfs as a beets plugin, exposing the `mount` subcommand.
|
||||
|
||||
### 1.2 beetFs_command
|
||||
|
||||
**Location**: Lines 47, 185
|
||||
|
||||
```python
|
||||
beetFs_command = Subcommand('mount', help='Mount a beets filesystem')
|
||||
beetFs_command.func = mount
|
||||
```
|
||||
|
||||
**Purpose**: CLI subcommand definition for `beet mount`.
|
||||
|
||||
### 1.3 mount() Function
|
||||
|
||||
**Location**: Lines 119-183
|
||||
|
||||
```python
|
||||
def mount(lib, config, opts, args):
|
||||
# 1. Validate arguments
|
||||
if not args:
|
||||
raise beets.ui.UserError('no mountpoint specified')
|
||||
|
||||
# 2. Parse path template
|
||||
global structure_split
|
||||
structure_split = PATH_FORMAT.split("/")
|
||||
global structure_depth
|
||||
structure_depth = len(structure_split)
|
||||
|
||||
# 3. Store library reference
|
||||
global library
|
||||
library = lib
|
||||
|
||||
# 4. Build virtual directory tree
|
||||
global directory_structure
|
||||
directory_structure = FSNode({}, {})
|
||||
|
||||
# 5. Iterate all library items
|
||||
for item in lib.items():
|
||||
mapping = template_mapping(lib, item)
|
||||
# ... build tree ...
|
||||
directory_structure.addfile(sub_elements, filename, item.id)
|
||||
|
||||
# 6. Create and run FUSE server
|
||||
server = beetFileSystem(...)
|
||||
server.main()
|
||||
```
|
||||
|
||||
**Key Variables Set**:
|
||||
| Variable | Type | Purpose |
|
||||
|----------|------|---------|
|
||||
| `structure_split` | `List[str]` | Path template components |
|
||||
| `structure_depth` | `int` | Number of path levels |
|
||||
| `library` | `Library` | Beets library reference |
|
||||
| `directory_structure` | `FSNode` | Root of virtual tree |
|
||||
|
||||
### 1.4 template_mapping() Function
|
||||
|
||||
**Location**: Lines 82-116
|
||||
|
||||
```python
|
||||
def template_mapping(lib, item):
|
||||
"""Builds a template substitution map from beets item."""
|
||||
mapping = {}
|
||||
for key in METADATA_KEYS:
|
||||
value = getattr(item, key)
|
||||
# Sanitize value for filesystem paths
|
||||
if isinstance(value, basestring):
|
||||
value = re.sub(r'[\\/:]|^\.', '_', value)
|
||||
elif key in ('track', 'tracktotal', 'disc', 'disctotal'):
|
||||
value = '%02i' % value # Zero-pad numbers
|
||||
mapping[key] = value
|
||||
|
||||
# Add format info
|
||||
format_ = os.path.splitext(item.path)[1][1:]
|
||||
mapping['format'] = format_
|
||||
mapping['format_upper'] = format_.upper()
|
||||
|
||||
# Default values for missing fields
|
||||
if mapping['artist'] == '':
|
||||
mapping['artist'] = 'Unknown Artist'
|
||||
# ... etc
|
||||
|
||||
return mapping
|
||||
```
|
||||
|
||||
**Template Variables Available**:
|
||||
| Variable | Source | Example |
|
||||
|----------|--------|---------|
|
||||
| `$artist` | `item.artist` | "Pink Floyd" |
|
||||
| `$album` | `item.album` | "The Wall" |
|
||||
| `$title` | `item.title` | "Comfortably Numb" |
|
||||
| `$year` | `item.year` | "1979" |
|
||||
| `$track` | `item.track` | "06" |
|
||||
| `$format` | file extension | "flac" |
|
||||
| `$format_upper` | file extension | "FLAC" |
|
||||
|
||||
---
|
||||
|
||||
## 2. Virtual Filesystem Layer
|
||||
|
||||
### 2.1 FSNode Class
|
||||
|
||||
**Location**: Lines 390-436
|
||||
|
||||
```python
|
||||
class FSNode(object):
|
||||
"""A directory node in the virtual filesystem tree."""
|
||||
|
||||
def __init__(self, dirs, files):
|
||||
self.dirs = dirs # Dict[str, FSNode] - subdirectories
|
||||
self.files = files # Dict[str, int] - filename → beets item ID
|
||||
```
|
||||
|
||||
**Methods**:
|
||||
|
||||
| Method | Purpose | Signature |
|
||||
|--------|---------|-----------|
|
||||
| `getnode()` | Navigate to nested node | `getnode(elements, root=None) → FSNode` |
|
||||
| `adddir()` | Add a directory | `adddir(elements, directory, root=None)` |
|
||||
| `addfile()` | Add a file entry | `addfile(elements, filename, id, root=None)` |
|
||||
| `listdir()` | List contents | `listdir(elements, directories, root=None) → List[str]` |
|
||||
|
||||
**Example Tree Navigation**:
|
||||
```python
|
||||
# Path: /Artist/Album/track.flac
|
||||
# structure_split = ["$artist", "$album ($year) [$format_upper]", "$track - $artist - $title.$format"]
|
||||
|
||||
elements = ["Artist", "Album (2020) [FLAC]"]
|
||||
node = directory_structure.getnode(elements)
|
||||
# node.files = {"01 - Artist - Track.flac": 42, ...}
|
||||
|
||||
item_id = node.files["01 - Artist - Track.flac"]
|
||||
# item_id = 42
|
||||
```
|
||||
|
||||
### 2.2 Stat Class
|
||||
|
||||
**Location**: Lines 568-619
|
||||
|
||||
```python
|
||||
class Stat(fuse.Stat):
|
||||
DIRSIZE = 4096
|
||||
|
||||
def __init__(self, st_mode, st_size, st_nlink=1, st_uid=None, st_gid=None,
|
||||
dt_atime=None, dt_mtime=None, dt_ctime=None):
|
||||
self.st_mode = st_mode
|
||||
self.st_ino = 0
|
||||
self.st_dev = 0
|
||||
self.st_nlink = st_nlink
|
||||
self.st_uid = st_uid or os.getuid()
|
||||
self.st_gid = st_gid or os.getgid()
|
||||
self.st_size = st_size
|
||||
# ... timestamps ...
|
||||
```
|
||||
|
||||
**Purpose**: Represents file/directory metadata for FUSE stat operations.
|
||||
|
||||
### 2.3 beetFileSystem Class
|
||||
|
||||
**Location**: Lines 622-1144
|
||||
|
||||
```python
|
||||
class beetFileSystem(fuse.Fuse):
|
||||
"""Main FUSE filesystem implementation."""
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
logging.basicConfig(filename="LOG", level=logging.INFO)
|
||||
super(beetFileSystem, self).__init__(*args, **kwargs)
|
||||
|
||||
def fsinit(self):
|
||||
"""Called after filesystem is mounted."""
|
||||
self.lib = library
|
||||
self.files = {} # Dict[path, FileHandler]
|
||||
```
|
||||
|
||||
**FUSE Operations Implemented**:
|
||||
|
||||
| Operation | Lines | Purpose |
|
||||
|-----------|-------|---------|
|
||||
| `fsinit()` | 630-636 | Post-mount initialization |
|
||||
| `fsdestroy()` | 638-639 | Pre-unmount cleanup |
|
||||
| `statfs()` | 641-646 | Filesystem statistics |
|
||||
| `getattr()` | 648-707 | Get file/dir attributes |
|
||||
| `access()` | 723-756 | Check permissions |
|
||||
| `readdir()` | 931-975 | List directory contents |
|
||||
| `open()` | 988-1021 | Open file |
|
||||
| `read()` | 1077-1106 | Read file data |
|
||||
| `write()` | 1108-1135 | Write file data |
|
||||
| `release()` | 1049-1059 | Close file |
|
||||
|
||||
**Not Implemented (return EOPNOTSUPP)**:
|
||||
- `mknod()`, `mkdir()`, `unlink()`, `rmdir()`
|
||||
- `symlink()`, `link()`, `rename()`
|
||||
- `chmod()`, `chown()`, `truncate()`
|
||||
|
||||
---
|
||||
|
||||
## 3. Metadata Interpolation Layer
|
||||
|
||||
### 3.1 FileHandler Class
|
||||
|
||||
**Location**: Lines 439-565
|
||||
|
||||
This is the **core component** that implements metadata overlay.
|
||||
|
||||
```python
|
||||
class FileHandler(object):
|
||||
def __init__(self, path, lib):
|
||||
self.path = path # Virtual path
|
||||
self.lib = lib # Beets library
|
||||
|
||||
# Resolve virtual path to real file
|
||||
pathsplit = path[1:].split('/')
|
||||
self.item = self.lib.get_item(id=directory_structure
|
||||
.getnode(pathsplit[0:structure_depth-1])
|
||||
.files[pathsplit[structure_depth-1]])
|
||||
self.real_path = self.item.path
|
||||
|
||||
# Open real file
|
||||
self.file_object = open(self.real_path, 'r+')
|
||||
self.instance_count = 1
|
||||
|
||||
# Determine format
|
||||
self.format = os.path.splitext(path)[1][1:].lower()
|
||||
|
||||
if self.format == "flac":
|
||||
# Load file into interpolated FLAC object
|
||||
self.inf = InterpolatedFLAC(self.file_object.read())
|
||||
|
||||
# INJECT DATABASE METADATA
|
||||
self.inf["title"] = self.item.title
|
||||
self.inf["album"] = self.item.album
|
||||
self.inf["artist"] = self.item.artist
|
||||
self.inf["genre"] = self.item.genre
|
||||
|
||||
# Generate new header with DB metadata
|
||||
self.header = self.inf.get_header(self.real_path)
|
||||
self.bound = len(self.header)
|
||||
self.music_offset = self.inf.offset()
|
||||
|
||||
elif self.format == "mp3":
|
||||
self.bound = 0 # MP3 interpolation disabled
|
||||
self.music_offset = 0
|
||||
|
||||
# Cache audio data
|
||||
self.file_object.seek(self.music_offset)
|
||||
self.music_data = self.file_object.read()
|
||||
self.file_object.close()
|
||||
```
|
||||
|
||||
**Key Attributes**:
|
||||
|
||||
| Attribute | Type | Purpose |
|
||||
|-----------|------|---------|
|
||||
| `path` | `str` | Virtual path (e.g., `/Artist/Album/track.flac`) |
|
||||
| `real_path` | `str` | Actual file path on disk |
|
||||
| `item` | `Item` | Beets library item (has DB metadata) |
|
||||
| `format` | `str` | File format ("flac", "mp3") |
|
||||
| `inf` | `InterpolatedFLAC` | Mutagen object with injected metadata |
|
||||
| `header` | `bytes` | Generated header with DB tags |
|
||||
| `bound` | `int` | Byte offset where header ends |
|
||||
| `music_offset` | `int` | Byte offset in original file where audio starts |
|
||||
| `music_data` | `bytes` | Cached audio data |
|
||||
| `instance_count` | `int` | Reference count for file handles |
|
||||
|
||||
### 3.2 FileHandler.read() Method
|
||||
|
||||
**Location**: Lines 497-517
|
||||
|
||||
```python
|
||||
def read(self, size, offset):
|
||||
# Case 1: Reading within header boundary
|
||||
if offset < self.bound:
|
||||
if offset + size < len(self.header):
|
||||
# Entire read is within header
|
||||
return self.header[offset:offset+size]
|
||||
else:
|
||||
# Read spans header and audio
|
||||
ret = self.header[offset:len(self.header)]
|
||||
ret = ret + self.music_data[0:size - (len(self.header) - offset)]
|
||||
return ret
|
||||
|
||||
# Case 2: Reading audio data only
|
||||
return self.music_data[offset - len(self.header):offset - len(self.header) + size]
|
||||
```
|
||||
|
||||
**Read Logic Diagram**:
|
||||
|
||||
```
|
||||
Virtual File Layout:
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ 0 bound EOF │
|
||||
│ ├─────────┼────────────────────────────────────────────────┤ │
|
||||
│ │ HEADER │ AUDIO DATA │ │
|
||||
│ │ (from │ (from self.music_data) │ │
|
||||
│ │ self. │ │ │
|
||||
│ │ header) │ │ │
|
||||
│ └─────────┴────────────────────────────────────────────────┘ │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
|
||||
Read scenarios:
|
||||
1. offset=0, size=100, bound=500 → Return header[0:100]
|
||||
2. offset=400, size=200, bound=500 → Return header[400:500] + music[0:100]
|
||||
3. offset=600, size=100, bound=500 → Return music[100:200]
|
||||
```
|
||||
|
||||
### 3.3 FileHandler.write() Method
|
||||
|
||||
**Location**: Lines 519-565
|
||||
|
||||
```python
|
||||
def write(self, offset, buf):
|
||||
# Only handle writes to header area
|
||||
if offset < self.bound:
|
||||
# Reconstruct full file in memory
|
||||
filedata = self.header + self.music_data
|
||||
|
||||
# Patch in new data
|
||||
filedata = filedata[0:offset] + buf + filedata[offset + len(buf):]
|
||||
|
||||
if self.format == "flac":
|
||||
# Parse the patched data
|
||||
self.inf = InterpolatedFLAC(filedata)
|
||||
|
||||
# EXTRACT new tag values and save to DB
|
||||
self.item.title = str(self.inf["title"][0]).encode('utf-8')
|
||||
self.item.album = str(self.inf["album"][0]).encode('utf-8')
|
||||
self.item.artist = str(self.inf["artist"][0]).encode('utf-8')
|
||||
self.item.genre = str(self.inf["genre"][0]).encode('utf-8')
|
||||
|
||||
# Persist to beets database
|
||||
self.lib.store(self.item)
|
||||
self.lib.save()
|
||||
|
||||
# Regenerate header with updated values
|
||||
self.inf["title"] = self.item.title
|
||||
self.inf["album"] = self.item.album
|
||||
self.inf["artist"] = self.item.artist
|
||||
self.inf["genre"] = self.item.genre
|
||||
|
||||
self.header = self.inf.get_header(self.real_path)
|
||||
self.bound = len(self.header)
|
||||
|
||||
return len(buf)
|
||||
```
|
||||
|
||||
**Write Flow**:
|
||||
```
|
||||
1. App writes new tag data to header region
|
||||
│
|
||||
▼
|
||||
2. Patch header + music_data with new bytes
|
||||
│
|
||||
▼
|
||||
3. Parse patched data as FLAC
|
||||
│
|
||||
▼
|
||||
4. Extract tag values from parsed FLAC
|
||||
│
|
||||
▼
|
||||
5. Update beets Item with new values
|
||||
│
|
||||
▼
|
||||
6. lib.store(item) + lib.save() → SQLite
|
||||
│
|
||||
▼
|
||||
7. Regenerate header for subsequent reads
|
||||
```
|
||||
|
||||
### 3.4 InterpolatedFLAC Class
|
||||
|
||||
**Location**: Lines 274-388
|
||||
|
||||
```python
|
||||
class InterpolatedFLAC(FLAC):
|
||||
"""Custom FLAC handler that can load from bytes and generate headers."""
|
||||
|
||||
def load(self, filedata):
|
||||
"""Load FLAC from byte string instead of file."""
|
||||
self.metadata_blocks = []
|
||||
self.tags = None
|
||||
self.filedata = filedata
|
||||
self.fileobj = BytesIO(filedata)
|
||||
self.__check_header(self.fileobj)
|
||||
|
||||
while self.__read_metadata_block(self.fileobj):
|
||||
pass
|
||||
|
||||
# Verify audio frame starts correctly
|
||||
if self.fileobj.read(2) not in ["\xff\xf8", "\xff\xf9"]:
|
||||
raise FLACNoHeaderError("End of metadata did not start audio")
|
||||
|
||||
def get_header(self, filename=None):
|
||||
"""Generate FLAC header with current metadata."""
|
||||
# Add padding block
|
||||
self.metadata_blocks.append(Padding('\x00' * 1020))
|
||||
MetadataBlock.group_padding(self.metadata_blocks)
|
||||
|
||||
# Calculate available space
|
||||
header = self.__check_header(self.fileobj)
|
||||
available = self.__find_audio_offset(self.fileobj) - header
|
||||
data = MetadataBlock.writeblocks(self.metadata_blocks)
|
||||
|
||||
# Adjust padding to match available space
|
||||
if len(data) > available:
|
||||
# Reduce padding
|
||||
padding = self.metadata_blocks[-1]
|
||||
padding.length -= (len(data) - available)
|
||||
data = MetadataBlock.writeblocks(self.metadata_blocks)
|
||||
elif len(data) < available:
|
||||
# Increase padding
|
||||
self.metadata_blocks[-1].length += (available - len(data))
|
||||
data = MetadataBlock.writeblocks(self.metadata_blocks)
|
||||
|
||||
self.__offset = len("fLaC" + data)
|
||||
return "fLaC" + data
|
||||
|
||||
def offset(self):
|
||||
"""Return byte offset where audio data starts."""
|
||||
return self.__offset
|
||||
```
|
||||
|
||||
**FLAC Structure**:
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ "fLaC" │ STREAMINFO │ VORBIS_COMMENT │ ... │ PADDING │ AUDIO... │
|
||||
│ (4B) │ block │ block │ │ block │ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│◄──────── metadata_blocks ─────────►│
|
||||
│ │
|
||||
└──── get_header() returns this ─────┘
|
||||
```
|
||||
|
||||
### 3.5 InterpolatedID3 Class
|
||||
|
||||
**Location**: Lines 200-271
|
||||
|
||||
```python
|
||||
class InterpolatedID3(ID3):
|
||||
"""Custom ID3 handler for MP3 files."""
|
||||
|
||||
def save(self, filename=None, v1=0):
|
||||
"""Save ID3 tags to file."""
|
||||
# Sort frames by importance
|
||||
order = ["TIT2", "TPE1", "TRCK", "TALB", "TPOS", "TDRC", "TCON"]
|
||||
# ... write header ...
|
||||
```
|
||||
|
||||
**Note**: MP3 support is **incomplete** in the current implementation. The `FileHandler.__init__` sets `self.bound = 0` for MP3, effectively disabling interpolation.
|
||||
|
||||
---
|
||||
|
||||
## 4. Supported Metadata Fields
|
||||
|
||||
**Location**: Lines 55-77
|
||||
|
||||
```python
|
||||
METADATA_RW_FIELDS = [
|
||||
('title', 'text'),
|
||||
('artist', 'text'),
|
||||
('album', 'text'),
|
||||
('genre', 'text'),
|
||||
('composer', 'text'),
|
||||
('grouping', 'text'),
|
||||
('year', 'int'),
|
||||
('month', 'int'),
|
||||
('day', 'int'),
|
||||
('track', 'int'),
|
||||
('tracktotal', 'int'),
|
||||
('disc', 'int'),
|
||||
('disctotal', 'int'),
|
||||
('lyrics', 'text'),
|
||||
('comments', 'text'),
|
||||
('bpm', 'int'),
|
||||
('comp', 'bool'),
|
||||
]
|
||||
```
|
||||
|
||||
**Actually Implemented** (in FileHandler):
|
||||
| Field | Read | Write |
|
||||
|-------|------|-------|
|
||||
| `title` | ✅ | ✅ |
|
||||
| `artist` | ✅ | ✅ |
|
||||
| `album` | ✅ | ✅ |
|
||||
| `genre` | ✅ | ✅ |
|
||||
| Others | ❌ | ❌ |
|
||||
|
||||
---
|
||||
|
||||
## 5. Error Handling
|
||||
|
||||
**Error Codes Used**:
|
||||
|
||||
| Code | Constant | Usage |
|
||||
|------|----------|-------|
|
||||
| 2 | `ENOENT` | File/directory not found |
|
||||
| 13 | `EACCES` | Permission denied |
|
||||
| 1 | `EPERM` | Operation not permitted |
|
||||
| 95 | `EOPNOTSUPP` | Operation not supported |
|
||||
|
||||
**Exception Handling Pattern**:
|
||||
```python
|
||||
def getattr(self, path):
|
||||
try:
|
||||
# ... logic ...
|
||||
except Exception as e:
|
||||
logging.error(e)
|
||||
return -errno.ENOENT
|
||||
```
|
||||
@@ -1,412 +0,0 @@
|
||||
# beetfs Data Flow
|
||||
|
||||
## Overview
|
||||
|
||||
This document details the complete data flow for read and write operations in beetfs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Initialization Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beet mount /mountpoint │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ mount(lib, config, opts, args) │
|
||||
│ │
|
||||
│ 1. Parse PATH_FORMAT into structure_split │
|
||||
│ PATH_FORMAT = "$artist/$album ($year) [$format_upper]/..." │
|
||||
│ structure_split = ["$artist", "$album ($year) [$format_upper]", ...] │
|
||||
│ structure_depth = 3 │
|
||||
│ │
|
||||
│ 2. Store global library reference │
|
||||
│ library = lib │
|
||||
│ │
|
||||
│ 3. Create empty virtual directory tree │
|
||||
│ directory_structure = FSNode({}, {}) │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ for item in lib.items(): │
|
||||
│ │
|
||||
│ For each item in beets library: │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 1. Build template mapping │ │
|
||||
│ │ mapping = { │ │
|
||||
│ │ 'artist': 'Pink Floyd', │ │
|
||||
│ │ 'album': 'The Wall', │ │
|
||||
│ │ 'year': '1979', │ │
|
||||
│ │ 'format_upper': 'FLAC', │ │
|
||||
│ │ 'track': '01', │ │
|
||||
│ │ 'title': 'In The Flesh?', │ │
|
||||
│ │ } │ │
|
||||
│ │ │ │
|
||||
│ │ 2. Substitute template for each level │ │
|
||||
│ │ level_subbed[0] = "Pink Floyd" │ │
|
||||
│ │ level_subbed[1] = "The Wall (1979) [FLAC]" │ │
|
||||
│ │ level_subbed[2] = "01 - Pink Floyd - In The Flesh?.flac" │ │
|
||||
│ │ │ │
|
||||
│ │ 3. Add directories to tree │ │
|
||||
│ │ directory_structure.adddir([], "Pink Floyd") │ │
|
||||
│ │ directory_structure.adddir(["Pink Floyd"], "The Wall (1979)...") │ │
|
||||
│ │ │ │
|
||||
│ │ 4. Add file entry (filename → item.id) │ │
|
||||
│ │ directory_structure.addfile( │ │
|
||||
│ │ ["Pink Floyd", "The Wall (1979) [FLAC]"], │ │
|
||||
│ │ "01 - Pink Floyd - In The Flesh?.flac", │ │
|
||||
│ │ item.id # e.g., 42 │ │
|
||||
│ │ ) │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem FUSE Server │
|
||||
│ │
|
||||
│ server = beetFileSystem(...) │
|
||||
│ server.multithreaded = 0 │
|
||||
│ server.main() ← Enters FUSE event loop │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. File Open Flow
|
||||
|
||||
```
|
||||
Application: open("/mount/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The Flesh?.flac")
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem.open(path, flags) │
|
||||
│ Lines 988-1021 │
|
||||
│ │
|
||||
│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - Pink Floyd - In The..." │
|
||||
│ flags = os.O_RDONLY (or O_RDWR) │
|
||||
│ │
|
||||
│ if path in self.files: │
|
||||
│ # File already open - increment reference count │
|
||||
│ self.files[path].open() │
|
||||
│ return self.files[path] │
|
||||
│ else: │
|
||||
│ # Create new FileHandler │
|
||||
│ self.files[path] = FileHandler(path, self.lib) │
|
||||
│ return self.files[path] │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FileHandler.__init__(path, lib) │
|
||||
│ Lines 440-483 │
|
||||
│ │
|
||||
│ Step 1: Resolve virtual path to beets item │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ pathsplit = ["Pink Floyd", "The Wall (1979) [FLAC]", │ │
|
||||
│ │ "01 - Pink Floyd - In The Flesh?.flac"] │ │
|
||||
│ │ │ │
|
||||
│ │ # Navigate to parent directory in virtual tree │ │
|
||||
│ │ node = directory_structure.getnode(pathsplit[0:2]) │ │
|
||||
│ │ # node.files = {"01 - Pink Floyd - In The Flesh?.flac": 42, ...} │ │
|
||||
│ │ │ │
|
||||
│ │ # Get beets item by ID │ │
|
||||
│ │ item_id = node.files[pathsplit[2]] # 42 │ │
|
||||
│ │ self.item = lib.get_item(id=42) │ │
|
||||
│ │ self.real_path = self.item.path │ │
|
||||
│ │ # e.g., "/mnt/music/torrents/pink_floyd_wall.flac" │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 2: Open real file and detect format │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.file_object = open(self.real_path, 'r+') │ │
|
||||
│ │ self.format = "flac" # from file extension │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 3: Create InterpolatedFLAC with database metadata │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.inf = InterpolatedFLAC(self.file_object.read()) │ │
|
||||
│ │ │ │
|
||||
│ │ # INJECT DATABASE METADATA (this is the key operation!) │ │
|
||||
│ │ self.inf["title"] = self.item.title # "In The Flesh?" │ │
|
||||
│ │ self.inf["album"] = self.item.album # "The Wall" │ │
|
||||
│ │ self.inf["artist"] = self.item.artist # "Pink Floyd" │ │
|
||||
│ │ self.inf["genre"] = self.item.genre # "Progressive Rock" │ │
|
||||
│ │ │ │
|
||||
│ │ # Generate header with injected metadata │ │
|
||||
│ │ self.header = self.inf.get_header(self.real_path) │ │
|
||||
│ │ self.bound = len(self.header) # e.g., 8192 bytes │ │
|
||||
│ │ self.music_offset = self.inf.offset() │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 4: Cache audio data │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.file_object.seek(self.music_offset) │ │
|
||||
│ │ self.music_data = self.file_object.read() # All audio data │ │
|
||||
│ │ self.file_object.close() │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. File Read Flow
|
||||
|
||||
```
|
||||
Application: read(fd, buffer, 4096) # offset managed by kernel
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem.read(path, size, offset, fh) │
|
||||
│ Lines 1077-1106 │
|
||||
│ │
|
||||
│ path = "/Pink Floyd/The Wall (1979) [FLAC]/01 - ..." │
|
||||
│ size = 4096 │
|
||||
│ offset = 0 (first read) or previous offset + bytes_read │
|
||||
│ fh = FileHandler instance │
|
||||
│ │
|
||||
│ return self.files[path].read(size, offset) │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FileHandler.read(size, offset) │
|
||||
│ Lines 497-517 │
|
||||
│ │
|
||||
│ Variables: │
|
||||
│ self.bound = 8192 (header size) │
|
||||
│ self.header = bytes (generated FLAC header with DB metadata) │
|
||||
│ self.music_data = bytes (original audio frames) │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────────┼───────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
|
||||
│ Case 1: Header Only │ │ Case 2: Span Both │ │ Case 3: Audio Only │
|
||||
│ offset < bound │ │ offset < bound │ │ offset >= bound │
|
||||
│ offset+size < bound │ │ offset+size >= bound│ │ │
|
||||
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
|
||||
│ Example: │ │ Example: │ │ Example: │
|
||||
│ offset=0 │ │ offset=8000 │ │ offset=10000 │
|
||||
│ size=4096 │ │ size=4096 │ │ size=4096 │
|
||||
│ bound=8192 │ │ bound=8192 │ │ bound=8192 │
|
||||
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
|
||||
│ Return: │ │ Return: │ │ Return: │
|
||||
│ header[0:4096] │ │ header[8000:8192] │ │ music_data[ │
|
||||
│ │ │ + music_data[0:3904]│ │ 1808:5904] │
|
||||
│ (DB metadata!) │ │ │ │ │
|
||||
│ │ │ (mixed) │ │ (original audio) │
|
||||
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
|
||||
|
||||
|
||||
Visual representation of virtual file:
|
||||
|
||||
0 bound (8192) EOF
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌───────────────────────┬────────────────────────────────────────────┐
|
||||
│ HEADER │ AUDIO DATA │
|
||||
│ (self.header) │ (self.music_data) │
|
||||
│ │ │
|
||||
│ Contains: │ Contains: │
|
||||
│ - "fLaC" magic │ - Original FLAC frames │
|
||||
│ - STREAMINFO block │ - Unchanged from disk │
|
||||
│ - VORBIS_COMMENT │ │
|
||||
│ with DB values: │ │
|
||||
│ title, artist, │ │
|
||||
│ album, genre │ │
|
||||
│ - PADDING block │ │
|
||||
└───────────────────────┴────────────────────────────────────────────┘
|
||||
▲ ▲
|
||||
│ │
|
||||
From InterpolatedFLAC From original file
|
||||
with injected DB tags (passed through)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. File Write Flow
|
||||
|
||||
```
|
||||
Application: write(fd, "TITLE=New Title\0", 16) # Hypothetical tag edit
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem.write(path, buf, offset, fh) │
|
||||
│ Lines 1108-1135 │
|
||||
│ │
|
||||
│ return self.files[path].write(offset, buf) │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FileHandler.write(offset, buf) │
|
||||
│ Lines 519-565 │
|
||||
│ │
|
||||
│ if offset >= self.bound: │
|
||||
│ # Write is in audio area - DISCARD │
|
||||
│ return # Do nothing, audio is read-only │
|
||||
│ │
|
||||
│ # Write is in header area - process tag update │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Step 1: Reconstruct full virtual file in memory │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ filedata = self.header + self.music_data │ │
|
||||
│ │ │ │
|
||||
│ │ # Patch in new data │ │
|
||||
│ │ filedata = filedata[0:offset] + buf + filedata[offset + len(buf):] │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 2: Parse patched data as FLAC │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.inf = InterpolatedFLAC(filedata) │ │
|
||||
│ │ # This parses the FLAC structure and extracts Vorbis comments │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 3: Extract tag values from parsed FLAC │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.item.title = str(self.inf["title"][0]).encode('utf-8') │ │
|
||||
│ │ self.item.album = str(self.inf["album"][0]).encode('utf-8') │ │
|
||||
│ │ self.item.artist = str(self.inf["artist"][0]).encode('utf-8') │ │
|
||||
│ │ self.item.genre = str(self.inf["genre"][0]).encode('utf-8') │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 4: Save to beets database │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.lib.store(self.item) # Update item in library │ │
|
||||
│ │ self.lib.save() # Persist to SQLite │ │
|
||||
│ │ │ │
|
||||
│ │ # NOTE: Original file on disk is NEVER touched! │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Step 5: Regenerate header for subsequent reads │
|
||||
│ ┌───────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ self.inf["title"] = self.item.title │ │
|
||||
│ │ self.inf["album"] = self.item.album │ │
|
||||
│ │ self.inf["artist"] = self.item.artist │ │
|
||||
│ │ self.inf["genre"] = self.item.genre │ │
|
||||
│ │ │ │
|
||||
│ │ self.header = self.inf.get_header(self.real_path) │ │
|
||||
│ │ self.bound = len(self.header) │ │
|
||||
│ └───────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ return len(buf) # Success │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
|
||||
Write data flow summary:
|
||||
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Application │ │ beetfs │ │ Beets │ │ Original │
|
||||
│ writes │────▶│ parses │────▶│ database │ │ file │
|
||||
│ new tags │ │ extracts │ │ updated │ │ UNTOUCHED │
|
||||
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. File Release Flow
|
||||
|
||||
```
|
||||
Application: close(fd)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem.release(path, flags, fh) │
|
||||
│ Lines 1049-1059 │
|
||||
│ │
|
||||
│ if self.files[path].release(): │
|
||||
│ # Reference count reached 0, clean up │
|
||||
│ del self.files[path] │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FileHandler.release() │
|
||||
│ Lines 489-495 │
|
||||
│ │
|
||||
│ self.instance_count -= 1 │
|
||||
│ │
|
||||
│ if self.instance_count == 0: │
|
||||
│ return True # OK to delete │
|
||||
│ else: │
|
||||
│ return False # Still in use │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Directory Listing Flow
|
||||
|
||||
```
|
||||
Application: ls /mount/Pink\ Floyd/
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ beetFileSystem.readdir(path, offset, dh) │
|
||||
│ Lines 931-975 │
|
||||
│ │
|
||||
│ path = "/Pink Floyd" │
|
||||
│ pathsplit = ["Pink Floyd"] │
|
||||
│ │
|
||||
│ yield fuse.Direntry(".") │
|
||||
│ yield fuse.Direntry("..") │
|
||||
│ │
|
||||
│ # len(pathsplit) == 1, structure_depth - 1 == 2 │
|
||||
│ # So we're listing directories (albums), not files │
|
||||
│ │
|
||||
│ for dirname in directory_structure.listdir(pathsplit, True): │
|
||||
│ yield fuse.Direntry(dirname.encode('utf-8')) │
|
||||
│ # "The Wall (1979) [FLAC]" │
|
||||
│ # "Animals (1977) [FLAC]" │
|
||||
│ # etc. │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Complete Request Lifecycle
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||
│ COMPLETE LIFECYCLE │
|
||||
│ │
|
||||
│ 1. User mounts: beet mount /mnt/music │
|
||||
│ ├─ Build virtual tree from beets library │
|
||||
│ └─ Start FUSE event loop │
|
||||
│ │
|
||||
│ 2. Application opens file: open("/mnt/music/Artist/Album/track.flac") │
|
||||
│ ├─ Resolve virtual path to beets item ID │
|
||||
│ ├─ Load original file into memory │
|
||||
│ ├─ Inject database metadata into FLAC structure │
|
||||
│ ├─ Generate new header with DB tags │
|
||||
│ └─ Cache audio data │
|
||||
│ │
|
||||
│ 3. Application reads file: read(fd, buf, 4096) │
|
||||
│ ├─ If reading header region → return header (DB metadata) │
|
||||
│ ├─ If reading audio region → return cached audio (original) │
|
||||
│ └─ If spanning both → return combined data │
|
||||
│ │
|
||||
│ 4. Application writes tags: write(fd, new_tags, offset) │
|
||||
│ ├─ If audio region → discard (read-only) │
|
||||
│ ├─ If header region: │
|
||||
│ │ ├─ Parse new tag values │
|
||||
│ │ ├─ Update beets database │
|
||||
│ │ └─ Regenerate header │
|
||||
│ └─ Original file NEVER modified │
|
||||
│ │
|
||||
│ 5. Application closes file: close(fd) │
|
||||
│ ├─ Decrement reference count │
|
||||
│ └─ Clean up if count == 0 │
|
||||
│ │
|
||||
│ 6. User unmounts: fusermount -u /mnt/music │
|
||||
│ └─ fsdestroy() called, cleanup │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
@@ -1,479 +0,0 @@
|
||||
# beetfs Drawbacks & Limitations
|
||||
|
||||
## Overview
|
||||
|
||||
This document catalogs all identified issues, limitations, and missing features in beetfs. Issues are categorized by severity and type.
|
||||
|
||||
---
|
||||
|
||||
## Critical Issues (🔴)
|
||||
|
||||
### 1. Full File Loading into Memory
|
||||
|
||||
**Location**: Lines 463, 480-481
|
||||
|
||||
```python
|
||||
self.inf = InterpolatedFLAC(self.file_object.read()) # Entire file
|
||||
# ...
|
||||
self.music_data = self.file_object.read() # Audio portion again
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Memory usage = O(file_size) per open file
|
||||
- 50MB FLAC = ~50MB RAM
|
||||
- Library scan of 100 files = 5GB+ RAM
|
||||
- Out-of-memory crashes on large libraries
|
||||
|
||||
**Fix Required**: Implement lazy loading with seek-based reads.
|
||||
|
||||
---
|
||||
|
||||
### 2. MP3 Support Disabled
|
||||
|
||||
**Location**: Lines 475-477
|
||||
|
||||
```python
|
||||
elif self.format == "mp3":
|
||||
self.bound = 0 # disable interpolation for now
|
||||
self.music_offset = 0 # disable interpolation for now
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- MP3 files return original metadata, not database metadata
|
||||
- Breaks the core promise of metadata overlay
|
||||
- MP3 is still one of the most common formats
|
||||
|
||||
**Fix Required**: Implement `InterpolatedID3` header generation.
|
||||
|
||||
---
|
||||
|
||||
### 3. Python 2 Only
|
||||
|
||||
**Location**: Throughout
|
||||
|
||||
```python
|
||||
except fuse.FuseError, e: # Python 2 syntax
|
||||
if isinstance(value, basestring): # Removed in Python 3
|
||||
return reduce(lambda a, b: (a << 8) + ord(b), string, 0L) # Long literals
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Python 2 EOL was January 2020
|
||||
- Security vulnerabilities unfixed
|
||||
- No modern library support
|
||||
- Cannot run on Python 3 without migration
|
||||
|
||||
**Fix Required**: Full Python 3 migration (see modernization.md).
|
||||
|
||||
---
|
||||
|
||||
### 4. Deprecated FUSE Library
|
||||
|
||||
**Location**: Line 25, 51
|
||||
|
||||
```python
|
||||
import fuse
|
||||
fuse.fuse_python_api = (0, 2)
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- fuse-python is unmaintained
|
||||
- Missing modern FUSE features (FUSE 3.x)
|
||||
- Compatibility issues with recent kernels
|
||||
- No async support
|
||||
|
||||
**Fix Required**: Migrate to pyfuse3 or llfuse.
|
||||
|
||||
---
|
||||
|
||||
### 5. Single-Threaded Execution
|
||||
|
||||
**Location**: Line 178
|
||||
|
||||
```python
|
||||
server.multithreaded = 0
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- All operations serialized
|
||||
- One slow open blocks all other operations
|
||||
- Cannot utilize multiple CPU cores
|
||||
- Poor performance under concurrent access
|
||||
|
||||
**Fix Required**: Enable multithreading with proper locking.
|
||||
|
||||
---
|
||||
|
||||
## Major Issues (🟡)
|
||||
|
||||
### 6. Limited Metadata Fields
|
||||
|
||||
**Location**: Lines 466-469, 540-547
|
||||
|
||||
```python
|
||||
# Only these 4 fields are actually used:
|
||||
self.inf["title"] = self.item.title
|
||||
self.inf["album"] = self.item.album
|
||||
self.inf["artist"] = self.item.artist
|
||||
self.inf["genre"] = self.item.genre
|
||||
```
|
||||
|
||||
**Defined but not implemented** (lines 55-77):
|
||||
- `composer`, `grouping`
|
||||
- `year`, `month`, `day`
|
||||
- `track`, `tracktotal`
|
||||
- `disc`, `disctotal`
|
||||
- `lyrics`, `comments`
|
||||
- `bpm`, `comp`
|
||||
- `albumartist` (not even defined)
|
||||
|
||||
**Impact**:
|
||||
- Track numbers not from database
|
||||
- Album artist not supported
|
||||
- Year/date not interpolated
|
||||
- Cover art not handled
|
||||
|
||||
---
|
||||
|
||||
### 7. No File Handle Caching/Eviction
|
||||
|
||||
**Location**: Lines 1004-1018
|
||||
|
||||
```python
|
||||
if path in self.files:
|
||||
self.files[path].open()
|
||||
else:
|
||||
self.files[path] = FileHandler(path, self.lib)
|
||||
```
|
||||
|
||||
**Missing**:
|
||||
- No maximum cache size
|
||||
- No LRU eviction
|
||||
- No memory pressure handling
|
||||
- Files stay in memory until explicitly closed
|
||||
|
||||
**Impact**:
|
||||
- Memory grows unbounded
|
||||
- No protection against OOM
|
||||
- Applications that open-then-close still leave data cached
|
||||
|
||||
---
|
||||
|
||||
### 8. Blocking Database Operations
|
||||
|
||||
**Location**: Lines 549-550
|
||||
|
||||
```python
|
||||
self.lib.store(self.item)
|
||||
self.lib.save()
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- SQLite operations in FUSE thread
|
||||
- Write operations block all reads
|
||||
- No transaction batching
|
||||
- Potential deadlocks with beets
|
||||
|
||||
---
|
||||
|
||||
### 9. No Library Hot Reload
|
||||
|
||||
**Issue**: Virtual directory tree built once at mount time.
|
||||
|
||||
**Location**: Lines 142-172
|
||||
|
||||
```python
|
||||
for item in lib.items():
|
||||
# Build tree...
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- New files added to beets library not visible
|
||||
- Deleted files still appear (ENOENT on access)
|
||||
- Metadata changes in beets not reflected until remount
|
||||
- Must unmount/remount to see changes
|
||||
|
||||
---
|
||||
|
||||
### 10. Static Path Format
|
||||
|
||||
**Location**: Lines 44-45
|
||||
|
||||
```python
|
||||
PATH_FORMAT = ("$artist/$album ($year) [$format_upper]/"
|
||||
"$track - $artist - $title.$format")
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Cannot customize organization
|
||||
- Hard-coded template
|
||||
- No configuration option
|
||||
- Incompatible with different organizational preferences
|
||||
|
||||
---
|
||||
|
||||
### 11. No Extended Attribute Support
|
||||
|
||||
**Location**: Not implemented
|
||||
|
||||
**Impact**:
|
||||
- Cannot store/retrieve xattrs
|
||||
- Some applications use xattrs for metadata
|
||||
- macOS Finder metadata lost
|
||||
- Linux capabilities not supported
|
||||
|
||||
---
|
||||
|
||||
### 12. No Symlink Support
|
||||
|
||||
**Location**: Lines 758-765
|
||||
|
||||
```python
|
||||
def readlink(self, path):
|
||||
return -errno.EOPNOTSUPP
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Cannot create symlinks in mount
|
||||
- Some applications expect symlink support
|
||||
- Cannot link to external files
|
||||
|
||||
---
|
||||
|
||||
### 13. Silent Error Swallowing
|
||||
|
||||
**Location**: Lines 705-707, 1019-1021, 1103-1104
|
||||
|
||||
```python
|
||||
except Exception as e:
|
||||
logging.error(e)
|
||||
return -errno.ENOENT # Always returns same error
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- All errors appear as "file not found"
|
||||
- Hard to debug issues
|
||||
- No distinction between permission, I/O, parse errors
|
||||
- Lost stack traces in many cases
|
||||
|
||||
---
|
||||
|
||||
## Minor Issues (🟢)
|
||||
|
||||
### 14. Global State
|
||||
|
||||
**Location**: Lines 125-140
|
||||
|
||||
```python
|
||||
global structure_split
|
||||
global structure_depth
|
||||
global library
|
||||
global directory_structure
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Cannot mount multiple instances
|
||||
- Difficult to unit test
|
||||
- Tight coupling between components
|
||||
- No dependency injection
|
||||
|
||||
---
|
||||
|
||||
### 15. Hard-coded Log File
|
||||
|
||||
**Location**: Lines 624-625
|
||||
|
||||
```python
|
||||
LOG_FILENAME = "LOG"
|
||||
logging.basicConfig(filename=LOG_FILENAME, level=logging.INFO,)
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- Log file created in current directory
|
||||
- No log rotation
|
||||
- No configurable log level
|
||||
- Fills disk on busy systems
|
||||
|
||||
---
|
||||
|
||||
### 16. Reference Count Manual Management
|
||||
|
||||
**Location**: Lines 485-495
|
||||
|
||||
```python
|
||||
def open(self):
|
||||
self.instance_count = self.instance_count + 1
|
||||
|
||||
def release(self):
|
||||
if self.instance_count > 0:
|
||||
self.instance_count = self.instance_count - 1
|
||||
```
|
||||
|
||||
**Issues**:
|
||||
- Race conditions possible if multithreaded
|
||||
- No context manager support
|
||||
- Manual counting error-prone
|
||||
- Off-by-one potential
|
||||
|
||||
---
|
||||
|
||||
### 17. Inefficient Directory Building
|
||||
|
||||
**Location**: Lines 153-172
|
||||
|
||||
```python
|
||||
for level in range(0, structure_depth - 1):
|
||||
if level-1 in level_subbed:
|
||||
sub_elements.append(level_subbed[level-1])
|
||||
directory_structure.adddir(sub_elements, level_subbed[level])
|
||||
```
|
||||
|
||||
**Issues**:
|
||||
- Rebuilds path for every item
|
||||
- O(items × depth) complexity
|
||||
- String allocations in inner loop
|
||||
- Could use trie-based insertion
|
||||
|
||||
---
|
||||
|
||||
### 18. No Cover Art Handling
|
||||
|
||||
**Issue**: Cover art embedded in FLAC not addressed.
|
||||
|
||||
**Impact**:
|
||||
- Cover art from original file used, not database
|
||||
- Cannot replace/add cover art through overlay
|
||||
- PICTURE metadata blocks passed through unchanged
|
||||
|
||||
---
|
||||
|
||||
### 19. No Cue Sheet Support
|
||||
|
||||
**Issue**: Cue sheets not handled specially.
|
||||
|
||||
**Impact**:
|
||||
- `.cue` files point to original file paths
|
||||
- Cannot play cue-referenced tracks correctly
|
||||
- Split-by-cue not supported
|
||||
|
||||
---
|
||||
|
||||
### 20. File Size Mismatch Potential
|
||||
|
||||
**Issue**: Virtual file size differs from physical if header size changes.
|
||||
|
||||
**Location**: Lines 675-688
|
||||
|
||||
```python
|
||||
statinfo = os.stat(item)
|
||||
st = Stat(st_mode=statinfo.st_mode,
|
||||
st_size=statinfo.st_size, # Original size, not virtual!
|
||||
...)
|
||||
```
|
||||
|
||||
**Impact**:
|
||||
- `stat()` returns original file size
|
||||
- If generated header is larger/smaller, size is wrong
|
||||
- Some applications may fail on size mismatch
|
||||
- Range requests could break
|
||||
|
||||
---
|
||||
|
||||
## Missing Features
|
||||
|
||||
### Essential
|
||||
|
||||
| Feature | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| MP3 metadata interpolation | ❌ Disabled | Code exists but disabled |
|
||||
| OGG/Opus support | ❌ Missing | No implementation |
|
||||
| AAC/M4A support | ❌ Missing | No implementation |
|
||||
| Lazy file loading | ❌ Missing | Full file loaded |
|
||||
| Memory management | ❌ Missing | No limits or eviction |
|
||||
| Configuration file | ❌ Missing | Hard-coded values |
|
||||
|
||||
### Nice to Have
|
||||
|
||||
| Feature | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| Cover art interpolation | ❌ Missing | Would need PICTURE block handling |
|
||||
| ReplayGain from database | ❌ Missing | Tags not interpolated |
|
||||
| Lyrics from database | ❌ Missing | Listed in fields, not implemented |
|
||||
| Watch mode (hot reload) | ❌ Missing | No inotify integration |
|
||||
| Multiple mount points | ❌ Missing | Global state prevents |
|
||||
| Remote database | ❌ Missing | Local beets only |
|
||||
| Read-only mode | ❌ Missing | Always allows writes |
|
||||
| Custom path templates | ❌ Missing | Hard-coded PATH_FORMAT |
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### 1. No Input Validation
|
||||
|
||||
**Location**: Throughout
|
||||
|
||||
```python
|
||||
pathsplit = path[1:].split('/')
|
||||
item_id = node.files[pathsplit[structure_depth-1]] # No bounds check
|
||||
```
|
||||
|
||||
**Risk**: Path traversal, injection attacks unlikely but possible.
|
||||
|
||||
### 2. Database Credentials Exposed
|
||||
|
||||
**Issue**: Uses beets library directly with stored credentials.
|
||||
|
||||
**Risk**: Low - local access only.
|
||||
|
||||
### 3. No Permission Enforcement
|
||||
|
||||
**Location**: Lines 749-756
|
||||
|
||||
```python
|
||||
if flags | os.R_OK:
|
||||
pass # TODO: actually check the file permissions
|
||||
if flags | os.W_OK:
|
||||
pass
|
||||
```
|
||||
|
||||
**Risk**: All users can read/write through mount.
|
||||
|
||||
---
|
||||
|
||||
## Compatibility Issues
|
||||
|
||||
| Component | Issue |
|
||||
|-----------|-------|
|
||||
| **Jellyfin** | May scan entire library, causing OOM |
|
||||
| **Plex** | Same library scan issue |
|
||||
| **Navidrome** | Expects certain tag fields not implemented |
|
||||
| **mpd** | Works for playback, database features limited |
|
||||
| **macOS** | fuse-python macOS support questionable |
|
||||
| **Docker** | FUSE in containers requires privileged mode |
|
||||
|
||||
---
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Category | Critical | Major | Minor |
|
||||
|----------|----------|-------|-------|
|
||||
| Performance | 2 | 4 | 2 |
|
||||
| Functionality | 2 | 5 | 4 |
|
||||
| Code Quality | 2 | 2 | 4 |
|
||||
| **Total** | **6** | **11** | **10** |
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Fix List
|
||||
|
||||
1. 🔴 **Memory**: Implement lazy loading (Critical for usability)
|
||||
2. 🔴 **Python 3**: Migrate to Python 3 (Required for any changes)
|
||||
3. 🔴 **FUSE lib**: Switch to pyfuse3/llfuse (Required for Python 3)
|
||||
4. 🔴 **MP3**: Enable MP3 interpolation (Core functionality)
|
||||
5. 🟡 **Metadata**: Implement all fields (Feature completeness)
|
||||
6. 🟡 **Threading**: Enable multithreading (Performance)
|
||||
7. 🟡 **Config**: Add configuration file (Usability)
|
||||
8. 🟡 **Hot reload**: Watch for library changes (Usability)
|
||||
9. 🟢 **Globals**: Remove global state (Code quality)
|
||||
10. 🟢 **Logging**: Configurable logging (Operations)
|
||||
@@ -1,493 +0,0 @@
|
||||
# beetfs E2E Test Plan
|
||||
|
||||
> **Reviewed by Oracle** - Critical bug discovered, plan updated accordingly
|
||||
|
||||
## Test Results (Latest Run)
|
||||
|
||||
```
|
||||
Tests run: 74
|
||||
Passed: 12
|
||||
Failures: 56
|
||||
Errors: 3
|
||||
Skipped: 3
|
||||
Duration: ~103 seconds
|
||||
```
|
||||
|
||||
### Bugs Detected by Tests
|
||||
|
||||
| Bug | Tests Affected | Description |
|
||||
|-----|----------------|-------------|
|
||||
| **Nested Methods** | 56 | Lines 758-1144 indented inside `access()` - FUSE operations unreachable |
|
||||
| **Directory Tree Building** | 3 | `KeyError` in `FSNode.getnode()` when adding files |
|
||||
| **Unmount** | 1 | Filesystem not unmounting cleanly |
|
||||
|
||||
### Passing Tests (12)
|
||||
|
||||
- `test_fuse_available` - FUSE/fusermount detected
|
||||
- `test_library_fixture_created` - SQLite DB and music dir created
|
||||
- `test_temp_directory_created` - Temp dirs set up correctly
|
||||
- `test_mount_empty_library` - **Mount works with empty library!**
|
||||
- `test_list_empty_root` - Empty root returns empty list
|
||||
- `test_list_root_returns_list` - Returns list type
|
||||
- `test_access_empty_path` - Handles empty path
|
||||
- Plus 5 nested bug detection tests (confirming bug exists)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
E2E tests for beetfs FUSE filesystem using real music files from qBittorrent container. No mocks - actual filesystem operations against mounted beetfs.
|
||||
|
||||
### Critical Finding
|
||||
|
||||
**BUG DISCOVERED**: Lines 758-1144 in `beetFs.py` are indented inside `access()` method, making these FUSE operations unreachable as class methods:
|
||||
- `readdir`, `open`, `read`, `write`, `mkdir`, `unlink`, `rmdir`, `symlink`, `link`, `rename`, `chmod`, `chown`, `truncate`, `opendir`, `releasedir`, `fsyncdir`, `create`, `fgetattr`, `release`, `fsync`, `flush`, `ftruncate`
|
||||
|
||||
Tests will expose this immediately - write `test_readdir.py` first.
|
||||
|
||||
---
|
||||
|
||||
## Test Environment
|
||||
|
||||
| Component | Status | Details |
|
||||
|-----------|--------|---------|
|
||||
| Real Music | Available | Metallica "72 Seasons" (12 FLAC, 650MB) at `/home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/` |
|
||||
| Synthetic Music | Create | 5-10MB FLACs for most tests (avoid RAM explosion) |
|
||||
| Beets Config | Create | `~/.config/beets/config.yaml` for test isolation |
|
||||
| Beets Library | Empty | Needs import of test files |
|
||||
| Python | 2.7.15 | Via Nix flake (nixpkgs-18.09) |
|
||||
| Test Framework | unittest | stdlib, no external deps for Py2.7 |
|
||||
|
||||
---
|
||||
|
||||
## Test Architecture
|
||||
|
||||
```
|
||||
beetfs/tests/
|
||||
├── __init__.py
|
||||
├── conftest.py # Test fixtures, beets library setup, synthetic FLAC creation
|
||||
├── test_smoke.py # Mount/unmount lifecycle (run FIRST)
|
||||
├── test_nested_bug.py # Verify the indentation bug (run SECOND)
|
||||
├── test_readdir.py # Directory listing operations
|
||||
├── test_read.py # File reading with metadata overlay (CORE FEATURE)
|
||||
├── test_stat.py # getattr, fgetattr, statfs
|
||||
├── test_write.py # Metadata write operations
|
||||
├── test_error_handling.py # ENOENT, EOPNOTSUPP scenarios
|
||||
├── test_edge_cases.py # Unicode, concurrent opens, special chars
|
||||
├── test_integration.py # Real 650MB files (skip by default)
|
||||
└── fixtures/
|
||||
├── synthetic/ # Generated 5-10MB test FLACs
|
||||
└── real -> /home/fujin/.local/share/docker/volumes/containers_downloads/_data/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Tiers
|
||||
|
||||
### Tier 1: Unit-ish (Synthetic FLACs, ~500KB each)
|
||||
- Fast execution
|
||||
- No memory issues (FileHandler loads entire file to RAM)
|
||||
- Run on every commit
|
||||
|
||||
### Tier 2: Integration (Subset of real files, 1-2 tracks)
|
||||
- Uses real Metallica FLACs
|
||||
- Tests real-world metadata
|
||||
- Run before merge
|
||||
|
||||
### Tier 3: E2E (All 12 tracks, 650MB)
|
||||
- Full album processing
|
||||
- Memory stress testing
|
||||
- Run via `E2E=1 python -m unittest discover`
|
||||
- Skip by default
|
||||
|
||||
---
|
||||
|
||||
## Test Isolation Strategy
|
||||
|
||||
| Resource | Strategy | Rationale |
|
||||
|----------|----------|-----------|
|
||||
| Audio Files | **Symlinks** for reads | beetfs NEVER writes to source files, only to beets DB |
|
||||
| Beets DB | **Copy per test** | Writes mutate DB; need isolation |
|
||||
| Mount Point | **Fresh tempdir** | Each test gets clean mount |
|
||||
| Global State | **Fresh subprocess** | `library`, `directory_structure` are module globals |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
> Reordered per Oracle recommendation: smoke → nested-bug → read → write → errors → edge
|
||||
|
||||
### Phase 1: Infrastructure (Day 1 AM)
|
||||
|
||||
1. Create `tests/` directory structure
|
||||
2. Implement `BeetFSTestCase` base class with:
|
||||
- Subprocess timeout via `threading.Timer` (Py2.7 compatible)
|
||||
- Mount wait polling (`os.path.ismount()`)
|
||||
- Proper cleanup (`fusermount -u`)
|
||||
3. Create synthetic FLAC generator using ffmpeg + flac CLI
|
||||
4. Setup isolated beets config and library
|
||||
|
||||
### Phase 2: Bug Detection (Day 1 PM)
|
||||
|
||||
5. `test_smoke.py` - Mount/unmount lifecycle
|
||||
6. `test_nested_bug.py` - Verify `readdir`, `open` are callable (will fail, exposing bug)
|
||||
|
||||
### Phase 3: Core Tests (Day 2)
|
||||
|
||||
7. `test_readdir.py` - Directory listing
|
||||
8. `test_read.py` - **Metadata overlay verification** (critical)
|
||||
9. `test_stat.py` - File/directory attributes
|
||||
|
||||
### Phase 4: Write & Errors (Day 3)
|
||||
|
||||
10. `test_write.py` - Metadata modification, DB persistence
|
||||
11. `test_error_handling.py` - ENOENT, EOPNOTSUPP
|
||||
|
||||
### Phase 5: Edge Cases (Day 3-4)
|
||||
|
||||
12. `test_edge_cases.py` - Unicode, concurrent opens, special chars
|
||||
13. `test_integration.py` - Real 650MB files (optional tier)
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
|
||||
### 1. Smoke Tests (`test_smoke.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_mount_success` | Mount beetfs | `os.path.ismount()` returns True |
|
||||
| `test_unmount_clean` | Unmount | Process exits 0, dir accessible |
|
||||
| `test_mount_empty_library` | Mount with 0 items | Mounts successfully, root empty |
|
||||
| `test_mount_invalid_path` | Mount to non-existent | Fails gracefully |
|
||||
| `test_fsinit_called` | Check initialization | No crash on mount |
|
||||
|
||||
### 2. Nested Methods Bug (`test_nested_bug.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_readdir_exists` | `hasattr(beetFileSystem, 'readdir')` | True (currently False!) |
|
||||
| `test_open_exists` | `hasattr(beetFileSystem, 'open')` | True (currently False!) |
|
||||
| `test_read_exists` | `hasattr(beetFileSystem, 'read')` | True (currently False!) |
|
||||
| `test_readdir_callable` | `os.listdir(mount)` | Returns list (currently fails!) |
|
||||
|
||||
### 3. Directory Operations (`test_readdir.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_list_root` | `os.listdir(mount)` | Returns artist directories |
|
||||
| `test_list_artist` | `os.listdir(mount/artist)` | Returns album directories |
|
||||
| `test_list_album` | `os.listdir(mount/artist/album)` | Returns track files |
|
||||
| `test_path_format` | Check structure | Matches `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format` |
|
||||
| `test_unicode_paths` | Non-ASCII chars | Handles "Lux Aeterna" correctly |
|
||||
|
||||
### 4. Read Operations (`test_read.py`) - CORE FEATURE
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_read_header_overlay` | Read + parse with mutagen | Tags match DB, not file |
|
||||
| `test_read_audio_passthrough` | Compare audio bytes | Identical to original after header |
|
||||
| `test_read_full_file` | Read entire file | Header from DB + audio from file |
|
||||
| `test_metadata_artist` | Check artist tag | DB value, not file value |
|
||||
| `test_metadata_title` | Check title tag | DB value, not file value |
|
||||
| `test_metadata_album` | Check album tag | DB value, not file value |
|
||||
| `test_metadata_genre` | Check genre tag | DB value, not file value |
|
||||
| `test_original_unchanged` | Read original file | Original metadata intact |
|
||||
|
||||
#### Metadata Overlay Verification Pattern
|
||||
|
||||
```python
|
||||
import mutagen.flac
|
||||
from io import BytesIO
|
||||
|
||||
def test_read_header_overlay(self):
|
||||
# Setup: Import file, modify DB metadata
|
||||
# beet import /path/to/file
|
||||
# beet modify artist="DB Artist" # File has "Original Artist"
|
||||
|
||||
# Read mounted file as bytes
|
||||
with open(os.path.join(self.mount_dir, 'DB Artist/...'), 'rb') as f:
|
||||
mounted_data = f.read()
|
||||
|
||||
# Parse with mutagen
|
||||
flac = mutagen.flac.FLAC(BytesIO(mounted_data))
|
||||
|
||||
# Verify overlay worked
|
||||
self.assertEqual(flac['artist'][0], 'DB Artist') # From DB
|
||||
self.assertNotEqual(flac['artist'][0], 'Original Artist') # Not from file
|
||||
```
|
||||
|
||||
### 5. Stat Operations (`test_stat.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_stat_file` | `os.stat(file)` | Valid stat with size, mtime |
|
||||
| `test_stat_directory` | `os.stat(dir)` | Directory mode (S_IFDIR) |
|
||||
| `test_statfs` | `os.statvfs(mount)` | Valid filesystem stats |
|
||||
| `test_access_read` | `os.access(file, R_OK)` | True |
|
||||
| `test_access_write` | `os.access(file, W_OK)` | True (header writable) |
|
||||
|
||||
### 6. Write Operations (`test_write.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_write_title` | Modify title in header | DB updated, file unchanged |
|
||||
| `test_write_artist` | Modify artist | DB updated |
|
||||
| `test_write_album` | Modify album | DB updated |
|
||||
| `test_write_genre` | Modify genre | DB updated |
|
||||
| `test_write_audio_discarded` | Write at offset > bound | Silently discarded |
|
||||
| `test_write_persistence` | Write -> unmount -> remount | Changes persisted in DB |
|
||||
| `test_write_mp3_noop` | Write to MP3 header | No error, but no effect (bound=0) |
|
||||
|
||||
### 7. Error Handling (`test_error_handling.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_enoent_file` | Read non-existent | `OSError(ENOENT)` |
|
||||
| `test_enoent_dir` | List non-existent | `OSError(ENOENT)` |
|
||||
| `test_eopnotsupp_mkdir` | `os.mkdir()` | `OSError(EOPNOTSUPP)` |
|
||||
| `test_eopnotsupp_unlink` | `os.unlink()` | `OSError(EOPNOTSUPP)` |
|
||||
| `test_eopnotsupp_rename` | `os.rename()` | `OSError(EOPNOTSUPP)` |
|
||||
| `test_eopnotsupp_symlink` | `os.symlink()` | `OSError(EOPNOTSUPP)` |
|
||||
|
||||
### 8. Edge Cases (`test_edge_cases.py`)
|
||||
|
||||
| Test | Operation | Expected |
|
||||
|------|-----------|----------|
|
||||
| `test_special_chars_sanitized` | Path with `?/` | Sanitized via `sanitize()` |
|
||||
| `test_concurrent_opens` | Open same file twice | `instance_count` increments |
|
||||
| `test_concurrent_release` | Release after double open | File stays cached until count=0 |
|
||||
| `test_unicode_metadata` | Non-ASCII in artist/title | Handled correctly |
|
||||
| `test_empty_metadata` | None/empty fields | Doesn't crash |
|
||||
| `test_mp3_no_interpolation` | Read MP3 | Returns original file (no overlay) |
|
||||
|
||||
### 9. Integration (`test_integration.py`)
|
||||
|
||||
| Test | Env Var | Expected |
|
||||
|------|---------|----------|
|
||||
| `test_real_album_listing` | `E2E=1` | Lists all 12 Metallica tracks |
|
||||
| `test_real_file_read` | `E2E=1` | Reads 67MB file successfully |
|
||||
| `test_memory_usage` | `E2E=1` | Documents but doesn't fail on high RAM |
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure Code
|
||||
|
||||
### Base Test Class (Python 2.7 Compatible)
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import unittest
|
||||
import subprocess
|
||||
import tempfile
|
||||
import shutil
|
||||
import os
|
||||
import time
|
||||
import threading
|
||||
|
||||
class BeetFSTestCase(unittest.TestCase):
|
||||
"""Base class for beetfs e2e tests - Python 2.7 compatible"""
|
||||
|
||||
MOUNT_TIMEOUT = 30 # seconds
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
"""Check FUSE availability"""
|
||||
try:
|
||||
with open(os.devnull, 'w') as devnull:
|
||||
subprocess.check_call(['which', 'fusermount'],
|
||||
stdout=devnull, stderr=devnull)
|
||||
except subprocess.CalledProcessError:
|
||||
raise unittest.SkipTest("fusermount not available")
|
||||
|
||||
def setUp(self):
|
||||
self.mount_dir = tempfile.mkdtemp(prefix='beetfs_test_')
|
||||
self.fs_process = None
|
||||
|
||||
def mount_beetfs(self, library_path=None):
|
||||
"""Mount beetfs in background with timeout"""
|
||||
cmd = ['python', '-c',
|
||||
'from beetsplug.beetFs import mount; mount()']
|
||||
# Add mount point and other args as needed
|
||||
|
||||
self.fs_process = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=open(os.devnull, 'w'),
|
||||
stderr=subprocess.STDOUT
|
||||
)
|
||||
|
||||
# Python 2.7 timeout workaround
|
||||
timer = threading.Timer(self.MOUNT_TIMEOUT, self._timeout_kill)
|
||||
timer.start()
|
||||
|
||||
try:
|
||||
self._wait_for_mount()
|
||||
finally:
|
||||
timer.cancel()
|
||||
|
||||
def _timeout_kill(self):
|
||||
if self.fs_process and self.fs_process.poll() is None:
|
||||
self.fs_process.kill()
|
||||
|
||||
def _wait_for_mount(self):
|
||||
"""Wait for filesystem to be mounted"""
|
||||
start = time.time()
|
||||
while time.time() - start < self.MOUNT_TIMEOUT:
|
||||
if os.path.ismount(self.mount_dir):
|
||||
return
|
||||
if self.fs_process.poll() is not None:
|
||||
self.fail("Filesystem process terminated prematurely")
|
||||
time.sleep(0.1)
|
||||
self.fail("Mount timeout after {} seconds".format(self.MOUNT_TIMEOUT))
|
||||
|
||||
def tearDown(self):
|
||||
"""Cleanup: unmount and kill process"""
|
||||
if self.fs_process:
|
||||
with open(os.devnull, 'w') as devnull:
|
||||
subprocess.call(['fusermount', '-z', '-u', self.mount_dir],
|
||||
stdout=devnull, stderr=devnull)
|
||||
|
||||
self.fs_process.terminate()
|
||||
|
||||
# Wait for termination (Py2.7 compatible)
|
||||
start = time.time()
|
||||
while time.time() - start < 5:
|
||||
if self.fs_process.poll() is not None:
|
||||
break
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
self.fs_process.kill()
|
||||
|
||||
shutil.rmtree(self.mount_dir, ignore_errors=True)
|
||||
```
|
||||
|
||||
### Synthetic FLAC Generator
|
||||
|
||||
```python
|
||||
# tests/conftest.py (continued)
|
||||
import subprocess
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
def create_synthetic_flac(duration_sec=5, artist="Test Artist",
|
||||
title="Test Track", album="Test Album"):
|
||||
"""Create minimal FLAC with known metadata (~500KB for 5s silence)"""
|
||||
wav_fd, wav_path = tempfile.mkstemp(suffix='.wav')
|
||||
os.close(wav_fd)
|
||||
flac_path = wav_path.replace('.wav', '.flac')
|
||||
|
||||
try:
|
||||
# Generate silence WAV
|
||||
subprocess.check_call([
|
||||
'ffmpeg', '-f', 'lavfi', '-i',
|
||||
'anullsrc=r=44100:cl=stereo', '-t', str(duration_sec),
|
||||
'-y', wav_path
|
||||
], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT)
|
||||
|
||||
# Convert to FLAC with metadata
|
||||
subprocess.check_call([
|
||||
'flac', '--best',
|
||||
'-T', 'ARTIST={}'.format(artist),
|
||||
'-T', 'TITLE={}'.format(title),
|
||||
'-T', 'ALBUM={}'.format(album),
|
||||
'-o', flac_path, wav_path
|
||||
], stdout=open(os.devnull, 'w'), stderr=subprocess.STDOUT)
|
||||
|
||||
return flac_path
|
||||
finally:
|
||||
if os.path.exists(wav_path):
|
||||
os.unlink(wav_path)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies to Add to flake.nix
|
||||
|
||||
```nix
|
||||
# In devShell buildInputs, add:
|
||||
pkgs.ffmpeg # For synthetic FLAC generation
|
||||
pkgs.flac # For FLAC encoding
|
||||
|
||||
# pythonEnv already has mutagen for verification
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| Memory explosion | High | Use 5-10MB synthetic FLACs, skip 650MB tests by default |
|
||||
| Nested methods bug | Critical | Tests will expose; fix required before other tests pass |
|
||||
| Python 2.7 EOL | Medium | Nix provides isolated environment |
|
||||
| Global state pollution | Medium | Fresh subprocess per test |
|
||||
| FUSE permissions | Low | Run as regular user, skip privileged tests |
|
||||
| Concurrent access | Low | Single-threaded mode, sequential tests |
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **All smoke tests pass** - beetfs mounts and unmounts cleanly
|
||||
2. **Nested bug exposed and fixed** - All FUSE methods callable
|
||||
3. **Metadata overlay verified** - Reads return DB metadata, not file metadata
|
||||
4. **Writes update DB** - Metadata changes persist
|
||||
5. **Errors handled gracefully** - Correct errno for unsupported ops
|
||||
6. **No crashes on edge cases** - Unicode, special chars, concurrent access
|
||||
|
||||
---
|
||||
|
||||
## Findings from Test Execution
|
||||
|
||||
### Bug #1: Nested Methods (CRITICAL)
|
||||
|
||||
**Location**: `beetFs.py` lines 758-1144
|
||||
|
||||
**Problem**: All FUSE operation methods are indented inside the `access()` method, making them local functions instead of class methods.
|
||||
|
||||
**Evidence**:
|
||||
```python
|
||||
def access(self, path, flags): # Line 723 - correct class method
|
||||
...
|
||||
return 0
|
||||
|
||||
def readdir(self, path, ...): # Line 931 - WRONG! Nested inside access()
|
||||
...
|
||||
def open(self, path, flags): # Line 988 - Also nested
|
||||
...
|
||||
def read(self, path, ...): # Line 1077 - Also nested
|
||||
...
|
||||
```
|
||||
|
||||
**Symptom**: `os.listdir()` returns `OSError: [Errno 38] Function not implemented`
|
||||
|
||||
**Fix Required**: Dedent lines 758-1144 by 8 spaces to make them class methods.
|
||||
|
||||
### Bug #2: Directory Tree Building
|
||||
|
||||
**Location**: `beetFs.py` lines 403-414 (`FSNode.getnode()` and `FSNode.adddir()`)
|
||||
|
||||
**Problem**: When adding files to the directory structure, the code assumes parent directories already exist.
|
||||
|
||||
**Evidence**:
|
||||
```
|
||||
KeyError: u'Test Artist'
|
||||
File "beetFs.py", line 403, in getnode
|
||||
return self.getnode(elements, root=root.dirs[topdir])
|
||||
```
|
||||
|
||||
**Symptom**: Mount fails when library contains tracks.
|
||||
|
||||
### Bug #3: Unmount Not Clean
|
||||
|
||||
**Problem**: After unmounting, `os.path.ismount()` still returns `True`.
|
||||
|
||||
**Likely Cause**: FUSE process not terminating properly, or lazy unmount not completing.
|
||||
|
||||
---
|
||||
|
||||
## Notes from Oracle Review
|
||||
|
||||
1. **MP3 is not "readonly"** - metadata overlay is disabled (`bound=0`), but reads still work
|
||||
2. **Write returns None for MP3** - no explicit return in MP3 path (falls through)
|
||||
3. **Path format is hardcoded** - tests must match `$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format`
|
||||
4. **basestring vs str** - use `isinstance(x, basestring)` for Py2.7 string checks
|
||||
5. **Global variables** - `library`, `directory_structure` must be reset between tests (use subprocesses)
|
||||
@@ -1,249 +0,0 @@
|
||||
# beetfs Feature Set
|
||||
|
||||
## Overview
|
||||
|
||||
beetfs is a FUSE filesystem plugin for [beets](https://beets.io/) that presents your music library as a virtual filesystem organized by metadata. Files appear with paths derived from their database metadata, and reading file headers returns metadata from the beets database rather than the actual file tags.
|
||||
|
||||
**Author**: Martin Eve (2010)
|
||||
**License**: GPLv3
|
||||
**Python**: 2.7 (uses fuse-python)
|
||||
|
||||
## Core Features
|
||||
|
||||
### 1. Virtual Metadata-Based Directory Structure
|
||||
|
||||
Files are presented in a configurable path format based on beets database fields:
|
||||
|
||||
```
|
||||
$artist/$album ($year) [$format_upper]/$track - $artist - $title.$format
|
||||
```
|
||||
|
||||
**Example**:
|
||||
```
|
||||
/mnt/beetfs/
|
||||
├── Metallica/
|
||||
│ └── 72 Seasons (2023) [FLAC]/
|
||||
│ ├── 01 - Metallica - 72 Seasons.flac
|
||||
│ ├── 02 - Metallica - Shadows Follow.flac
|
||||
│ └── ...
|
||||
├── Pink Floyd/
|
||||
│ └── The Dark Side of the Moon (1973) [FLAC]/
|
||||
│ └── ...
|
||||
```
|
||||
|
||||
**Available template variables**:
|
||||
- `$artist`, `$album`, `$title`, `$genre`, `$composer`, `$grouping`
|
||||
- `$year`, `$month`, `$day`
|
||||
- `$track`, `$tracktotal`, `$disc`, `$disctotal`
|
||||
- `$format`, `$format_upper` (file extension)
|
||||
- `$lyrics`, `$comments`, `$bpm`, `$comp`
|
||||
|
||||
### 2. Metadata Overlay (Read)
|
||||
|
||||
When you read a file through beetfs, the **metadata header is synthesized from the beets database**, not read from the actual file on disk.
|
||||
|
||||
**How it works**:
|
||||
1. Open file → beetfs reads the real file from disk
|
||||
2. Parse the audio format header (FLAC/MP3)
|
||||
3. Replace metadata fields with values from beets database
|
||||
4. Return synthesized header + original audio data
|
||||
|
||||
**Supported fields for overlay**:
|
||||
- `title`, `artist`, `album`, `genre` (FLAC only currently)
|
||||
|
||||
**Use case**: Your files may have inconsistent or wrong tags, but beetfs presents them with the corrected metadata from your beets library.
|
||||
|
||||
### 3. Metadata Passthrough (Write)
|
||||
|
||||
When you write to file headers through beetfs, the **changes are saved to the beets database**, not to the actual file.
|
||||
|
||||
**How it works**:
|
||||
1. Application writes new metadata to file header region
|
||||
2. beetfs intercepts the write
|
||||
3. Parses the new metadata values
|
||||
4. Updates the beets database (`lib.store()`, `lib.save()`)
|
||||
5. Regenerates the synthesized header
|
||||
|
||||
**Result**: Tag editors (Picard, Kid3, etc.) can edit metadata through beetfs, and changes persist in the beets database without modifying the original files.
|
||||
|
||||
### 4. Format Support
|
||||
|
||||
| Format | Read | Metadata Overlay | Write to DB |
|
||||
|--------|------|------------------|-------------|
|
||||
| FLAC | ✅ | ✅ Full | ✅ |
|
||||
| MP3 | ✅ | ❌ Disabled | ❌ |
|
||||
| Other | ❌ | ❌ | ❌ |
|
||||
|
||||
**FLAC Implementation**:
|
||||
- Uses `InterpolatedFLAC` class extending mutagen
|
||||
- Reconstructs Vorbis comment block with DB values
|
||||
- Preserves audio data and other metadata blocks
|
||||
|
||||
**MP3 Implementation**:
|
||||
- Passthrough only (no interpolation)
|
||||
- `self.bound = 0` disables header replacement
|
||||
|
||||
### 5. File Caching
|
||||
|
||||
Open files are cached in `FileHandler` objects:
|
||||
|
||||
- First open: Load entire file into memory, parse headers
|
||||
- Subsequent opens: Reuse cached `FileHandler`
|
||||
- Reference counting for multiple opens
|
||||
- Release when reference count reaches zero
|
||||
|
||||
**Memory impact**: Each open file consumes ~filesize RAM.
|
||||
|
||||
## FUSE Operations
|
||||
|
||||
### Implemented (Functional)
|
||||
|
||||
| Operation | Description |
|
||||
|-----------|-------------|
|
||||
| `getattr` | File/directory stat (size, mode, timestamps) |
|
||||
| `access` | Permission checking |
|
||||
| `opendir` | Open directory for listing |
|
||||
| `readdir` | List directory contents |
|
||||
| `releasedir` | Close directory |
|
||||
| `open` | Open file for reading/writing |
|
||||
| `read` | Read file contents |
|
||||
| `write` | Write to file (header region only) |
|
||||
| `release` | Close file |
|
||||
| `fgetattr` | Stat with file handle |
|
||||
| `statfs` | Filesystem statistics |
|
||||
|
||||
### Not Implemented (Return EOPNOTSUPP)
|
||||
|
||||
| Operation | Reason |
|
||||
|-----------|--------|
|
||||
| `create` | Read-only structure |
|
||||
| `mknod` | Read-only structure |
|
||||
| `mkdir` | Read-only structure |
|
||||
| `unlink` | Read-only structure |
|
||||
| `rmdir` | Read-only structure |
|
||||
| `symlink` | Not needed |
|
||||
| `link` | Not needed |
|
||||
| `rename` | Would break DB consistency |
|
||||
| `chmod` | Metadata-only FS |
|
||||
| `chown` | Metadata-only FS |
|
||||
| `truncate` | Would corrupt audio |
|
||||
| `utime` | Metadata-only FS |
|
||||
|
||||
## Usage
|
||||
|
||||
### Mount
|
||||
|
||||
```bash
|
||||
beet mount /mnt/beetfs
|
||||
```
|
||||
|
||||
### Unmount
|
||||
|
||||
```bash
|
||||
fusermount -u /mnt/beetfs
|
||||
```
|
||||
|
||||
### Example Session
|
||||
|
||||
```bash
|
||||
# Mount the filesystem
|
||||
beet mount /mnt/music
|
||||
|
||||
# Browse by artist
|
||||
ls /mnt/music/
|
||||
# Metallica/ Pink Floyd/ The Beatles/ ...
|
||||
|
||||
# List an album
|
||||
ls "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/"
|
||||
# 01 - Metallica - 72 Seasons.flac
|
||||
# 02 - Metallica - Shadows Follow.flac
|
||||
# ...
|
||||
|
||||
# Play through any music player
|
||||
mpv "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/01 - Metallica - 72 Seasons.flac"
|
||||
|
||||
# Edit tags (changes go to beets DB)
|
||||
kid3 "/mnt/music/Metallica/72 Seasons (2023) [FLAC]/"
|
||||
|
||||
# Unmount
|
||||
fusermount -u /mnt/music
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ User Applications │
|
||||
│ (mpv, Rhythmbox, Kid3, etc.) │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│ POSIX calls (open, read, write)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Linux Kernel │
|
||||
│ FUSE module │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│ /dev/fuse
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ beetfs │
|
||||
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
|
||||
│ │ FSNode Tree │ │ FileHandler │ │ InterpolatedFLAC │ │
|
||||
│ │ (in-memory) │ │ (cache) │ │ (header synth) │ │
|
||||
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
|
||||
└────────┬────────────────┬───────────────────┬───────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||
│ Beets DB │ │ Real Files │ │ Mutagen │
|
||||
│ (SQLite) │ │ (on disk) │ │ (parsing) │
|
||||
└─────────────┘ └─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
### Current Bugs (Non-Functional)
|
||||
|
||||
1. **Nested Methods Bug**: Lines 758-1144 are indented inside `access()`, making FUSE operations unreachable
|
||||
2. **Directory Tree Bug**: `FSNode.adddir()` crashes when building tree for non-empty library
|
||||
|
||||
### Design Limitations
|
||||
|
||||
1. **Memory Usage**: Entire file loaded into RAM on open
|
||||
2. **Mount Time**: O(N) - loads all library items at mount
|
||||
3. **No Lazy Loading**: Full directory tree built upfront
|
||||
4. **Single Format**: Only FLAC has full metadata overlay
|
||||
5. **No Real File Modification**: Writes only update DB, not actual files
|
||||
6. **Python 2.7 GIL**: Single-threaded performance
|
||||
|
||||
### Not Supported
|
||||
|
||||
- Creating/deleting files or directories
|
||||
- Moving/renaming files
|
||||
- Modifying audio content
|
||||
- Album art / embedded images
|
||||
- Multi-value tags
|
||||
- Non-ASCII in some edge cases
|
||||
|
||||
## Configuration
|
||||
|
||||
Currently hardcoded. Potential configuration points:
|
||||
|
||||
| Setting | Current Value | Description |
|
||||
|---------|---------------|-------------|
|
||||
| `PATH_FORMAT` | `$artist/$album ($year)...` | Directory structure template |
|
||||
| `METADATA_RW_FIELDS` | 17 fields | Fields available for read/write |
|
||||
| Caching | Always on | FileHandler caching behavior |
|
||||
| Threading | Disabled | `multithreaded = 0` |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Python 2.7
|
||||
- fuse-python
|
||||
- beets 1.4.x
|
||||
- mutagen (FLAC/MP3 parsing)
|
||||
|
||||
## See Also
|
||||
|
||||
- [e2e-test-plan.md](e2e-test-plan.md) - Test strategy and bug documentation
|
||||
- [benchmark-plan.md](benchmark-plan.md) - Performance measurement methodology
|
||||
- [benchmark-results.md](benchmark-results.md) - Current benchmark status
|
||||
@@ -1,459 +0,0 @@
|
||||
# beetfs Modernization Guide
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
### Technical Debt
|
||||
|
||||
| Issue | Severity | Location |
|
||||
|-------|----------|----------|
|
||||
| Python 2 syntax | 🔴 Critical | Throughout |
|
||||
| fuse-python (deprecated) | 🔴 Critical | Lines 25, 51 |
|
||||
| `basestring` usage | 🔴 Critical | Line 89 |
|
||||
| `reduce` without import | 🟡 Medium | Line 197 |
|
||||
| `0755` octal syntax | 🟡 Medium | Lines 654, 700 |
|
||||
| `print` as statement | 🟡 Medium | N/A (not used) |
|
||||
| `except Exception, e` | 🔴 Critical | Line 181 |
|
||||
| Long integers (`0L`) | 🟡 Medium | Line 197 |
|
||||
| Global state | 🟡 Medium | Lines 125-140 |
|
||||
| Memory-heavy design | 🟡 Medium | Line 481 |
|
||||
|
||||
### Dependencies to Update
|
||||
|
||||
| Original | Replacement | Notes |
|
||||
|----------|-------------|-------|
|
||||
| `fuse-python` | `pyfuse3` or `llfuse` | Modern FUSE bindings |
|
||||
| `beets` (old API) | `beets >= 1.6` | Check API compatibility |
|
||||
| `mutagen` | `mutagen >= 1.45` | Mostly compatible |
|
||||
| Python 2.7 | Python 3.9+ | Full migration needed |
|
||||
|
||||
---
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Phase 1: Python 3 Compatibility
|
||||
|
||||
#### 1.1 Fix Syntax Issues
|
||||
|
||||
```python
|
||||
# BEFORE (Python 2)
|
||||
except fuse.FuseError, e:
|
||||
log.error(str(e))
|
||||
|
||||
# AFTER (Python 3)
|
||||
except fuse.FuseError as e:
|
||||
log.error(str(e))
|
||||
```
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
if isinstance(value, basestring):
|
||||
|
||||
# AFTER
|
||||
if isinstance(value, str):
|
||||
```
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
return reduce(lambda a, b: (a << 8) + ord(b), string, 0L)
|
||||
|
||||
# AFTER
|
||||
from functools import reduce
|
||||
return reduce(lambda a, b: (a << 8) + b, string, 0)
|
||||
```
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
mode = stat.S_IFDIR | 0755
|
||||
|
||||
# AFTER
|
||||
mode = stat.S_IFDIR | 0o755
|
||||
```
|
||||
|
||||
#### 1.2 Fix String/Bytes Handling
|
||||
|
||||
```python
|
||||
# BEFORE - implicit string/bytes mixing
|
||||
self.header = self.inf.get_header(self.real_path)
|
||||
return self.header[offset:offset+size]
|
||||
|
||||
# AFTER - explicit bytes handling
|
||||
self.header: bytes = self.inf.get_header(self.real_path)
|
||||
return self.header[offset:offset+size]
|
||||
```
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
self.item.title = str(self.inf["title"][0]).encode('utf-8')
|
||||
|
||||
# AFTER
|
||||
self.item.title = self.inf["title"][0] # Already str in Python 3
|
||||
```
|
||||
|
||||
#### 1.3 Fix Dictionary Methods
|
||||
|
||||
```python
|
||||
# BEFORE
|
||||
return node.dirs.keys()
|
||||
|
||||
# AFTER
|
||||
return list(node.dirs.keys()) # If list is needed
|
||||
# or just
|
||||
return node.dirs.keys() # If iteration is sufficient
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: FUSE Library Migration
|
||||
|
||||
#### Option A: pyfuse3 (Recommended)
|
||||
|
||||
Modern, async-capable FUSE bindings.
|
||||
|
||||
```python
|
||||
# BEFORE (fuse-python)
|
||||
import fuse
|
||||
fuse.fuse_python_api = (0, 2)
|
||||
|
||||
class beetFileSystem(fuse.Fuse):
|
||||
def read(self, path, size, offset):
|
||||
return data
|
||||
|
||||
# AFTER (pyfuse3)
|
||||
import pyfuse3
|
||||
import trio
|
||||
|
||||
class BeetFS(pyfuse3.Operations):
|
||||
async def read(self, fh, offset, size):
|
||||
return data
|
||||
|
||||
async def main():
|
||||
fs = BeetFS()
|
||||
fuse_options = set(pyfuse3.default_options)
|
||||
fuse_options.add('fsname=beetfs')
|
||||
pyfuse3.init(fs, mountpoint, fuse_options)
|
||||
try:
|
||||
await pyfuse3.main()
|
||||
finally:
|
||||
pyfuse3.close()
|
||||
|
||||
trio.run(main)
|
||||
```
|
||||
|
||||
**Key Differences**:
|
||||
| fuse-python | pyfuse3 |
|
||||
|-------------|---------|
|
||||
| `read(path, size, offset)` | `read(fh, offset, size)` |
|
||||
| Synchronous | Async (trio) |
|
||||
| Return data directly | Return bytes |
|
||||
| Path-based | File handle based |
|
||||
|
||||
#### Option B: llfuse (Alternative)
|
||||
|
||||
Lower-level, synchronous.
|
||||
|
||||
```python
|
||||
import llfuse
|
||||
|
||||
class BeetFS(llfuse.Operations):
|
||||
def read(self, fh, offset, size):
|
||||
return data
|
||||
|
||||
def main():
|
||||
fs = BeetFS()
|
||||
llfuse.init(fs, mountpoint, options)
|
||||
try:
|
||||
llfuse.main()
|
||||
finally:
|
||||
llfuse.close()
|
||||
```
|
||||
|
||||
#### Option C: fusepy (Simple)
|
||||
|
||||
Simple wrapper, but less maintained.
|
||||
|
||||
```python
|
||||
from fuse import FUSE, Operations
|
||||
|
||||
class BeetFS(Operations):
|
||||
def read(self, path, size, offset, fh):
|
||||
return data
|
||||
|
||||
FUSE(BeetFS(), mountpoint, foreground=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Architecture Improvements
|
||||
|
||||
#### 3.1 Remove Global State
|
||||
|
||||
```python
|
||||
# BEFORE - Global variables
|
||||
global structure_split
|
||||
global structure_depth
|
||||
global library
|
||||
global directory_structure
|
||||
|
||||
# AFTER - Instance variables
|
||||
class BeetFS:
|
||||
def __init__(self, lib: Library, path_format: str):
|
||||
self.lib = lib
|
||||
self.path_format = path_format
|
||||
self.structure_split = path_format.split("/")
|
||||
self.structure_depth = len(self.structure_split)
|
||||
self.directory_structure = FSNode({}, {})
|
||||
self._build_tree()
|
||||
```
|
||||
|
||||
#### 3.2 Reduce Memory Usage
|
||||
|
||||
```python
|
||||
# BEFORE - Load entire audio into memory
|
||||
self.music_data = self.file_object.read() # Could be 100MB+
|
||||
|
||||
# AFTER - Lazy loading with mmap or seek
|
||||
class FileHandler:
|
||||
def __init__(self, path, lib):
|
||||
self.real_path = self._resolve_path(path)
|
||||
self.file_object = open(self.real_path, 'rb')
|
||||
self._header = None # Lazy load
|
||||
self._music_offset = None
|
||||
|
||||
@property
|
||||
def header(self) -> bytes:
|
||||
if self._header is None:
|
||||
self._header = self._generate_header()
|
||||
return self._header
|
||||
|
||||
def read(self, size: int, offset: int) -> bytes:
|
||||
if offset < len(self.header):
|
||||
# Header region - return from generated header
|
||||
if offset + size <= len(self.header):
|
||||
return self.header[offset:offset+size]
|
||||
else:
|
||||
# Span header and audio
|
||||
header_part = self.header[offset:]
|
||||
audio_offset = 0
|
||||
audio_size = size - len(header_part)
|
||||
audio_part = self._read_audio(audio_offset, audio_size)
|
||||
return header_part + audio_part
|
||||
else:
|
||||
# Audio region - read directly from file
|
||||
audio_offset = offset - len(self.header)
|
||||
return self._read_audio(audio_offset, size)
|
||||
|
||||
def _read_audio(self, offset: int, size: int) -> bytes:
|
||||
self.file_object.seek(self._music_offset + offset)
|
||||
return self.file_object.read(size)
|
||||
```
|
||||
|
||||
#### 3.3 Add Type Hints
|
||||
|
||||
```python
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from pathlib import Path
|
||||
|
||||
class FSNode:
|
||||
def __init__(self, dirs: Dict[str, 'FSNode'], files: Dict[str, int]):
|
||||
self.dirs: Dict[str, FSNode] = dirs
|
||||
self.files: Dict[str, int] = files
|
||||
|
||||
def getnode(self, elements: List[str], root: Optional['FSNode'] = None) -> 'FSNode':
|
||||
...
|
||||
|
||||
def addfile(self, elements: List[str], filename: str, item_id: int) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
#### 3.4 Add MP3 Support
|
||||
|
||||
```python
|
||||
class FileHandler:
|
||||
def __init__(self, path: str, lib: Library):
|
||||
self.format = Path(path).suffix[1:].lower()
|
||||
|
||||
if self.format == "flac":
|
||||
self._handler = FLACHandler(self.real_path, self.item)
|
||||
elif self.format == "mp3":
|
||||
self._handler = MP3Handler(self.real_path, self.item)
|
||||
elif self.format in ("ogg", "opus"):
|
||||
self._handler = OggHandler(self.real_path, self.item)
|
||||
else:
|
||||
raise UnsupportedFormatError(f"Format {self.format} not supported")
|
||||
|
||||
class FLACHandler:
|
||||
def generate_header(self, item: Item) -> bytes:
|
||||
inf = InterpolatedFLAC(self.file_data)
|
||||
inf["title"] = item.title
|
||||
inf["album"] = item.album
|
||||
inf["artist"] = item.artist
|
||||
inf["genre"] = item.genre
|
||||
return inf.get_header()
|
||||
|
||||
class MP3Handler:
|
||||
def generate_header(self, item: Item) -> bytes:
|
||||
# Implement ID3v2 header generation
|
||||
id3 = InterpolatedID3()
|
||||
id3.add(TIT2(encoding=3, text=item.title))
|
||||
id3.add(TPE1(encoding=3, text=item.artist))
|
||||
id3.add(TALB(encoding=3, text=item.album))
|
||||
id3.add(TCON(encoding=3, text=item.genre))
|
||||
|
||||
# Calculate padding to match original header size
|
||||
...
|
||||
return id3.render()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Testing
|
||||
|
||||
#### 4.1 Unit Tests
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from beetfs import FSNode, FileHandler
|
||||
|
||||
class TestFSNode:
|
||||
def test_adddir(self):
|
||||
root = FSNode({}, {})
|
||||
root.adddir([], "Artist")
|
||||
assert "Artist" in root.dirs
|
||||
|
||||
def test_addfile(self):
|
||||
root = FSNode({}, {})
|
||||
root.adddir([], "Artist")
|
||||
root.addfile(["Artist"], "track.flac", 42)
|
||||
assert root.dirs["Artist"].files["track.flac"] == 42
|
||||
|
||||
def test_getnode(self):
|
||||
root = FSNode({}, {})
|
||||
root.adddir([], "Artist")
|
||||
root.adddir(["Artist"], "Album")
|
||||
node = root.getnode(["Artist", "Album"])
|
||||
assert node is not None
|
||||
|
||||
class TestFileHandler:
|
||||
def test_read_header(self, mock_flac_file, mock_beets_item):
|
||||
handler = FileHandler("/Artist/Album/track.flac", mock_lib)
|
||||
data = handler.read(100, 0)
|
||||
assert data.startswith(b"fLaC")
|
||||
|
||||
def test_read_audio(self, mock_flac_file, mock_beets_item):
|
||||
handler = FileHandler("/Artist/Album/track.flac", mock_lib)
|
||||
data = handler.read(100, handler.bound + 100)
|
||||
# Should be audio data from original file
|
||||
assert data == mock_flac_file.audio_data[100:200]
|
||||
```
|
||||
|
||||
#### 4.2 Integration Tests
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
class TestFUSEMount:
|
||||
def test_mount_unmount(self, beets_library):
|
||||
with tempfile.TemporaryDirectory() as mountpoint:
|
||||
# Mount
|
||||
proc = subprocess.Popen(
|
||||
["beet", "mount", mountpoint],
|
||||
stdout=subprocess.PIPE
|
||||
)
|
||||
time.sleep(1)
|
||||
|
||||
# Verify mount
|
||||
assert os.path.ismount(mountpoint)
|
||||
|
||||
# List files
|
||||
files = os.listdir(mountpoint)
|
||||
assert len(files) > 0
|
||||
|
||||
# Unmount
|
||||
subprocess.run(["fusermount", "-u", mountpoint])
|
||||
proc.wait()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 5: Standalone Mode (Optional)
|
||||
|
||||
Remove beets dependency for use as standalone metadata overlay.
|
||||
|
||||
```python
|
||||
class StandaloneFS:
|
||||
"""Metadata overlay without beets dependency."""
|
||||
|
||||
def __init__(self,
|
||||
source_dir: Path,
|
||||
metadata_db: Path,
|
||||
path_format: str):
|
||||
self.source_dir = source_dir
|
||||
self.db = sqlite3.connect(metadata_db)
|
||||
self.path_format = path_format
|
||||
self._build_tree()
|
||||
|
||||
def _build_tree(self):
|
||||
"""Build virtual tree from source directory and metadata DB."""
|
||||
for audio_file in self.source_dir.rglob("*.flac"):
|
||||
# Get metadata from DB or scan file
|
||||
metadata = self._get_metadata(audio_file)
|
||||
# Build virtual path from template
|
||||
virtual_path = self._format_path(metadata)
|
||||
# Add to tree
|
||||
self.directory_structure.addfile(
|
||||
virtual_path.parent.parts,
|
||||
virtual_path.name,
|
||||
str(audio_file) # Store actual path instead of ID
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommended Migration Order
|
||||
|
||||
```
|
||||
1. [ ] Fork and set up development environment
|
||||
2. [ ] Add type hints throughout (helps catch issues)
|
||||
3. [ ] Fix Python 3 syntax issues
|
||||
4. [ ] Replace fuse-python with pyfuse3/llfuse
|
||||
5. [ ] Add unit tests for FSNode and FileHandler
|
||||
6. [ ] Refactor global state to instance variables
|
||||
7. [ ] Implement lazy loading for audio data
|
||||
8. [ ] Add MP3 support
|
||||
9. [ ] Add integration tests
|
||||
10. [ ] Optional: Create standalone mode
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
| Phase | Effort | Risk |
|
||||
|-------|--------|------|
|
||||
| Phase 1 (Python 3) | 2-3 days | Low |
|
||||
| Phase 2 (FUSE migration) | 3-5 days | Medium |
|
||||
| Phase 3 (Architecture) | 3-5 days | Medium |
|
||||
| Phase 4 (Testing) | 2-3 days | Low |
|
||||
| Phase 5 (Standalone) | 3-5 days | Medium |
|
||||
| **Total** | **13-21 days** | |
|
||||
|
||||
---
|
||||
|
||||
## Alternative: Rewrite from Scratch
|
||||
|
||||
Given the age of the codebase, a rewrite might be more efficient:
|
||||
|
||||
**Pros of Rewrite**:
|
||||
- Clean architecture from start
|
||||
- Modern async design
|
||||
- Better memory management
|
||||
- Easier to test
|
||||
|
||||
**Cons of Rewrite**:
|
||||
- More initial effort
|
||||
- Risk of missing edge cases
|
||||
- Need to re-discover FLAC/ID3 intricacies
|
||||
|
||||
**Recommended Approach**: Start with Phase 1-2 to understand the code deeply, then decide whether to continue refactoring or rewrite.
|
||||
@@ -1,451 +0,0 @@
|
||||
# Rust Migration Analysis for beetfs
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Migrating beetfs from Python to Rust is **strongly recommended** based on research findings. Expected improvements:
|
||||
|
||||
| Metric | Python (Current) | Rust (Expected) | Improvement |
|
||||
|--------|------------------|-----------------|-------------|
|
||||
| **Memory per file** | ~280 bytes overhead | ~60 bytes | **4-5x reduction** |
|
||||
| **File open latency** | 200-500ms | 20-50ms | **10x faster** |
|
||||
| **Read latency** | 5-10ms | 0.5-2ms | **5-10x faster** |
|
||||
| **Concurrent opens** | ~1,000 (threading) | ~100,000+ (Tokio) | **100x more** |
|
||||
| **GC pauses** | 50-2200ms | 0ms | **Eliminated** |
|
||||
|
||||
---
|
||||
|
||||
## 1. Rust FUSE Ecosystem
|
||||
|
||||
### Recommended: **fuser**
|
||||
|
||||
| Attribute | Value |
|
||||
|-----------|-------|
|
||||
| **Downloads** | 3.2M+ |
|
||||
| **Maturity** | Production-ready |
|
||||
| **Platforms** | Linux, macOS, FreeBSD |
|
||||
| **Async** | Experimental (stable sync API) |
|
||||
| **Used by** | AWS Mountpoint for S3 |
|
||||
|
||||
**API Example:**
|
||||
```rust
|
||||
use fuser::{Filesystem, Request, ReplyData};
|
||||
|
||||
impl Filesystem for BeetFS {
|
||||
fn read(&self, _req: &Request, ino: u64, _fh: u64,
|
||||
offset: i64, size: u32, _flags: i32,
|
||||
_lock: Option<u64>, reply: ReplyData) {
|
||||
|
||||
let file = self.get_file(ino);
|
||||
|
||||
if offset < file.header_len {
|
||||
// Return metadata from database (interpolated)
|
||||
reply.data(&file.header[offset as usize..]);
|
||||
} else {
|
||||
// Return audio from original file (zero-copy via mmap)
|
||||
let audio_offset = offset - file.header_len;
|
||||
reply.data(&file.mmap[audio_offset as usize..]);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Alternatives
|
||||
|
||||
| Library | Async | Maturity | Best For |
|
||||
|---------|-------|----------|----------|
|
||||
| **fuser** | Experimental | ⭐⭐⭐⭐⭐ | General purpose |
|
||||
| **fuse3** | Native | ⭐⭐⭐⭐ | Async-heavy, Linux-only |
|
||||
| **polyfuse** | Native | ⭐⭐⭐ | Custom control flow |
|
||||
|
||||
---
|
||||
|
||||
## 2. Rust Audio Metadata: **lofty**
|
||||
|
||||
Full feature parity with Python's mutagen:
|
||||
|
||||
| Feature | mutagen (Python) | lofty (Rust) |
|
||||
|---------|------------------|--------------|
|
||||
| FLAC Vorbis Comments | ✅ | ✅ |
|
||||
| MP3 ID3v2 (all versions) | ✅ | ✅ |
|
||||
| OGG Vorbis Comments | ✅ | ✅ |
|
||||
| Opus metadata | ✅ | ✅ |
|
||||
| In-memory manipulation | ✅ | ✅ |
|
||||
| Header generation | ✅ | ✅ `dump_to()` |
|
||||
| Picture/artwork | ✅ | ✅ |
|
||||
|
||||
**API Comparison:**
|
||||
```python
|
||||
# Python mutagen
|
||||
audio = mutagen.File("song.flac")
|
||||
audio['artist'] = 'New Artist'
|
||||
audio['title'] = 'New Title'
|
||||
audio.save()
|
||||
```
|
||||
|
||||
```rust
|
||||
// Rust lofty
|
||||
let mut file = lofty::read_from_path("song.flac")?;
|
||||
let tag = file.primary_tag_mut().unwrap();
|
||||
tag.set_artist("New Artist".to_string());
|
||||
tag.set_title("New Title".to_string());
|
||||
tag.save_to_path("song.flac", WriteOptions::default())?;
|
||||
```
|
||||
|
||||
**Header Generation (Critical for beetfs):**
|
||||
```rust
|
||||
// Generate FLAC header with modified tags WITHOUT writing to file
|
||||
let mut buffer = Vec::new();
|
||||
tag.dump_to(&mut buffer, WriteOptions::default())?;
|
||||
// `buffer` contains serialized metadata header
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Memory Benefits
|
||||
|
||||
### Python Object Overhead
|
||||
|
||||
| Python Type | Size | Notes |
|
||||
|-------------|------|-------|
|
||||
| Empty dict | 232 bytes | Base overhead |
|
||||
| Dict entry | +184 bytes | Per key-value |
|
||||
| Empty string | 49 bytes | Base overhead |
|
||||
| Empty list | 56 bytes | Base overhead |
|
||||
| Small int | 28 bytes | Even for `0` |
|
||||
|
||||
**Current beetfs FileHandler (Python):**
|
||||
```
|
||||
self.path → str → 49 + len(path) bytes
|
||||
self.real_path → str → 49 + len(path) bytes
|
||||
self.item → dict → 232 + entries
|
||||
self.header → bytes → 33 + len(header)
|
||||
self.music_data → bytes → 33 + len(audio) ← CRITICAL: full file!
|
||||
self.inf → object → 100+ bytes
|
||||
─────────────────────────────────────────
|
||||
TOTAL: ~500 bytes + entire file in RAM
|
||||
```
|
||||
|
||||
### Rust Struct Efficiency
|
||||
|
||||
```rust
|
||||
struct FileHandler {
|
||||
path: PathBuf, // 24 bytes (ptr+len+cap)
|
||||
real_path: PathBuf, // 24 bytes
|
||||
item_id: u64, // 8 bytes
|
||||
header: Vec<u8>, // 24 bytes (ptr+len+cap) + header data
|
||||
mmap: Mmap, // 24 bytes (NO file data in RAM!)
|
||||
header_len: u64, // 8 bytes
|
||||
audio_offset: u64, // 8 bytes
|
||||
}
|
||||
// TOTAL: ~120 bytes + header only (audio via mmap)
|
||||
```
|
||||
|
||||
### Memory Comparison
|
||||
|
||||
| Scenario | Python | Rust | Savings |
|
||||
|----------|--------|------|---------|
|
||||
| 1 file (50MB) | ~50 MB | ~64 KB | **780x** |
|
||||
| 10 files (50MB each) | ~500 MB | ~640 KB | **780x** |
|
||||
| 100 files (50MB each) | ~5 GB | ~6.4 MB | **780x** |
|
||||
| Library scan (1000 files) | **OOM** | ~64 MB | ∞ |
|
||||
|
||||
**Key insight**: Rust can use memory-mapped files (`mmap`) to serve audio data with zero copies, eliminating the need to load files into RAM.
|
||||
|
||||
---
|
||||
|
||||
## 4. Latency Benefits
|
||||
|
||||
### Python FUSE Bottlenecks
|
||||
|
||||
1. **Dict-to-struct conversion**: Every FUSE callback requires converting Python dicts to C structs
|
||||
2. **GIL contention**: Single-threaded execution despite multi-core CPUs
|
||||
3. **GC pauses**: Stop-the-world pauses of 50-2200ms under load
|
||||
4. **Object allocation**: Creating Python objects for every I/O operation
|
||||
|
||||
### Rust FUSE Advantages
|
||||
|
||||
1. **Zero-cost abstractions**: No runtime overhead for type conversions
|
||||
2. **No GIL**: True parallelism across all cores
|
||||
3. **No GC**: Deterministic memory management, no pauses
|
||||
4. **Stack allocation**: Small objects allocated on stack, not heap
|
||||
|
||||
### Benchmark Data
|
||||
|
||||
| Operation | Python FUSE | Rust FUSE | Improvement |
|
||||
|-----------|-------------|-----------|-------------|
|
||||
| File stat | 5-10ms | 0.5-1ms | **10x** |
|
||||
| Small read | 5-10ms | 0.5-2ms | **5-10x** |
|
||||
| Large read | 115 MB/s | 260+ MB/s | **2-3x** |
|
||||
| Metadata lookup | 10ms | <1ms | **10x** |
|
||||
|
||||
### GC Pause Elimination
|
||||
|
||||
```
|
||||
Python GC Pauses (measured):
|
||||
├── P50: ~10ms
|
||||
├── P95: ~50ms
|
||||
├── P99: ~320ms
|
||||
└── Max: ~2200ms (!)
|
||||
|
||||
Rust (no GC):
|
||||
├── P50: ~0.5ms
|
||||
├── P95: ~1ms
|
||||
├── P99: ~2ms
|
||||
└── Max: ~5ms (deterministic)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Concurrency Benefits
|
||||
|
||||
### Python Threading Limitations
|
||||
|
||||
```python
|
||||
# Python (current beetfs)
|
||||
server.multithreaded = 0 # Single-threaded!
|
||||
|
||||
# Even with threading enabled:
|
||||
# - GIL prevents true parallelism
|
||||
# - ~8MB per thread
|
||||
# - OS limits: ~1000-2000 threads max
|
||||
# - Context switch: 1-10μs (kernel)
|
||||
```
|
||||
|
||||
### Rust Async (Tokio)
|
||||
|
||||
```rust
|
||||
// Rust with Tokio
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
// Can handle 100K+ concurrent operations
|
||||
// - ~2KB per task (4000x less than thread)
|
||||
// - Work-stealing scheduler
|
||||
// - Context switch: ~10ns (userspace)
|
||||
}
|
||||
```
|
||||
|
||||
| Metric | Python Threading | Rust Tokio |
|
||||
|--------|------------------|------------|
|
||||
| Memory per task | 8 MB | 2 KB |
|
||||
| Max concurrent | ~1,000 | ~100,000+ |
|
||||
| Context switch | 1-10μs | ~10ns |
|
||||
| Parallelism | Blocked by GIL | True multi-core |
|
||||
|
||||
---
|
||||
|
||||
## 6. Zero-Copy I/O
|
||||
|
||||
### Python (Current)
|
||||
|
||||
```python
|
||||
# Every read copies data through Python:
|
||||
self.file_object.read() # syscall → kernel buffer
|
||||
# kernel buffer → Python bytes object
|
||||
# Python bytes → FUSE reply buffer
|
||||
# = 2-3 copies per read
|
||||
```
|
||||
|
||||
### Rust (Proposed)
|
||||
|
||||
```rust
|
||||
// Memory-mapped file + zero-copy reply:
|
||||
let mmap = unsafe { MmapOptions::new().map(&file)? };
|
||||
|
||||
fn read(&self, ..., reply: ReplyData) {
|
||||
// Direct slice from mmap → FUSE kernel
|
||||
reply.data(&self.mmap[offset..offset+size]);
|
||||
// = 0 copies (kernel reads directly from mapped pages)
|
||||
}
|
||||
```
|
||||
|
||||
### I/O Comparison
|
||||
|
||||
| Scenario | Python | Rust | Benefit |
|
||||
|----------|--------|------|---------|
|
||||
| Serve 50MB file | 50MB copied to RAM | 0 bytes copied | **50MB saved** |
|
||||
| 100 concurrent reads | 5GB buffers | ~0 (shared mmap) | **5GB saved** |
|
||||
| Throughput | 115 MB/s | 260+ MB/s | **2.3x faster** |
|
||||
|
||||
---
|
||||
|
||||
## 7. Real-World Migration Results
|
||||
|
||||
### Case Studies
|
||||
|
||||
| Project | Metric | Python | Rust | Improvement |
|
||||
|---------|--------|--------|------|-------------|
|
||||
| API Service | Response time | 200ms | 8ms | **96% faster** |
|
||||
| Data Pipeline | Processing | 3 hours | 4.5 min | **40x faster** |
|
||||
| Web Backend | Memory | 1.2 GB | 180 MB | **85% less** |
|
||||
| Trajectory Lib | Compute | baseline | 10x faster | **10x** |
|
||||
|
||||
### AWS Mountpoint for S3
|
||||
|
||||
- Built on **fuser** (Rust FUSE)
|
||||
- Handles **terabits/sec** aggregate throughput
|
||||
- Production-ready since 2024
|
||||
- Validates Rust FUSE at scale
|
||||
|
||||
---
|
||||
|
||||
## 8. Migration Architecture
|
||||
|
||||
### Proposed Rust beetfs Structure
|
||||
|
||||
```
|
||||
beetfs-rs/
|
||||
├── Cargo.toml
|
||||
├── src/
|
||||
│ ├── main.rs # Entry point, mount logic
|
||||
│ ├── lib.rs # Library root
|
||||
│ ├── fs/
|
||||
│ │ ├── mod.rs # FUSE filesystem impl
|
||||
│ │ ├── tree.rs # Virtual directory tree (FSNode equivalent)
|
||||
│ │ ├── file.rs # File handler with mmap
|
||||
│ │ └── stat.rs # File attributes
|
||||
│ ├── metadata/
|
||||
│ │ ├── mod.rs # Metadata overlay logic
|
||||
│ │ ├── flac.rs # FLAC header generation (using lofty)
|
||||
│ │ ├── mp3.rs # MP3 ID3 header generation
|
||||
│ │ └── db.rs # Database interface (SQLite or custom)
|
||||
│ └── config.rs # Configuration (path templates, etc.)
|
||||
└── tests/
|
||||
├── fs_tests.rs
|
||||
└── metadata_tests.rs
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
```rust
|
||||
// Virtual directory tree (equivalent to FSNode)
|
||||
pub struct VirtualTree {
|
||||
root: Arc<RwLock<DirNode>>,
|
||||
}
|
||||
|
||||
pub struct DirNode {
|
||||
dirs: HashMap<OsString, Arc<RwLock<DirNode>>>,
|
||||
files: HashMap<OsString, FileEntry>,
|
||||
}
|
||||
|
||||
pub struct FileEntry {
|
||||
inode: u64,
|
||||
real_path: PathBuf,
|
||||
metadata_id: i64, // Database reference
|
||||
}
|
||||
|
||||
// File handler with memory-mapped audio
|
||||
pub struct OpenFile {
|
||||
header: Vec<u8>, // Generated header with DB metadata
|
||||
header_len: usize,
|
||||
mmap: Mmap, // Memory-mapped original file
|
||||
audio_offset: usize, // Where audio starts in original
|
||||
}
|
||||
|
||||
impl OpenFile {
|
||||
pub fn read(&self, offset: usize, size: usize) -> &[u8] {
|
||||
if offset < self.header_len {
|
||||
// Return from generated header (DB metadata)
|
||||
&self.header[offset..min(offset + size, self.header_len)]
|
||||
} else {
|
||||
// Return from mmap (original audio, zero-copy)
|
||||
let audio_off = offset - self.header_len + self.audio_offset;
|
||||
&self.mmap[audio_off..audio_off + size]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Migration Effort Estimate
|
||||
|
||||
### Timeline
|
||||
|
||||
| Phase | Duration | Deliverable |
|
||||
|-------|----------|-------------|
|
||||
| **1. Prototype** | 1-2 weeks | Basic FUSE mount, read-only |
|
||||
| **2. Core features** | 2-3 weeks | Metadata overlay, FLAC support |
|
||||
| **3. Full parity** | 2-3 weeks | MP3, write support, all fields |
|
||||
| **4. Testing** | 1-2 weeks | Unit tests, integration tests |
|
||||
| **5. Optimization** | 1-2 weeks | mmap, async, benchmarking |
|
||||
|
||||
**Total: 7-12 weeks**
|
||||
|
||||
### Skill Requirements
|
||||
|
||||
- Rust fundamentals (ownership, borrowing, lifetimes)
|
||||
- FUSE protocol knowledge (from Python experience)
|
||||
- Audio metadata formats (FLAC, ID3)
|
||||
- Async Rust (Tokio) - optional for Phase 5
|
||||
|
||||
---
|
||||
|
||||
## 10. Risk Assessment
|
||||
|
||||
### Low Risk ✅
|
||||
|
||||
| Factor | Why Low Risk |
|
||||
|--------|--------------|
|
||||
| FUSE library | fuser is production-proven (AWS) |
|
||||
| Metadata library | lofty has full mutagen parity |
|
||||
| Core algorithm | Same logic, different language |
|
||||
| File format support | FLAC/MP3/OGG all supported |
|
||||
|
||||
### Medium Risk ⚠️
|
||||
|
||||
| Factor | Mitigation |
|
||||
|--------|------------|
|
||||
| Learning curve | Existing Rust experience helps |
|
||||
| Edge cases | Port Python tests to Rust |
|
||||
| Async complexity | Start with sync API, add async later |
|
||||
|
||||
### Benefits vs Effort
|
||||
|
||||
```
|
||||
Current Python Issues:
|
||||
├── Memory: OOM on library scan → Fixed by mmap
|
||||
├── Latency: 200-500ms file open → Fixed by zero-copy
|
||||
├── GC pauses: 50-2200ms → Eliminated
|
||||
├── Concurrency: single-threaded → Fixed by async
|
||||
└── MP3 support: disabled → Implemented properly
|
||||
|
||||
Migration Effort: 7-12 weeks
|
||||
Expected Lifetime: 5+ years
|
||||
ROI: Highly positive
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Recommendation
|
||||
|
||||
### ✅ **Proceed with Rust Migration**
|
||||
|
||||
**Justification:**
|
||||
1. **10x memory reduction** via mmap (eliminates OOM)
|
||||
2. **5-10x latency improvement** (eliminates blocking reads)
|
||||
3. **GC pauses eliminated** (deterministic performance)
|
||||
4. **100x concurrency** improvement (Tokio async)
|
||||
5. **Production-proven** ecosystem (fuser + lofty)
|
||||
6. **Reasonable effort** (7-12 weeks)
|
||||
|
||||
### Next Steps
|
||||
|
||||
1. **Set up Rust project** with fuser and lofty dependencies
|
||||
2. **Port FSNode** to Rust VirtualTree
|
||||
3. **Implement basic FUSE** operations (read, getattr, readdir)
|
||||
4. **Add metadata overlay** with lofty for FLAC
|
||||
5. **Add mmap** for zero-copy audio serving
|
||||
6. **Benchmark** against Python implementation
|
||||
7. **Add MP3/OGG** support
|
||||
8. **Add async** with Tokio (optional)
|
||||
|
||||
### Dependencies
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
fuser = "0.17"
|
||||
lofty = "0.21"
|
||||
memmap2 = "0.9"
|
||||
tokio = { version = "1", features = ["full"], optional = true }
|
||||
rusqlite = "0.31" # For beets DB compatibility
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,579 +0,0 @@
|
||||
# Metadata Enrichment (Standalone Mode): Design Doc
|
||||
|
||||
**Authors:** Sisyphus
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-18
|
||||
**Reviewers:** —
|
||||
**Approvers:** —
|
||||
**Document Link:** `docs/v2/plans/metadata-enrichment-standalone.md`
|
||||
**Prerequisites:** [architecture.md](../architecture.md), [week-12-external-metadata.md](week-12-external-metadata.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
When musicfs operates without the music-agregator orchestrator, it should
|
||||
still be able to enrich file metadata (genres, label, artwork URL, album
|
||||
type) by querying the metadata-agregator service directly. This document
|
||||
describes a **built-in metadata provider** compiled into musicfs that
|
||||
queries metadata-agregator's gRPC `SearchAlbums` endpoint using
|
||||
artist + album names extracted from file tags. Enrichment is lazy and
|
||||
non-blocking — file access always returns immediately using embedded
|
||||
tags, while a background worker enriches metadata asynchronously.
|
||||
|
||||
This plan **supersedes** the week-12 plan's approach of embedding
|
||||
MusicBrainz/Discogs/Last.fm HTTP clients directly into musicfs. Instead,
|
||||
musicfs delegates all external metadata resolution to metadata-agregator,
|
||||
which already handles provider APIs, rate limiting, and caching.
|
||||
|
||||
## 2. Background
|
||||
|
||||
### 2.1. Current State
|
||||
|
||||
musicfs extracts audio metadata via symphonia (FLAC, MP3, AAC, OGG,
|
||||
Opus) and stores it in `AudioMeta`. This metadata is whatever the file
|
||||
tags contain — typically title, artist, album, year, track number.
|
||||
|
||||
The existing plugin system (`musicfs-plugins`) defines a `MetadataPlugin`
|
||||
trait for external metadata lookup, but:
|
||||
|
||||
- No plugins have been implemented yet.
|
||||
- The plugin system only supports native `.so` and WASM plugins.
|
||||
- A gRPC client to metadata-agregator would require bundling an async
|
||||
runtime and tonic inside a `.so` — an awkward fit.
|
||||
|
||||
Meanwhile, metadata-agregator is a Go gRPC service that:
|
||||
|
||||
- Searches MusicBrainz by artist + album name (`SearchAlbums` RPC).
|
||||
- Caches results in PostgreSQL.
|
||||
- Returns rich metadata: genres, cover URL, label, release date, album
|
||||
type, artist credits.
|
||||
|
||||
### 2.2. Pain Points
|
||||
|
||||
- musicfs files lack genres, artwork URLs, and label info unless the
|
||||
original files were meticulously tagged.
|
||||
- The week-12 plan proposed embedding 4 separate HTTP API clients
|
||||
(MusicBrainz, Discogs, Last.fm, AcoustID) directly into musicfs,
|
||||
duplicating what metadata-agregator already does.
|
||||
- The `MetadataPlugin` trait is designed for `.so`/WASM plugins, which
|
||||
is wrong for a core infrastructure gRPC client.
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1. Goals
|
||||
|
||||
- **G1:** Enrich file metadata with genres, label, album type, and cover
|
||||
URL by querying metadata-agregator via gRPC.
|
||||
- **G2:** Never block file access — enrichment happens in background.
|
||||
- **G3:** Make the provider entirely optional — disabled by default,
|
||||
musicfs works identically without it.
|
||||
- **G4:** Respect enrichment source priority so orchestrator pushes
|
||||
(from the full-system mode) are not overwritten.
|
||||
|
||||
### 3.2. Non-Goals
|
||||
|
||||
- **NG1:** Embedding MusicBrainz/Discogs/Last.fm HTTP clients directly
|
||||
into musicfs (metadata-agregator handles this).
|
||||
- **NG2:** Audio fingerprinting (AcoustID) — deferred to future work.
|
||||
- **NG3:** Modifying the existing `MetadataPlugin` trait — the built-in
|
||||
provider is separate from the plugin system.
|
||||
- **NG4:** Bidirectional communication — musicfs only queries
|
||||
metadata-agregator, never the reverse.
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1. High-Level Architecture
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
skinparam componentStyle rectangle
|
||||
|
||||
package "musicfs" as mfs {
|
||||
component "FUSE Layer\n(readdir/open/read)" as fuse
|
||||
component "MetadataCache / DB" as db
|
||||
component "OverlayReader\n(synthesize headers)" as overlay
|
||||
component "EnrichmentQueue\n(bounded, async)" as queue
|
||||
component "EnrichmentWorker\n(background)" as worker
|
||||
}
|
||||
|
||||
component "metadata-agregator\nSearchAlbums(query, artist)" as meta
|
||||
|
||||
fuse -right-> db : lookup metadata
|
||||
db -right-> overlay : serve with overlay
|
||||
|
||||
fuse -down-> queue : enriched_at NULL?\npush request
|
||||
queue -down-> worker : dequeue
|
||||
worker -down-> meta : gRPC:\nSearchAlbums(\n query=album,\n artist=artist)
|
||||
meta -up-> worker : Album (genres,\nlabel, cover_url)
|
||||
worker -up-> db : write enriched\nmetadata to overlay
|
||||
|
||||
note bottom of meta
|
||||
metadata-agregator handles:
|
||||
• MusicBrainz API
|
||||
• rate limiting
|
||||
• PostgreSQL cache
|
||||
end note
|
||||
|
||||
note right of fuse
|
||||
File access is never blocked.
|
||||
Returns embedded tags immediately.
|
||||
Enrichment happens async.
|
||||
end note
|
||||
@enduml
|
||||
```
|
||||
|
||||
### 4.2. Enrichment Flow
|
||||
|
||||
```plantuml
|
||||
@startuml
|
||||
!theme plain
|
||||
skinparam sequenceMessageAlign center
|
||||
|
||||
participant "Media Player" as mp
|
||||
participant "FUSE Layer" as fuse
|
||||
participant "MetadataCache\n(SQLite)" as db
|
||||
participant "EnrichmentQueue" as queue
|
||||
participant "EnrichmentWorker" as worker
|
||||
participant "metadata-agregator" as meta
|
||||
|
||||
== File Access (non-blocking) ==
|
||||
|
||||
mp -> fuse : open("/Pink Floyd/The Wall/01 - In the Flesh.flac")
|
||||
fuse -> db : lookup(virtual_path)
|
||||
db --> fuse : AudioMeta(artist, album, title, ...)\nenriched_at = NULL
|
||||
|
||||
fuse -> queue : try_push(file_id, artist="Pink Floyd", album="The Wall")
|
||||
note right of queue : non-blocking,\nbounded queue
|
||||
|
||||
fuse --> mp : return file handle\n(with embedded tags only)
|
||||
|
||||
== Background Enrichment (async) ==
|
||||
|
||||
queue -> worker : dequeue(file_id, artist, album)
|
||||
|
||||
worker -> worker : check enrichment_source\n(skip if 'orchestrator' or 'provider')
|
||||
|
||||
worker -> worker : dedup check:\nalready enriched same album?\n(reuse cached result)
|
||||
|
||||
worker -> meta : SearchAlbums(\n query="The Wall",\n artist="Pink Floyd",\n limit=1)
|
||||
meta --> worker : Album(\n genres=["Progressive Rock", "Art Rock"],\n label="Harvest",\n cover_url="https://...",\n album_type="album")
|
||||
|
||||
worker -> db : update_metadata(\n file_id,\n genres, label, cover_url,\n enrichment_source='provider',\n enriched_at=now())
|
||||
|
||||
worker -> worker : publish EventBus::FileModified
|
||||
|
||||
note over mp : next access sees\nenriched metadata
|
||||
@enduml
|
||||
```
|
||||
|
||||
### 4.3. Detailed Design
|
||||
|
||||
#### 4.3.1. Configuration
|
||||
|
||||
Add `[metadata_provider]` section to `config.toml`:
|
||||
|
||||
```toml
|
||||
[metadata_provider]
|
||||
enabled = false # disabled by default
|
||||
endpoint = "http://localhost:50051" # metadata-agregator gRPC
|
||||
timeout_ms = 5000 # per-request timeout
|
||||
retry_max = 3 # max retries on failure
|
||||
retry_backoff_ms = 1000 # initial backoff between retries
|
||||
queue_size = 256 # enrichment queue capacity
|
||||
```
|
||||
|
||||
Config struct addition in `musicfs-core/src/config.rs`:
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct MetadataProviderConfig {
|
||||
#[serde(default)]
|
||||
pub enabled: bool,
|
||||
#[serde(default = "default_provider_endpoint")]
|
||||
pub endpoint: String,
|
||||
#[serde(default = "default_provider_timeout_ms")]
|
||||
pub timeout_ms: u64,
|
||||
#[serde(default = "default_retry_max")]
|
||||
pub retry_max: u32,
|
||||
#[serde(default = "default_retry_backoff_ms")]
|
||||
pub retry_backoff_ms: u64,
|
||||
#[serde(default = "default_queue_size")]
|
||||
pub queue_size: usize,
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.3.2. Built-in Metadata Provider
|
||||
|
||||
New module in `musicfs-metadata` (not a plugin, compiled in):
|
||||
|
||||
```rust
|
||||
// musicfs-metadata/src/provider.rs
|
||||
|
||||
pub struct MetadataAgregatorProvider {
|
||||
client: MetadataServiceClient<Channel>,
|
||||
config: MetadataProviderConfig,
|
||||
}
|
||||
|
||||
impl MetadataAgregatorProvider {
|
||||
pub async fn connect(config: &MetadataProviderConfig)
|
||||
-> Result<Self>;
|
||||
|
||||
/// Query metadata-agregator by artist + album names.
|
||||
/// Returns enriched metadata if a match is found.
|
||||
pub async fn lookup(
|
||||
&self,
|
||||
artist: &str,
|
||||
album: &str,
|
||||
) -> Result<Option<EnrichedMetadata>>;
|
||||
}
|
||||
```
|
||||
|
||||
The `lookup` method calls `SearchAlbums(query=album, artist=artist,
|
||||
limit=1)` on metadata-agregator. If a result is returned, it maps
|
||||
the response to `EnrichedMetadata`:
|
||||
|
||||
```rust
|
||||
pub struct EnrichedMetadata {
|
||||
pub genres: Vec<String>,
|
||||
pub label: Option<String>,
|
||||
pub album_type: Option<String>,
|
||||
pub cover_url: Option<String>,
|
||||
pub release_date: Option<String>,
|
||||
pub total_tracks: Option<u32>,
|
||||
pub total_discs: Option<u32>,
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.3.3. ExternalMetadata Extension
|
||||
|
||||
Extend the existing `ExternalMetadata` in `musicfs-plugins/src/traits.rs`
|
||||
to carry richer data:
|
||||
|
||||
```rust
|
||||
pub struct ExternalMetadata {
|
||||
// existing fields...
|
||||
pub title: Option<String>,
|
||||
pub artist: Option<String>,
|
||||
pub album: Option<String>,
|
||||
pub album_artist: Option<String>,
|
||||
pub genre: Option<String>, // kept for backward compat
|
||||
pub year: Option<u32>,
|
||||
pub track: Option<u32>,
|
||||
pub disc: Option<u32>,
|
||||
pub musicbrainz_id: Option<String>,
|
||||
pub artwork_url: Option<String>,
|
||||
|
||||
// new fields
|
||||
pub genres: Vec<String>,
|
||||
pub label: Option<String>,
|
||||
pub album_type: Option<String>,
|
||||
pub cover_url: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
#### 4.3.4. Database Schema Changes
|
||||
|
||||
Add columns to `file_metadata` table in
|
||||
`musicfs-cache/src/schema.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE file_metadata ADD COLUMN enrichment_source TEXT;
|
||||
-- 'embedded' | 'provider' | 'orchestrator'
|
||||
ALTER TABLE file_metadata ADD COLUMN enriched_at INTEGER;
|
||||
-- unix timestamp, NULL = not enriched
|
||||
ALTER TABLE file_metadata ADD COLUMN enrichment_attempts INTEGER DEFAULT 0;
|
||||
-- number of failed enrichment attempts
|
||||
ALTER TABLE file_metadata ADD COLUMN last_enrichment_error TEXT;
|
||||
-- last error message, NULL if no error
|
||||
ALTER TABLE file_metadata ADD COLUMN genres_json TEXT;
|
||||
-- JSON array: '["Progressive Rock","Art Rock"]'
|
||||
-- separate from existing `genre` (singular) for backward compat
|
||||
ALTER TABLE file_metadata ADD COLUMN label TEXT;
|
||||
ALTER TABLE file_metadata ADD COLUMN album_type TEXT;
|
||||
ALTER TABLE file_metadata ADD COLUMN cover_url TEXT;
|
||||
```
|
||||
|
||||
> **Note:** The existing `genre TEXT` column (singular) is preserved
|
||||
> for backward compatibility. `genres_json` stores the full list.
|
||||
> The singular `genre` field is set to the first genre in the array
|
||||
> when enriched.
|
||||
|
||||
#### 4.3.5. Background Enrichment Queue + Worker
|
||||
|
||||
```rust
|
||||
// musicfs-metadata/src/enrichment.rs
|
||||
|
||||
pub struct EnrichmentQueue {
|
||||
tx: mpsc::Sender<EnrichmentRequest>,
|
||||
/// Tracks in-flight (artist, album) pairs to prevent duplicate
|
||||
/// API calls when multiple tracks from the same album are
|
||||
/// accessed simultaneously.
|
||||
in_flight: Arc<DashSet<(String, String)>>,
|
||||
}
|
||||
|
||||
struct EnrichmentRequest {
|
||||
file_id: FileId,
|
||||
artist: String,
|
||||
album: String,
|
||||
}
|
||||
|
||||
pub struct EnrichmentWorker {
|
||||
rx: mpsc::Receiver<EnrichmentRequest>,
|
||||
provider: Arc<MetadataAgregatorProvider>,
|
||||
db: Arc<Database>,
|
||||
event_bus: Arc<EventBus>,
|
||||
in_flight: Arc<DashSet<(String, String)>>,
|
||||
config: MetadataProviderConfig,
|
||||
}
|
||||
```
|
||||
|
||||
##### Enqueue-time dedup
|
||||
|
||||
When `EnrichmentQueue::try_push()` is called, it checks the
|
||||
`in_flight` `DashSet` before pushing. If `(artist, album)` is
|
||||
already in the set, the request is dropped (the worker will enrich
|
||||
all files with the same album in one pass). This prevents 12
|
||||
simultaneous track opens from making 12 identical API calls.
|
||||
|
||||
If `try_push` fails because the queue is full, log at WARN level
|
||||
and increment `enrichment_queue_drops_total` metric.
|
||||
|
||||
##### Worker loop (single-threaded, processes one at a time):
|
||||
|
||||
1. Dequeue `EnrichmentRequest`.
|
||||
2. Check `enrichment_attempts` — skip if `>= retry_max`.
|
||||
3. **Atomic conflict check**: write uses conditional SQL:
|
||||
```sql
|
||||
UPDATE file_metadata SET
|
||||
genres_json = ?, label = ?, album_type = ?, cover_url = ?,
|
||||
genre = ?, -- first genre for backward compat
|
||||
enrichment_source = 'provider',
|
||||
enriched_at = strftime('%s', 'now'),
|
||||
enrichment_attempts = 0,
|
||||
last_enrichment_error = NULL
|
||||
WHERE file_id = ?
|
||||
AND (enrichment_source IS NULL OR enrichment_source = 'embedded')
|
||||
```
|
||||
This prevents the TOCTOU race — if the orchestrator wrote between
|
||||
dequeue and now, the `WHERE` clause prevents overwrite. The UPDATE
|
||||
returns rows_affected=0, which the worker treats as "skip, already
|
||||
enriched by higher-priority source".
|
||||
4. Deduplicate by (artist, album) — if another file in the same album
|
||||
was already enriched, reuse the cached `EnrichedMetadata` result
|
||||
for all files with the same (artist, album) pair.
|
||||
5. Call `provider.lookup(artist, album)`.
|
||||
6. On success: execute atomic update (step 3) for all files with this
|
||||
(artist, album). Publish `EventBus::FileModified` for each updated
|
||||
file. Remove `(artist, album)` from `in_flight` set.
|
||||
7. On failure: increment `enrichment_attempts`, set
|
||||
`last_enrichment_error`. If `attempts < retry_max`, re-enqueue
|
||||
with exponential backoff (`retry_backoff_ms * 2^attempts`).
|
||||
If `attempts >= retry_max`, log at WARN and stop retrying.
|
||||
Remove from `in_flight` set.
|
||||
|
||||
##### Shutdown behavior
|
||||
|
||||
Queue contents are lost on shutdown. This is acceptable — files will
|
||||
be re-queued on next access since `enriched_at` is still NULL.
|
||||
Enrichment is idempotent.
|
||||
|
||||
#### 4.3.6. FUSE Integration Point
|
||||
|
||||
In the FUSE `readdir` / `getattr` / `open` path
|
||||
(`musicfs-fuse/src/ops.rs`), after loading `AudioMeta` from DB:
|
||||
|
||||
```rust
|
||||
if metadata_provider.is_enabled()
|
||||
&& file_meta.enriched_at.is_none()
|
||||
&& file_meta.enrichment_attempts < config.retry_max
|
||||
&& file_meta.audio.artist.is_some()
|
||||
&& file_meta.audio.album.is_some()
|
||||
{
|
||||
if let Err(_) = enrichment_queue.try_push(EnrichmentRequest {
|
||||
file_id: file_meta.id,
|
||||
artist: file_meta.audio.artist.unwrap(),
|
||||
album: file_meta.audio.album.unwrap(),
|
||||
}) {
|
||||
// Queue full — file will be retried on next access
|
||||
tracing::warn!(
|
||||
file_id = ?file_meta.id,
|
||||
"enrichment queue full, dropping request"
|
||||
);
|
||||
metrics::ENRICHMENT_QUEUE_DROPS.inc();
|
||||
}
|
||||
// Non-blocking: returns immediately with embedded tags
|
||||
}
|
||||
```
|
||||
|
||||
The `enrichment_attempts < retry_max` check prevents files that have
|
||||
permanently failed enrichment (e.g., metadata-agregator has no match)
|
||||
from being re-queued on every access.
|
||||
|
||||
#### 4.3.7. Conflict Resolution
|
||||
|
||||
| Source | Priority | Writes When |
|
||||
|--------|----------|-------------|
|
||||
| `orchestrator` | Highest | Always overwrites (full-system mode push) |
|
||||
| `provider` | Medium | Only if current source is NULL or `'embedded'` |
|
||||
| `embedded` | Lowest | Implicit default from file tag parsing |
|
||||
|
||||
Conflict resolution is enforced **atomically at write time** using
|
||||
conditional SQL (`WHERE enrichment_source IS NULL OR
|
||||
enrichment_source = 'embedded'`), not at dequeue time. This prevents
|
||||
the TOCTOU race where the orchestrator writes between the worker's
|
||||
check and the worker's write.
|
||||
|
||||
#### 4.3.8. Proto Changes Required
|
||||
|
||||
The existing `UpdateMetadataRequest` in `musicfs.proto` must be
|
||||
extended to carry the new enrichment fields:
|
||||
|
||||
```protobuf
|
||||
// Add to UpdateMetadataRequest:
|
||||
optional string label = 40;
|
||||
optional string album_type = 41;
|
||||
optional string cover_url = 42;
|
||||
```
|
||||
|
||||
> **Note on genres:** metadata-agregator returns `repeated Genre`
|
||||
> (objects with `id` + `name`). The provider extracts genre names
|
||||
> and stores them as a JSON array in `genres_json`. The singular
|
||||
> `genre` field in `UpdateMetadataRequest` (already exists at
|
||||
> field 9) is set to the first/primary genre for backward compat.
|
||||
|
||||
#### 4.3.9. `cover_url` Usage
|
||||
|
||||
`cover_url` is stored in the metadata overlay but is **not used by
|
||||
musicfs for artwork embedding or display** in this plan. It is
|
||||
stored for consumption by external tools (e.g., media players that
|
||||
query musicfs's gRPC `GetMetadata` and fetch artwork themselves).
|
||||
Artwork download and caching is deferred to future work.
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1. Security & Privacy
|
||||
|
||||
- gRPC connection to metadata-agregator is plaintext (internal network).
|
||||
TLS can be added via config if needed.
|
||||
- No PII involved — only music metadata.
|
||||
- No API keys stored in musicfs — metadata-agregator handles provider
|
||||
auth.
|
||||
|
||||
### 5.2. Observability
|
||||
|
||||
New tracing spans and metrics:
|
||||
|
||||
| Metric | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| `enrichment_queue_depth` | Gauge | Current queue size |
|
||||
| `enrichment_queue_drops_total` | Counter | Requests dropped (queue full) |
|
||||
| `enrichment_inflight_albums` | Gauge | In-flight (artist, album) dedup set size |
|
||||
| `enrichment_lookups_total` | Counter | Total provider lookups |
|
||||
| `enrichment_hits_total` | Counter | Successful matches |
|
||||
| `enrichment_misses_total` | Counter | No match found |
|
||||
| `enrichment_errors_total` | Counter | Provider errors |
|
||||
| `enrichment_skipped_total` | Counter | Skipped (higher-priority source already wrote) |
|
||||
| `enrichment_latency_ms` | Histogram | Lookup latency |
|
||||
|
||||
### 5.3. Scalability & Performance
|
||||
|
||||
- Queue is bounded (default 256) — backpressure via `try_push`.
|
||||
- Album-level deduplication: 12 tracks in same album = 1 lookup.
|
||||
- No impact on file read latency — enrichment is fully async.
|
||||
- metadata-agregator caches in PostgreSQL, so repeated lookups are
|
||||
cheap.
|
||||
|
||||
### 5.4. Testing Plan
|
||||
|
||||
| Test | Type | Validates |
|
||||
|------|------|-----------|
|
||||
| `test_provider_connect` | Unit | gRPC connection setup |
|
||||
| `test_lookup_match` | Unit (mock) | SearchAlbums → EnrichedMetadata mapping |
|
||||
| `test_lookup_no_match` | Unit (mock) | Graceful handling of empty results, increments attempts |
|
||||
| `test_enrichment_queue_push` | Unit | Queue push + in_flight dedup |
|
||||
| `test_enrichment_queue_full_drops` | Unit | try_push fails gracefully, logs, increments metric |
|
||||
| `test_enrichment_worker_writes_db` | Integration | DB write after lookup |
|
||||
| `test_enrichment_atomic_conflict` | Integration | Orchestrator writes between dequeue and worker write → worker does NOT overwrite |
|
||||
| `test_enrichment_retry_backoff` | Unit | Failed attempts increment counter, exponential backoff |
|
||||
| `test_enrichment_max_attempts_stop` | Unit | After retry_max failures, file not re-queued |
|
||||
| `test_config_disabled` | Unit | No queue/worker when disabled |
|
||||
| `test_album_dedup_simultaneous` | Integration | 12 tracks opened at once → 1 API call |
|
||||
| `test_genre_backward_compat` | Unit | genres_json stored as array, genre set to first entry |
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1. Native .so Plugin
|
||||
|
||||
Rejected. Requires bundling a separate async runtime + tonic gRPC
|
||||
stack inside a dynamically loaded library. ABI instability, duplicate
|
||||
runtimes, and deployment complexity outweigh the "purity" of using the
|
||||
plugin system.
|
||||
|
||||
### 6.2. Direct MusicBrainz/Discogs/Last.fm HTTP Clients (week-12 plan)
|
||||
|
||||
Rejected. metadata-agregator already handles these providers with rate
|
||||
limiting, caching, and deduplication. Embedding HTTP clients in musicfs
|
||||
would duplicate this work and couple musicfs to specific provider APIs.
|
||||
|
||||
### 6.3. WASM Plugin
|
||||
|
||||
Rejected. WASI networking is immature. gRPC over WASM adds unnecessary
|
||||
latency and complexity.
|
||||
|
||||
### 6.4. On-Demand Blocking Lookup
|
||||
|
||||
Rejected. Blocking file access while waiting for a gRPC response would
|
||||
cause latency spikes and kill media player UX. Background async is the
|
||||
only acceptable approach.
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### Phase 1: Foundation (Day 1)
|
||||
|
||||
- [ ] Add `MetadataProviderConfig` to config.rs
|
||||
- [ ] Add DB schema columns: `enrichment_source`, `enriched_at`,
|
||||
`enrichment_attempts`, `last_enrichment_error`, `genres_json`,
|
||||
`label`, `album_type`, `cover_url`
|
||||
- [ ] Add `label`, `album_type`, `cover_url` fields to
|
||||
`UpdateMetadataRequest` in `musicfs.proto`
|
||||
- [ ] Extend `ExternalMetadata` struct
|
||||
- [ ] Update `config.example.toml`
|
||||
|
||||
### Phase 2: Provider + Worker (Day 1–2)
|
||||
|
||||
- [ ] Implement `MetadataAgregatorProvider` (gRPC client wrapper)
|
||||
- [ ] Implement `EnrichmentQueue` with `DashSet` in-flight dedup
|
||||
- [ ] Implement `EnrichmentWorker` with:
|
||||
- Atomic conditional write (`WHERE enrichment_source IS NULL OR ...`)
|
||||
- Retry tracking (`enrichment_attempts`, exponential backoff)
|
||||
- Album-level result caching
|
||||
- [ ] Add queue drop logging + metrics
|
||||
- [ ] Wire into startup (musicfs-cli) — conditional on config
|
||||
|
||||
### Phase 3: Integration + Tests (Day 2)
|
||||
|
||||
- [ ] Wire enrichment trigger in FUSE getattr/readdir path
|
||||
(with `enrichment_attempts < retry_max` guard)
|
||||
- [ ] Write unit tests: atomic conflict, queue drops, retry backoff,
|
||||
max attempts, genre backward compat
|
||||
- [ ] Write integration test: 12-track simultaneous dedup
|
||||
- [ ] Write integration test with in-memory DB + mock gRPC server
|
||||
- [ ] Update architecture.md with metadata provider component
|
||||
|
||||
## 8. Glossary / References
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| metadata-agregator | Go gRPC service that searches MusicBrainz and caches results in PostgreSQL |
|
||||
| Enrichment | Adding genres, label, artwork URL to file metadata beyond what's in file tags |
|
||||
| Overlay | musicfs mechanism for serving modified metadata without changing origin files |
|
||||
| `AudioMeta` | Core metadata struct extracted from file tags by symphonia |
|
||||
| `ExternalMetadata` | Metadata returned by external providers (plugin trait) |
|
||||
| `enrichment_source` | Tracks who last wrote metadata: `embedded`, `provider`, or `orchestrator` |
|
||||
|
||||
- [metadata-agregator proto](../../../../metadata-agregator/proto/metadata/v1/metadata.proto)
|
||||
- [musicfs-plugins traits](../../crates/musicfs-plugins/src/traits.rs)
|
||||
- [musicfs-cache overlay](../../crates/musicfs-cache/src/overlay.rs)
|
||||
- [architecture.md](../architecture.md)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,105 +0,0 @@
|
||||
**Date**: 2026-05-17
|
||||
**Status**: Shipped
|
||||
|
||||
# Feature: Create Directory (mkdir)
|
||||
|
||||
## Overview
|
||||
|
||||
MusicFS supports creating directories in the virtual filesystem. This enables organizing files into custom folder structures beyond the auto-generated metadata-based layout.
|
||||
|
||||
## Behavior
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
mkdir "/mnt/music/New Artist"
|
||||
mkdir "/mnt/music/New Artist/New Album"
|
||||
```
|
||||
|
||||
- Creates empty directory at specified path
|
||||
- Parent directory must exist
|
||||
- Standard POSIX semantics
|
||||
|
||||
### Nested Directories
|
||||
|
||||
```bash
|
||||
# This works (shell handles -p)
|
||||
mkdir -p "/mnt/music/A/B/C"
|
||||
|
||||
# Equivalent to:
|
||||
mkdir "/mnt/music/A"
|
||||
mkdir "/mnt/music/A/B"
|
||||
mkdir "/mnt/music/A/B/C"
|
||||
```
|
||||
|
||||
The `-p` flag is handled by the shell, which makes multiple `mkdir` syscalls.
|
||||
|
||||
### Brace Expansion
|
||||
|
||||
```bash
|
||||
# Shell expands this to multiple mkdir calls
|
||||
mkdir "/mnt/music/Artist/{Album1,Album2,Album3}"
|
||||
|
||||
# Equivalent to:
|
||||
mkdir "/mnt/music/Artist/Album1"
|
||||
mkdir "/mnt/music/Artist/Album2"
|
||||
mkdir "/mnt/music/Artist/Album3"
|
||||
```
|
||||
|
||||
Brace expansion is shell functionality, not filesystem.
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | Error |
|
||||
|-----------|-------|
|
||||
| Parent doesn't exist | `ENOENT` |
|
||||
| Path already exists | `EEXIST` |
|
||||
|
||||
## Persistence
|
||||
|
||||
**Empty directories persist across remounts.**
|
||||
|
||||
- User-created directories are stored in the `directories` table
|
||||
- On mount, directories are restored from database
|
||||
- Directories survive even when empty
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Organizing Downloads
|
||||
|
||||
```bash
|
||||
# Create structure
|
||||
mkdir "/mnt/music/Unsorted"
|
||||
mkdir "/mnt/music/Unsorted/2026"
|
||||
|
||||
# Move untagged files
|
||||
mv "/mnt/music/Unknown Artist/Unknown Album/"*.flac "/mnt/music/Unsorted/2026/"
|
||||
```
|
||||
|
||||
### Custom Collections
|
||||
|
||||
```bash
|
||||
# Create playlist-like structure
|
||||
mkdir "/mnt/music/_Playlists"
|
||||
mkdir "/mnt/music/_Playlists/Road Trip"
|
||||
|
||||
# Move tracks (they'll still be in original location too - wait, no they won't)
|
||||
# Note: mv moves, doesn't copy
|
||||
```
|
||||
|
||||
## Implementation
|
||||
|
||||
| Component | File |
|
||||
|-----------|------|
|
||||
| Tree | `crates/musicfs-cache/src/tree.rs` |
|
||||
| FUSE | `crates/musicfs-fuse/src/filesystem.rs` |
|
||||
|
||||
### Key Functions
|
||||
|
||||
- `VirtualTree::mkdir()` - Create directory node in tree
|
||||
- `Filesystem::mkdir()` - FUSE operation handler
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No permissions**: Mode/umask parameters are ignored (always 0755)
|
||||
- **No ownership**: UID/GID set to mounting user
|
||||
@@ -1,94 +0,0 @@
|
||||
**Date**: 2026-05-17
|
||||
**Status**: Shipped
|
||||
|
||||
# Feature: Move/Rename (mv)
|
||||
|
||||
## Overview
|
||||
|
||||
MusicFS supports moving and renaming files and directories within the virtual filesystem. Moves are persisted to the SQLite database and survive remounts.
|
||||
|
||||
## Behavior
|
||||
|
||||
### File Rename
|
||||
|
||||
```bash
|
||||
mv "/mnt/music/Artist/Album/old.flac" "/mnt/music/Artist/Album/new.flac"
|
||||
```
|
||||
|
||||
- Renames file within same directory
|
||||
- Updates `virtual_path` in database
|
||||
- Original file on origin is unchanged
|
||||
|
||||
### File Move
|
||||
|
||||
```bash
|
||||
mv "/mnt/music/Artist/Album/track.flac" "/mnt/music/Other Artist/Other Album/track.flac"
|
||||
```
|
||||
|
||||
- Moves file to different directory
|
||||
- **Requires target directory to exist** (use `mkdir` first)
|
||||
- Returns `ENOENT` if target parent doesn't exist
|
||||
|
||||
### Directory Rename
|
||||
|
||||
```bash
|
||||
mv "/mnt/music/Old Artist" "/mnt/music/New Artist"
|
||||
```
|
||||
|
||||
- Renames directory and all descendants
|
||||
- All files under the directory have their `virtual_path` updated in DB
|
||||
- Single atomic operation
|
||||
|
||||
### Directory Move
|
||||
|
||||
```bash
|
||||
mv "/mnt/music/Artist/Album" "/mnt/music/Other Artist/Album"
|
||||
```
|
||||
|
||||
- Moves directory subtree to new parent
|
||||
- **Requires target parent to exist**
|
||||
- Returns `ENOENT` if target parent doesn't exist
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | Error |
|
||||
|-----------|-------|
|
||||
| Source doesn't exist | `ENOENT` |
|
||||
| Target already exists | `EEXIST` |
|
||||
| Target parent doesn't exist | `ENOENT` |
|
||||
| Source is file but treated as dir | `EISDIR` |
|
||||
| Source is dir but treated as file | `ENOTDIR` |
|
||||
|
||||
## Persistence
|
||||
|
||||
- File moves: `virtual_path` column updated in `files` table
|
||||
- Directory moves: All matching `virtual_path` entries updated with new prefix
|
||||
- User directories: Tracked in separate `directories` table
|
||||
- Changes persist across unmount/remount cycles
|
||||
|
||||
On mount, the CLI:
|
||||
1. Scans origin files
|
||||
2. For each file, checks DB for stored `virtual_path` (by origin_id + real_path)
|
||||
3. Uses stored path if found, otherwise generates from metadata
|
||||
4. Restores user-created directories from `directories` table
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Read-only content**: File contents cannot be modified, only paths
|
||||
- **No cross-origin moves**: All files remain on their original origin
|
||||
- **No overwrite**: Moving to existing path fails (no implicit delete)
|
||||
|
||||
## Implementation
|
||||
|
||||
| Component | File |
|
||||
|-----------|------|
|
||||
| Database | `crates/musicfs-cache/src/db.rs` |
|
||||
| Tree | `crates/musicfs-cache/src/tree.rs` |
|
||||
| FUSE | `crates/musicfs-fuse/src/filesystem.rs` |
|
||||
|
||||
### Key Functions
|
||||
|
||||
- `Database::update_virtual_path()` - Update single file path
|
||||
- `Database::rename_directory()` - Bulk update paths with prefix
|
||||
- `VirtualTree::rename_file()` - Move file node in tree
|
||||
- `VirtualTree::rename_directory()` - Move directory subtree
|
||||
@@ -1,166 +0,0 @@
|
||||
**Date**: 2026-05-17
|
||||
**Status**: Shipped
|
||||
|
||||
# Feature: Remove (rm)
|
||||
|
||||
## Overview
|
||||
|
||||
MusicFS supports removing files and directories. Deleted files are moved to a virtual `/.trash/` directory and can be restored. The trash is browsable — users can manually move files out.
|
||||
|
||||
## Behavior
|
||||
|
||||
### Remove File
|
||||
|
||||
```bash
|
||||
rm "/mnt/music/Artist/Album/track.flac"
|
||||
```
|
||||
|
||||
- File moves to `/.trash/Artist/Album/track.flac`
|
||||
- Original directory structure preserved in trash
|
||||
- File still accessible via `/.trash/` path
|
||||
- Database marks file as `trashed=1` with original path stored
|
||||
|
||||
### Remove Empty Directory
|
||||
|
||||
```bash
|
||||
rmdir "/mnt/music/Empty Folder"
|
||||
```
|
||||
|
||||
- Removes empty directory from tree
|
||||
- Removes from `directories` table if user-created
|
||||
- Fails with `ENOTEMPTY` if directory has children
|
||||
|
||||
### Remove Directory Recursively
|
||||
|
||||
```bash
|
||||
rm -rf "/mnt/music/Artist"
|
||||
```
|
||||
|
||||
- Shell handles recursion (depth-first unlink + rmdir)
|
||||
- All files moved to `/.trash/Artist/...`
|
||||
- Empty directories removed after files are trashed
|
||||
|
||||
## The `.trash/` Directory
|
||||
|
||||
Deleted files live in `/.trash/` with their original path structure:
|
||||
|
||||
```
|
||||
/.trash/
|
||||
├── Artist/
|
||||
│ └── Album/
|
||||
│ ├── track1.flac
|
||||
│ └── track2.flac
|
||||
└── Other Artist/
|
||||
└── song.flac
|
||||
```
|
||||
|
||||
### Browse Trash
|
||||
|
||||
```bash
|
||||
ls "/.trash/"
|
||||
ls "/.trash/Artist/Album/"
|
||||
```
|
||||
|
||||
### Manual Restore
|
||||
|
||||
```bash
|
||||
# Move file back manually - trashed flag is automatically cleared
|
||||
mv "/.trash/Artist/Album/track.flac" "/Artist/Album/"
|
||||
```
|
||||
|
||||
When moving a file out of `/.trash/`, the database `trashed` flag is automatically cleared.
|
||||
|
||||
## CLI Commands
|
||||
|
||||
All trash commands require either `--config` or `--cache-dir`:
|
||||
|
||||
```bash
|
||||
musicfs trash -c config.toml <command>
|
||||
musicfs trash --cache-dir ./dev/cache/musicfs <command>
|
||||
```
|
||||
|
||||
### List Deleted Files
|
||||
|
||||
```bash
|
||||
musicfs trash -c config.toml list
|
||||
musicfs trash -c config.toml list --origin local-storage
|
||||
musicfs trash -c config.toml list --since 7d
|
||||
musicfs trash -c config.toml list --path "/Artist"
|
||||
```
|
||||
|
||||
Output shows index, deletion time, and original path.
|
||||
|
||||
### Restore Files
|
||||
|
||||
```bash
|
||||
# Restore single file or folder
|
||||
musicfs trash -c config.toml restore "/Artist/Album/track.flac"
|
||||
|
||||
# Restore entire folder recursively
|
||||
musicfs trash -c config.toml restore "/Artist"
|
||||
|
||||
# Restore everything
|
||||
musicfs trash -c config.toml restore --all
|
||||
```
|
||||
|
||||
CLI restore writes paths to a pending restore file and sends SIGHUP to the daemon.
|
||||
The daemon processes pending restores and moves files back from `/.trash/`.
|
||||
|
||||
### Empty Trash
|
||||
|
||||
```bash
|
||||
# Permanently delete all trashed files
|
||||
musicfs trash -c config.toml empty
|
||||
|
||||
# Delete old items only
|
||||
musicfs trash -c config.toml empty --older-than 30d
|
||||
|
||||
# Delete by path pattern
|
||||
musicfs trash -c config.toml empty --pattern "/Artist"
|
||||
```
|
||||
|
||||
**Warning:** Empty permanently removes files from MusicFS database. Origin files are unaffected.
|
||||
|
||||
## Error Codes
|
||||
|
||||
| Condition | Error |
|
||||
|-----------|-------|
|
||||
| Path doesn't exist | `ENOENT` |
|
||||
| `rm` on directory (without `-r`) | `EISDIR` |
|
||||
| `rmdir` on file | `ENOTDIR` |
|
||||
| `rmdir` on non-empty directory | `ENOTEMPTY` |
|
||||
| `rmdir` on `/.trash/` | `EPERM` |
|
||||
|
||||
## Database Schema
|
||||
|
||||
Files table extended with trash columns:
|
||||
|
||||
```sql
|
||||
trashed INTEGER NOT NULL DEFAULT 0,
|
||||
original_path TEXT,
|
||||
trashed_at INTEGER
|
||||
```
|
||||
|
||||
Partial index for efficient trash queries:
|
||||
```sql
|
||||
CREATE INDEX idx_files_trashed ON files(trashed) WHERE trashed = 1;
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Delete (`rm`)**: FUSE `unlink` moves file to `/.trash/`, marks `trashed=1` in DB
|
||||
2. **Manual restore (`mv`)**: Moving out of `/.trash/` automatically clears `trashed` flag
|
||||
3. **CLI restore**: Writes pending paths, sends SIGHUP to daemon, daemon processes restores
|
||||
4. **Empty**: Deletes matching records from database
|
||||
|
||||
## Persistence
|
||||
|
||||
- Trashed files persist across remounts (stored in `/.trash/` subtree)
|
||||
- Files marked with `trashed=1`, `original_path`, `trashed_at` in database
|
||||
- PID file at `{cache_dir}/musicfs.pid` for CLI→daemon communication
|
||||
|
||||
## Limitations
|
||||
|
||||
- **No hard delete of remote files**: Origin content is never modified
|
||||
- **Trash uses virtual space**: Files still in tree under `/.trash/` until emptied
|
||||
- **CLI restore requires running daemon**: Manual `mv` works without daemon
|
||||
@@ -1,239 +0,0 @@
|
||||
# MusicFS MVP Performance Review
|
||||
|
||||
**Date**: 2026-05-12
|
||||
**Test Data**: Metallica - 72 Seasons (12 FLAC tracks, 625MB, 16-bit/44.1kHz)
|
||||
**Origin**: Local filesystem (Docker volume)
|
||||
**System**: Linux, NixOS
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Phase 1 MVP is functional** - the system mounts, browses, and reads files successfully. Audio playback works with valid FLAC headers served. However, there's a **critical gap** between the architecture specification and current implementation regarding content chunking.
|
||||
|
||||
---
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
### Throughput Comparison
|
||||
|
||||
| Metric | Direct FS | MusicFS Cold | MusicFS Warm | Target (Spec) | Status |
|
||||
|--------|-----------|--------------|--------------|---------------|--------|
|
||||
| Single file read (64MB) | 0.022s (3 GB/s) | 0.035s (1.8 GB/s) | 0.020s (3.2 GB/s) | >500 MB/s | ✅ |
|
||||
| Full album read (625MB) | 0.149s (4.2 GB/s) | 0.274s (2.3 GB/s) | 0.211s (3.0 GB/s) | >500 MB/s | ✅ |
|
||||
|
||||
### Metadata Operations
|
||||
|
||||
| Operation | Result | Target (Spec) | Status |
|
||||
|-----------|--------|---------------|--------|
|
||||
| Root listing | 0.006s | <10ms | ✅ |
|
||||
| Full tree traversal (12 files) | 0.007s | <50ms | ✅ |
|
||||
| stat() per operation | 0.003s | <1ms | ⚠️ |
|
||||
| 4KB small reads (per op) | 0.006s | <1ms | ⚠️ |
|
||||
| Random seek 1MB | 0.008-0.015s | <50ms | ✅ |
|
||||
| Mount time | ~8ms | <500ms | ✅ |
|
||||
|
||||
### Cache Performance
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Cache speedup (single file) | 1.75x |
|
||||
| Cache speedup (full album) | 1.30x |
|
||||
| Cache size | 25MB |
|
||||
| Chunk count | 12 |
|
||||
| Expected cache size | 625MB |
|
||||
|
||||
### FUSE Overhead
|
||||
|
||||
| Scenario | Overhead vs Direct |
|
||||
|----------|-------------------|
|
||||
| Single file cold cache | 59% slower |
|
||||
| Single file warm cache | 9% faster* |
|
||||
| Full album cold cache | 84% slower |
|
||||
| Full album warm cache | 42% slower |
|
||||
|
||||
*Warm cache appears faster due to OS page cache effects on both paths.
|
||||
|
||||
---
|
||||
|
||||
## What's Working Well ✅
|
||||
|
||||
### 1. Mount Performance
|
||||
- Mount completes in ~8ms (spec: <500ms) — **62x better than target**
|
||||
- O(1) mount time achieved — no file scanning blocks mount
|
||||
- Lazy loading working as designed per architecture section 4.3.1
|
||||
|
||||
### 2. Virtual Tree Organization
|
||||
- Correct Artist/Album/Track hierarchy derived from metadata
|
||||
- Example path: `/Metallica/72 Seasons/01. 72 Seasons.flac`
|
||||
- Special character sanitization working (`/`, `\`, `:`, etc.)
|
||||
|
||||
### 3. File Reading
|
||||
- Valid FLAC headers served (`fLaC` magic bytes verified)
|
||||
- Sequential reads work correctly
|
||||
- Random access (seek) functional
|
||||
- Concurrent reads from multiple processes work
|
||||
|
||||
### 4. FUSE Integration
|
||||
- Read-only enforcement (EROFS returned on write attempts)
|
||||
- Proper inode assignment and file attributes
|
||||
- AllowOther mount option working
|
||||
- Clean unmount via fusermount3
|
||||
|
||||
### 5. Throughput
|
||||
- Exceeds 500 MB/s target significantly (2-3 GB/s achieved)
|
||||
- Parallel reads scale appropriately (4 files in 0.060s)
|
||||
|
||||
---
|
||||
|
||||
## Critical Issues 🔴
|
||||
|
||||
### Issue 1: Incomplete File Caching
|
||||
|
||||
**Symptom**: Cache is 25MB instead of expected 625MB (12 files × ~2MB each instead of full files)
|
||||
|
||||
**Root Cause**: In `fetcher.rs:74`:
|
||||
```rust
|
||||
let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await?;
|
||||
```
|
||||
|
||||
And in `local.rs:96-98`:
|
||||
```rust
|
||||
let mut buffer = vec![0u8; size as usize];
|
||||
let bytes_read = file.read(&mut buffer).await?;
|
||||
buffer.truncate(bytes_read);
|
||||
```
|
||||
|
||||
`tokio::fs::File::read()` reads **up to** buffer size but returns when the kernel buffer is exhausted (~2MB typical). Only first ~2MB of each file is being cached.
|
||||
|
||||
**Impact**:
|
||||
- Subsequent reads beyond 2MB offset hit origin every time
|
||||
- No cache benefit for majority of file content
|
||||
- Cache eviction policy not being exercised
|
||||
|
||||
**Required Fix**: Use `read_to_end()` or loop until all bytes read:
|
||||
```rust
|
||||
let mut buffer = Vec::with_capacity(size as usize);
|
||||
file.read_to_end(&mut buffer).await?;
|
||||
```
|
||||
|
||||
### Issue 2: No CDC Chunking Implemented
|
||||
|
||||
**Architecture Spec** (Section 4.3.2):
|
||||
> "All file content is stored as content-addressed chunks... Avg chunk: 64KB, Min: 16KB, Max: 256KB"
|
||||
|
||||
**Current Implementation**: Each file stored as ONE chunk (no FastCDC integration)
|
||||
|
||||
**Impact**:
|
||||
- No content deduplication possible
|
||||
- Delta sync impossible (FR-11.2 unmet)
|
||||
- Cache efficiency severely reduced for similar files
|
||||
|
||||
---
|
||||
|
||||
## Architecture Gaps 🟡
|
||||
|
||||
| Spec Requirement | Current State | Gap |
|
||||
|------------------|---------------|-----|
|
||||
| CDC chunking (64KB avg) | No chunking | Missing FastCDC integration |
|
||||
| Delta sync (>90% bandwidth reduction) | Not implemented | Requires CDC first |
|
||||
| Deduplication (FR-20) | Not implemented | Requires CDC first |
|
||||
| Search engine (tantivy) | Not implemented | Phase 3 scope |
|
||||
| gRPC Control API | Not implemented | Phase 4 scope |
|
||||
| Multi-origin federation | Single origin only | Phase 2 scope |
|
||||
| Metadata persistence (SQLite) | In-memory HashMap | Missing persistence |
|
||||
|
||||
---
|
||||
|
||||
## Performance Analysis
|
||||
|
||||
### Why Warm Cache Appears Faster Than Direct FS
|
||||
|
||||
The warm cache shows 3.2 GB/s vs direct 3.0 GB/s because:
|
||||
1. OS page cache is warm for both MusicFS chunks AND origin files
|
||||
2. Both measurements are essentially hitting RAM, variance expected
|
||||
3. MusicFS chunks may have slightly better cache locality
|
||||
|
||||
### stat() Latency Above Target
|
||||
|
||||
Current: 3ms per stat() vs target <1ms
|
||||
|
||||
Possible causes:
|
||||
1. `RwLock<VirtualTree>` contention overhead
|
||||
2. HashMap lookup plus FUSE context switch
|
||||
3. Measurement includes full round-trip through FUSE
|
||||
|
||||
Mitigation options:
|
||||
- Consider lock-free concurrent data structures
|
||||
- Implement finer-grained locking
|
||||
- Cache hot inodes in separate fast-path structure
|
||||
|
||||
---
|
||||
|
||||
## Recommendations
|
||||
|
||||
### Immediate Fixes (Before Phase 2)
|
||||
|
||||
1. **Fix file reading** — Use `read_to_end()` or implement proper streaming read loop
|
||||
2. **Add CDC chunking** — Integrate FastCDC per architecture spec section 4.3.2
|
||||
3. **Persist metadata** — Move from in-memory HashMap to SQLite as specified
|
||||
|
||||
### Phase 2 Priorities
|
||||
|
||||
1. Complete CDC chunking implementation (prerequisite for delta sync)
|
||||
2. Add SQLite metadata persistence (FR-7.2)
|
||||
3. Implement multi-origin support (FR-13)
|
||||
|
||||
### Testing Gaps to Address
|
||||
|
||||
1. No automated E2E tests for real FUSE operations
|
||||
2. No stress testing with concurrent access patterns
|
||||
3. No large library testing (target: 1M+ files per NFR-3.1)
|
||||
4. No offline mode testing (origin unavailable scenarios)
|
||||
|
||||
---
|
||||
|
||||
## Test Environment Details
|
||||
|
||||
```
|
||||
Origin Path: /home/fujin/.local/share/docker/volumes/containers_downloads/_data/Metallica - 72 Seasons (2023) [FLAC] 88/
|
||||
Mount Point: /tmp/musicfs-benchmark/mount
|
||||
Cache Dir: /tmp/musicfs-benchmark/cache
|
||||
Binary: target/release/musicfs (via nix develop)
|
||||
|
||||
Files:
|
||||
01. 72 Seasons.flac 64MB
|
||||
02. Shadows Follow.flac 50MB
|
||||
03. Screaming Suicide.flac 45MB
|
||||
04. Sleepwalk My Life Away.flac 54MB
|
||||
05. You Must Burn!.flac 57MB
|
||||
06. Lux Æterna.flac 27MB
|
||||
07. Crown Of Barbed Wire.flac 46MB
|
||||
08. Chasing Light.flac 55MB
|
||||
09. If Darkness Had A Son.flac 51MB
|
||||
10. Too Far Gone_.flac 37MB
|
||||
11. Room Of Mirrors.flac 45MB
|
||||
12. Inamorata.flac 89MB
|
||||
Total: 625MB, 12 tracks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
**The MVP demonstrates core functionality works** — mounting, browsing, and reading audio files through FUSE. Throughput performance exceeds targets significantly.
|
||||
|
||||
However, **the cache implementation is incomplete**:
|
||||
- Only ~4% of file content is being cached (25MB/625MB)
|
||||
- No CDC chunking means no deduplication or delta sync capability
|
||||
- Architecture requirements FR-8.2, FR-11.2, FR-20 are unmet
|
||||
|
||||
**Recommendation**: Fix the file reading issue and add CDC chunking before proceeding to Phase 2. The architecture is sound; implementation needs to catch up to specification.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [Architecture Specification](architecture.md) — Section 4.3.2 (CAS), Section 4.3.5 (Read Flow)
|
||||
- [Requirements Specification](requirements.md) — FR-8 (Content Cache), FR-11 (Delta Sync), FR-20 (CAS)
|
||||
- [Week 4b Plan](plans/week-04b-origin-connector.md) — ContentFetcher implementation
|
||||
@@ -1,982 +0,0 @@
|
||||
# Comprehensive Logging Plan
|
||||
|
||||
**Goal**: Add production-grade logging with trace-level observability, file rotation, and systemd integration
|
||||
**Effort**: ~10-12 hours
|
||||
**Dependencies**: Existing libraries only (no custom code)
|
||||
|
||||
> **Review Status**: Reviewed by Oracle - all gaps addressed
|
||||
|
||||
---
|
||||
|
||||
## Libraries Used
|
||||
|
||||
| Need | Library | Status |
|
||||
|------|---------|--------|
|
||||
| Instrumentation | `tracing` | Already in workspace |
|
||||
| Subscriber/filtering | `tracing-subscriber` | Already in workspace |
|
||||
| File rotation | `tracing-appender` | Add to workspace |
|
||||
| systemd journal | `tracing-journald` | Add to workspace |
|
||||
| Compression | `logrotate` (Linux tool) | Config file only |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Config & Dependencies (2 hours)
|
||||
|
||||
### 1.1 Add dependencies to workspace
|
||||
|
||||
```toml
|
||||
# Cargo.toml [workspace.dependencies]
|
||||
tracing-appender = "0.2"
|
||||
tracing-journald = "0.3"
|
||||
```
|
||||
|
||||
```toml
|
||||
# crates/musicfs-cli/Cargo.toml
|
||||
tracing-appender.workspace = true
|
||||
tracing-journald.workspace = true
|
||||
```
|
||||
|
||||
### 1.2 Add LoggingConfig to config.rs
|
||||
|
||||
```rust
|
||||
// crates/musicfs-core/src/config.rs
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct Config {
|
||||
pub mount_point: PathBuf,
|
||||
pub cache_dir: PathBuf,
|
||||
pub origins: Vec<OriginConfig>,
|
||||
#[serde(default)]
|
||||
pub cache: CacheConfig,
|
||||
#[serde(default)]
|
||||
pub health: HealthConfig,
|
||||
#[serde(default)]
|
||||
pub logging: LoggingConfig, // NEW
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct LoggingConfig {
|
||||
#[serde(default = "default_log_dir")]
|
||||
pub log_dir: PathBuf,
|
||||
|
||||
#[serde(default)]
|
||||
pub json_output: bool,
|
||||
|
||||
#[serde(default = "default_true")]
|
||||
pub journald: bool,
|
||||
|
||||
#[serde(default = "default_log_level")]
|
||||
pub level: String,
|
||||
}
|
||||
|
||||
impl Default for LoggingConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
log_dir: default_log_dir(),
|
||||
json_output: false,
|
||||
journald: true,
|
||||
level: default_log_level(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn default_log_dir() -> PathBuf {
|
||||
PathBuf::from("/var/log/musicfs")
|
||||
}
|
||||
fn default_log_level() -> String {
|
||||
"musicfs=info,warn".to_string()
|
||||
}
|
||||
fn default_true() -> bool {
|
||||
true
|
||||
}
|
||||
```
|
||||
|
||||
### 1.3 Expand init_logging() in main.rs
|
||||
|
||||
```rust
|
||||
// crates/musicfs-cli/src/main.rs
|
||||
|
||||
use tracing_appender::non_blocking::WorkerGuard;
|
||||
use tracing_subscriber::{fmt, prelude::*, EnvFilter};
|
||||
|
||||
fn init_logging(config: &LoggingConfig) -> Result<WorkerGuard> {
|
||||
std::fs::create_dir_all(&config.log_dir)?;
|
||||
|
||||
// File layer with daily rotation
|
||||
let file_appender = tracing_appender::rolling::daily(&config.log_dir, "musicfs.log");
|
||||
let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
|
||||
|
||||
let file_layer = if config.json_output {
|
||||
fmt::layer()
|
||||
.json()
|
||||
.with_writer(non_blocking)
|
||||
.with_ansi(false)
|
||||
.boxed()
|
||||
} else {
|
||||
fmt::layer()
|
||||
.with_writer(non_blocking)
|
||||
.with_ansi(false)
|
||||
.boxed()
|
||||
};
|
||||
|
||||
// Journald layer (Linux only)
|
||||
#[cfg(target_os = "linux")]
|
||||
let journald_layer = if config.journald {
|
||||
tracing_journald::layer()
|
||||
.ok()
|
||||
.map(|l| l.with_syslog_identifier("musicfs".to_string()))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Stderr layer for interactive use
|
||||
let stderr_layer = fmt::layer()
|
||||
.with_writer(std::io::stderr)
|
||||
.compact();
|
||||
|
||||
// Filter from config or env
|
||||
let filter = EnvFilter::try_from_default_env()
|
||||
.unwrap_or_else(|_| EnvFilter::new(&config.level));
|
||||
|
||||
// Compose
|
||||
let subscriber = tracing_subscriber::registry()
|
||||
.with(filter)
|
||||
.with(file_layer)
|
||||
.with(stderr_layer);
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
let subscriber = subscriber.with(journald_layer);
|
||||
|
||||
subscriber.init();
|
||||
|
||||
tracing::info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting");
|
||||
Ok(guard)
|
||||
}
|
||||
```
|
||||
|
||||
### 1.4 Add logrotate config
|
||||
|
||||
```bash
|
||||
# dist/logrotate.d/musicfs
|
||||
/var/log/musicfs/*.log {
|
||||
daily
|
||||
rotate 30
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
create 0640 musicfs musicfs
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Add tracing to musicfs-core (1 hour)
|
||||
|
||||
### 2.1 Add dependency
|
||||
|
||||
```toml
|
||||
# crates/musicfs-core/Cargo.toml
|
||||
[dependencies]
|
||||
tracing.workspace = true # ADD THIS
|
||||
```
|
||||
|
||||
### 2.2 Instrument core modules
|
||||
|
||||
| File | What to Add |
|
||||
|------|-------------|
|
||||
| `config.rs` | Log config file loading, parse errors |
|
||||
| `credentials.rs` | Log credential loading (redacted values) |
|
||||
| `events.rs` | Log event publishing with counts |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Instrument Hot Paths (4 hours)
|
||||
|
||||
### Priority order by impact
|
||||
|
||||
| Crate | Files | What to Add |
|
||||
|-------|-------|-------------|
|
||||
| musicfs-fuse | `filesystem.rs` | `#[instrument]` on all FUSE ops, trace at decision points |
|
||||
| musicfs-origins | `failover.rs`, `health.rs`, `router.rs` | Retry loops, state transitions, selection logic |
|
||||
| musicfs-cache | `tree.rs`, `metadata.rs` | Tree mutations, cache hit/miss |
|
||||
| musicfs-cas | `reader.rs`, `store.rs` | Chunk operations, dedup decisions |
|
||||
| musicfs-sync | `delta.rs`, `watcher.rs` | Change detection, file events |
|
||||
|
||||
### Instrumentation patterns
|
||||
|
||||
```rust
|
||||
// Function level - add to all public async functions
|
||||
#[tracing::instrument(level = "debug", skip(self), fields(path = %path))]
|
||||
pub async fn read(&self, path: &str) -> Result<Bytes> {
|
||||
// ...
|
||||
}
|
||||
|
||||
// Decision points - add trace! at match/if branches
|
||||
match result {
|
||||
Ok(data) => {
|
||||
tracing::trace!(bytes = data.len(), "read success");
|
||||
data
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::trace!(error = %e, "read failed");
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
|
||||
// State changes - use info! for important transitions
|
||||
tracing::info!(old = ?old_status, new = ?new_status, origin = %id, "health changed");
|
||||
|
||||
// Cache operations
|
||||
tracing::trace!(hit = true, fresh = true, "cache hit");
|
||||
tracing::trace!(hit = false, "cache miss");
|
||||
```
|
||||
|
||||
### FUSE operations (filesystem.rs) - highest priority
|
||||
|
||||
| Operation | Level | Fields |
|
||||
|-----------|-------|--------|
|
||||
| `lookup()` | debug | parent, name, result_ino |
|
||||
| `getattr()` | debug | ino, file_type |
|
||||
| `readdir()` | debug | ino, entry_count |
|
||||
| `read()` | debug | ino, offset, size, bytes_read |
|
||||
| `open()` | debug | ino, flags |
|
||||
| `release()` | trace | ino |
|
||||
|
||||
### Origin operations - critical for debugging
|
||||
|
||||
| Function | Level | Fields |
|
||||
|----------|-------|--------|
|
||||
| `read_with_failover()` | debug | path, origins_tried, success |
|
||||
| `read_with_retry()` | trace | origin, attempt, success |
|
||||
| `check_health()` | debug | origin, old_status, new_status |
|
||||
| `select_origin()` | trace | candidates, selected, reason |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Update Production Files (1 hour)
|
||||
|
||||
### 4.1 Update systemd service
|
||||
|
||||
```ini
|
||||
# dist/musicfs.service (add these lines)
|
||||
Environment="RUST_LOG=musicfs=info,warn"
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=musicfs
|
||||
RateLimitIntervalSec=30s
|
||||
RateLimitBurst=1000
|
||||
```
|
||||
|
||||
### 4.2 Example config.toml
|
||||
|
||||
```toml
|
||||
# dist/config.example.toml
|
||||
mount_point = "/mnt/music"
|
||||
cache_dir = "/var/cache/musicfs"
|
||||
|
||||
[logging]
|
||||
log_dir = "/var/log/musicfs"
|
||||
json_output = true
|
||||
journald = true
|
||||
level = "musicfs=info,warn"
|
||||
|
||||
[cache]
|
||||
metadata_cache_mb = 100
|
||||
content_cache_gb = 10
|
||||
|
||||
[health]
|
||||
check_interval_secs = 30
|
||||
timeout_ms = 5000
|
||||
|
||||
[[origins]]
|
||||
id = "local"
|
||||
origin_type = "local"
|
||||
priority = 1
|
||||
path = "/srv/music"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Detailed Log Locations by Level
|
||||
|
||||
### ERROR Level (25+ locations) - Unrecoverable Failures
|
||||
|
||||
| File | Line | Log Message |
|
||||
|------|------|-------------|
|
||||
| `musicfs-grpc/src/webhook.rs` | 43 | `error!("Failed to initialize webhook HTTP client: {error}")` |
|
||||
| `musicfs-grpc/src/webhook.rs` | 133 | `error!("Invalid HMAC secret key for webhook signature: {error}")` |
|
||||
| `musicfs-plugins/src/manager.rs` | 272 | `error!("Plugin manager initialization failed: {error}")` |
|
||||
| `musicfs-plugins/src/wasm.rs` | 142,183 | `error!("WASM plugin host initialization failed: {error}")` |
|
||||
| `musicfs-search/src/index.rs` | 211,217 | `error!("Search index corrupted: failed to deserialize at position {pos}")` |
|
||||
| `musicfs-cas/src/store.rs` | 105 | `error!("CAS chunk not found: {hash} - possible data loss")` |
|
||||
| `musicfs-cas/src/store.rs` | 124-131 | `error!("CAS integrity check failed: expected {expected}, got {actual}")` |
|
||||
| `musicfs-fuse/src/filesystem.rs` | 103 | `error!("Failed to mount filesystem at {mountpoint}: {error}")` |
|
||||
| `musicfs-origins/src/failover.rs` | 76 | `error!("No origins available for path {path}")` |
|
||||
| `musicfs-origins/src/failover.rs` | 125,186 | `error!("Max retries ({max_attempts}) exceeded for origin {origin_id}")` |
|
||||
| `musicfs-origins/src/nfs.rs` | 63 | `error!("NFS stale file handle after {max_retries} retries for {path}")` |
|
||||
| `musicfs-cas/src/reader.rs` | 75 | `error!("File manifest not found for file_id {file_id}")` |
|
||||
| `musicfs-cas/src/fetcher.rs` | 60,68 | `error!("File/Origin not found for file_id {file_id}")` |
|
||||
| `musicfs-search/src/indexer.rs` | 44,56 | `error!("Search indexer/commit failed: {error}")` |
|
||||
| `musicfs-sync/src/watcher.rs` | 36,59,63 | `error!("Watcher failed for origin {origin_id}: {error}")` |
|
||||
|
||||
### WARN Level (50+ locations) - Recoverable Issues
|
||||
|
||||
| Category | File | Line | Log Message |
|
||||
|----------|------|------|-------------|
|
||||
| **Retry Logic** | `failover.rs` | 90 | `warn!("Origin {origin_id} failed: {error}, trying next (attempt {n}/{total})")` |
|
||||
| **Retry Logic** | `failover.rs` | 111-118 | `warn!("Retrying origin {origin_id} after {delay:?} (attempt {n}/{max})")` |
|
||||
| **Retry Logic** | `nfs.rs` | 47-52 | `warn!("NFS stale handle for {path} (attempt {n}/{max}), retrying")` |
|
||||
| **Retry Logic** | `smb.rs` | 45 | `warn!("SMB connection lost (ENOTCONN), retrying (attempt {n}/{max})")` |
|
||||
| **Retry Logic** | `webhook.rs` | 94-108 | `warn!("Webhook delivery failed to {url} (attempt {n}/{max}): {error}")` |
|
||||
| **Fallback** | `failover.rs` | 70-73 | `warn!("No healthy origins for {path}, using fallback {origin_id}")` |
|
||||
| **Timeout** | `smb.rs` | 107-109 | `warn!("SMB health check timed out after 5s for {origin_id}")` |
|
||||
| **Timeout** | `nfs.rs` | 104-106 | `warn!("NFS health check timed out after 5s for {origin_id}")` |
|
||||
| **Timeout** | `prefetch.rs` | 91 | `warn!("Prefetch event receive timed out after 1s")` |
|
||||
| **Health** | `health.rs` | 209 | `warn!("Origin {origin_id} is degraded (failures: {count})")` |
|
||||
| **Health** | `health.rs` | 217-220 | `warn!("Origin {origin_id} is now unhealthy after {n} consecutive failures")` |
|
||||
| **Remote FS** | `smb.rs` | 118 | `warn!("SMB watch using inotify on {share_path} - may be unreliable")` |
|
||||
| **Remote FS** | `nfs.rs` | 115 | `warn!("NFS watch using inotify on {mount_point} - may be unreliable")` |
|
||||
| **Plugin** | `manager.rs` | 152 | `warn!("Failed to load plugin from {path}: {error}")` |
|
||||
| **Plugin** | `manager.rs` | 193-194 | `warn!("Failed to unload plugin {plugin_id}: {error}")` |
|
||||
| **Prefetch** | `prefetch.rs` | 97 | `warn!("Failed to record access pattern for {file_id}: {error}")` |
|
||||
| **Prefetch** | `prefetch.rs` | 159-161 | `warn!("Prefetch skipped: concurrency limit reached ({max})")` |
|
||||
| **Search** | `indexer.rs` | 49 | `warn!("Search indexer event receive error: {error}")` |
|
||||
| **Search** | `indexer.rs` | 82 | `warn!("No metadata found for file {path}, skipping indexing")` |
|
||||
| **Collections** | `collections.rs` | 146,180 | `warn!("Failed to save/delete collection {name}: {error}")` |
|
||||
|
||||
### INFO Level (35+ locations) - Lifecycle & Major Operations
|
||||
|
||||
| Category | File | Line | Log Message |
|
||||
|----------|------|------|-------------|
|
||||
| **Lifecycle** | `main.rs` | 118 | `info!(version = env!("CARGO_PKG_VERSION"), "MusicFS starting")` |
|
||||
| **Lifecycle** | `filesystem.rs` | 94 | `info!("Mounting MusicFS at {:?}", mountpoint)` |
|
||||
| **Lifecycle** | `filesystem.rs` | 154 | `info!("MusicFS initialized")` |
|
||||
| **Lifecycle** | `filesystem.rs` | 159 | `info!("MusicFS destroyed")` |
|
||||
| **Origin** | `registry.rs` | 28 | `info!("Registering origin {} with priority {}", id, priority)` |
|
||||
| **Origin** | `registry.rs` | 36 | `info!("Unregistering origin {}", id)` |
|
||||
| **Origin** | `watcher.rs` | 65 | `info!("Watching origin {} at {:?}", origin_id, path)` |
|
||||
| **Config** | `main.rs` | 127 | `info!("Cache directory: {:?}", cache_dir)` |
|
||||
| **Config** | `main.rs` | 141 | `info!("CAS store initialized")` |
|
||||
| **Config** | `store.rs` | 51 | `info!("CAS store opened: {} chunks, {} bytes", count, size)` (ADD) |
|
||||
| **Sync** | `main.rs` | 150,152 | `info!("Scanning music files...")` / `info!("Found {} music files", count)` |
|
||||
| **Sync** | `delta.rs` | 104 | `info!("Delta complete: {} added, {} removed, {} modified", a, r, m)` |
|
||||
| **Sync** | `delta.rs` | 63 | `info!("Sync started for origin {}", origin_id)` (ADD) |
|
||||
| **Index** | `main.rs` | 160 | `info!("Virtual tree built")` |
|
||||
| **Index** | `indexer.rs` | 62 | `info!("Indexer stopping")` |
|
||||
| **Index** | `indexer.rs` | 114 | `info!("Indexed {} files", count)` |
|
||||
| **Index** | `index.rs` | 170 | `info!("Search index committed")` |
|
||||
| **Health** | `health.rs` | 202 | `info!("Origin {} is now healthy", id)` |
|
||||
| **Health** | `health.rs` | 150 | `info!("Health monitor started with interval {:?}", interval)` (ADD) |
|
||||
| **Plugin** | `manager.rs` | 127 | `info!("Initializing plugin system")` |
|
||||
| **Plugin** | `manager.rs` | 150 | `info!("Loaded plugin '{}' with id {:?}", name, id)` |
|
||||
| **Plugin** | `manager.rs` | 256 | `info!("Shutting down plugin system")` |
|
||||
| **Cache** | `prefetch.rs` | 123 | `info!("Prefetch engine stopped")` |
|
||||
| **Cache** | `prefetch.rs` | 174 | `info!("Prefetched {:?}: {} chunks, {} bytes", file_id, chunks, bytes)` |
|
||||
| **Cache** | `eviction.rs` | 51 | `info!("Evicted {} bytes from cache", bytes)` |
|
||||
| **Cache** | `prefetch.rs` | 73 | `info!("Prefetch engine started (lookahead: {}, max_concurrent: {})")` (ADD) |
|
||||
|
||||
### DEBUG Level (60+ locations) - Operation Details
|
||||
|
||||
| Category | File | Line | Log Message |
|
||||
|----------|------|------|-------------|
|
||||
| **FUSE lookup** | `filesystem.rs` | 162,195,200 | Entry + result/miss |
|
||||
| **FUSE getattr** | `filesystem.rs` | 203,230,233 | Entry + result/miss |
|
||||
| **FUSE readdir** | `filesystem.rs` | 237,263,303 | Entry + result/miss |
|
||||
| **FUSE read** | `filesystem.rs` | 325,338,362,364 | Entry + file_id + result/error |
|
||||
| **Local origin** | `local.rs` | 51,68 | readdir entry + result |
|
||||
| **Local origin** | `local.rs` | 88-91,112 | read entry + result |
|
||||
| **SMB origin** | `smb.rs` | 86,93 | readdir/read entry + result |
|
||||
| **NFS origin** | `nfs.rs` | 81,89 | readdir/read entry + result |
|
||||
| **Failover** | `failover.rs` | 66,82,87 | Entry + trying origin + success |
|
||||
| **Tree lookup** | `tree.rs` | 124,132 | Entry + result |
|
||||
| **Metadata cache** | `metadata.rs` | 36,40 | lookup + is_fresh entry/result |
|
||||
| **CAS store** | `store.rs` | 70,101 | put/get entry |
|
||||
| **File reader** | `reader.rs` | 66,86 | manifest cache + read entry |
|
||||
| **Search** | `ops/search.rs` | 107,141,182 | readdir_query + readlink + execute_query |
|
||||
| **Search index** | `index.rs` | 98,174 | index_file + search entry |
|
||||
| **Fetcher** | `fetcher.rs` | 54,61,121 | fetch_file entry + meta + ensure_cached |
|
||||
|
||||
**Key DEBUG fields**: `ino`, `parent`, `name`, `offset`, `size`, `bytes_read`, `origin_id`, `path`, `file_id`, `query`, `results_count`, `latency_ms`
|
||||
|
||||
### TRACE Level (100+ locations) - Fine-Grained Flow
|
||||
|
||||
| Category | File | Lines | What to Log |
|
||||
|----------|------|-------|-------------|
|
||||
| **Manifest cache** | `reader.rs` | 67-74 | Cache hit/miss decision |
|
||||
| **Chunk iteration** | `reader.rs` | 107-127 | Each chunk: skip/read boundaries |
|
||||
| **CAS dedup** | `store.rs` | 74-77 | Dedup hit decision |
|
||||
| **CAS integrity** | `store.rs` | 121-134 | Verification result |
|
||||
| **Tree lookup** | `tree.rs` | 118-129 | Path→inode + child lookup |
|
||||
| **Tree parent** | `tree.rs` | 148-153 | Parent resolution path |
|
||||
| **Prefetch event** | `prefetch.rs` | 91-120 | Event type match arms |
|
||||
| **Prefetch semaphore** | `prefetch.rs` | 150-164 | In-flight check + acquire |
|
||||
| **Delta scan** | `delta.rs` | 79-102 | Each file: cached/modified/unchanged/removed |
|
||||
| **Delta entries** | `delta.rs` | 128-146 | Each entry: dir/audio/skip |
|
||||
| **CDC chunking** | `cdc.rs` | 84-93 | Each chunk: offset/length/hash |
|
||||
| **Failover origin** | `failover.rs` | 68-93 | Each origin attempt result |
|
||||
| **Failover retry** | `failover.rs` | 107-122 | Each retry: attempt/success/delay |
|
||||
| **Router select** | `router.rs` | 79-108 | Each candidate + selection reason |
|
||||
| **FUSE node→attr** | `filesystem.rs` | 109-145 | Directory vs file conversion |
|
||||
| **FUSE lookup** | `filesystem.rs` | 192-200 | Found/not found |
|
||||
| **FUSE readdir** | `filesystem.rs` | 274-291 | Each child entry |
|
||||
| **FUSE read** | `filesystem.rs` | 340-367 | file_id resolution + result |
|
||||
| **Metadata tag** | `parser.rs` | 86-100 | Each tag extraction |
|
||||
| **Health transition** | `health.rs` | 199-237 | State transition details |
|
||||
| **Latency recording** | `router.rs` | 23-42 | Stats update per sample |
|
||||
|
||||
**Key TRACE patterns**:
|
||||
- Every `match` arm: `trace!("match arm: {variant}")`
|
||||
- Every `if/else`: `trace!("branch: {condition}={value}")`
|
||||
- Every loop iteration: `trace!("iteration {i}/{total}: ...")`
|
||||
- Every cache lookup: `trace!("cache lookup key={key}, hit={hit}")`
|
||||
|
||||
---
|
||||
|
||||
## gRPC Handler Instrumentation (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: 8/10 gRPC handlers had no logging.
|
||||
|
||||
### server.rs - All Handlers
|
||||
|
||||
| Handler | Line | Level | Log Message |
|
||||
|---------|------|-------|-------------|
|
||||
| `get_status()` | 209 | DEBUG | `debug!("gRPC get_status called")` |
|
||||
| `get_cache_stats()` | 241 | DEBUG | `debug!("gRPC get_cache_stats called")` |
|
||||
| `clear_cache()` | 278 | INFO | `info!("gRPC clear_cache: clearing {tier}")` |
|
||||
| `prefetch()` | 296 | DEBUG | `debug!(file_count = paths.len(), "gRPC prefetch started")` |
|
||||
| `list_origins()` | 322 | DEBUG | `debug!("gRPC list_origins called")` |
|
||||
| `get_origin_health()` | 329 | DEBUG | `debug!(origin_id = %id, "gRPC get_origin_health")` |
|
||||
| `rescan_origin()` | 337 | INFO | `info!(origin_id = %id, "gRPC rescan_origin started")` |
|
||||
| `subscribe_events()` | 376 | INFO | `info!("gRPC subscribe_events: client connected")` |
|
||||
| `shutdown()` | 402 | INFO | `info!(graceful = graceful, "gRPC shutdown requested")` |
|
||||
|
||||
### search_service.rs
|
||||
|
||||
| Handler | Line | Level | Log Message |
|
||||
|---------|------|-------|-------------|
|
||||
| `search()` | entry | DEBUG | `debug!(query = %q, limit = limit, "gRPC search")` |
|
||||
| `search()` | result | DEBUG | `debug!(results = results.len(), "gRPC search completed")` |
|
||||
|
||||
### Pattern: Use `#[instrument]` on all handlers
|
||||
|
||||
```rust
|
||||
#[tracing::instrument(level = "debug", skip(self, request), fields(method = "get_status"))]
|
||||
async fn get_status(&self, request: Request<()>) -> Result<Response<StatusResponse>, Status> {
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Async Task Spawn Instrumentation (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: 14 `tokio::spawn` sites need correlation IDs and span propagation.
|
||||
|
||||
### Spawn Sites Requiring Instrumentation
|
||||
|
||||
| File | Line | Task | Instrumentation |
|
||||
|------|------|------|-----------------|
|
||||
| `server.rs` | 305 | prefetch stream | `spawn(async { ... }.instrument(info_span!("prefetch_stream")))` |
|
||||
| `server.rs` | 354 | rescan stream | `spawn(async { ... }.instrument(info_span!("rescan_stream", origin_id = %id)))` |
|
||||
| `server.rs` | 384 | subscribe events | `spawn(async { ... }.instrument(info_span!("event_subscriber")))` |
|
||||
| `search_service.rs` | spawn | search task | `spawn(async { ... }.instrument(debug_span!("search_task", query = %q)))` |
|
||||
| `indexer.rs` | spawn | indexer loop | `spawn(async { ... }.instrument(info_span!("indexer")))` |
|
||||
| `prefetch.rs` | 87 | prefetch engine | `spawn(async { ... }.instrument(info_span!("prefetch_engine")))` |
|
||||
| `prefetch.rs` | 169 | prefetch file | `spawn(async { ... }.instrument(debug_span!("prefetch_file", file_id = ?id)))` |
|
||||
| `health.rs` | 154 | health monitor | `spawn(async { ... }.instrument(info_span!("health_monitor")))` |
|
||||
| `watcher.rs` | 34 | file watcher | `spawn(async { ... }.instrument(info_span!("file_watcher", origin_id = %id)))` |
|
||||
| `artwork.rs` | spawn | image decode | `spawn_blocking(|| { ... })` - add span before spawn |
|
||||
|
||||
### Pattern: Span Propagation
|
||||
|
||||
```rust
|
||||
use tracing::Instrument;
|
||||
|
||||
// BEFORE (loses context)
|
||||
tokio::spawn(async move {
|
||||
do_work().await;
|
||||
});
|
||||
|
||||
// AFTER (preserves correlation)
|
||||
let span = tracing::info_span!("task_name", task_id = %id);
|
||||
tokio::spawn(async move {
|
||||
do_work().await;
|
||||
}.instrument(span));
|
||||
```
|
||||
|
||||
### Add to init_logging() for request IDs
|
||||
|
||||
```rust
|
||||
// Generate request ID for correlation
|
||||
use tracing::Span;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn with_request_id<F, R>(f: F) -> R
|
||||
where F: FnOnce() -> R {
|
||||
let request_id = Uuid::new_v4();
|
||||
let span = tracing::info_span!("request", request_id = %request_id);
|
||||
span.in_scope(f)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Operation Logging (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: Zero logging for rusqlite operations in db.rs, collections.rs, patterns.rs, artwork.rs.
|
||||
|
||||
### db.rs - Core Database
|
||||
|
||||
| Function | Line | Level | Log Message |
|
||||
|----------|------|-------|-------------|
|
||||
| `open()` | entry | INFO | `info!(path = ?path, "Opening metadata database")` |
|
||||
| `open()` | success | INFO | `info!(file_count = count, "Database opened")` |
|
||||
| `upsert_file()` | entry | DEBUG | `debug!(file_id = ?id, path = %path, "Upserting file")` |
|
||||
| `upsert_file()` | error | ERROR | `error!(file_id = ?id, error = %e, "Failed to upsert file")` |
|
||||
| `get_file_by_id()` | miss | TRACE | `trace!(file_id = ?id, "File not found in db")` |
|
||||
| `delete_file()` | entry | DEBUG | `debug!(file_id = ?id, "Deleting file from db")` |
|
||||
| `list_files_by_origin()` | result | DEBUG | `debug!(origin_id = %id, count = files.len(), "Listed files")` |
|
||||
|
||||
### collections.rs
|
||||
|
||||
| Function | Line | Level | Log Message |
|
||||
|----------|------|-------|-------------|
|
||||
| `create()` | entry | INFO | `info!(name = %name, "Creating collection")` |
|
||||
| `save()` | error | WARN | `warn!(name = %name, error = %e, "Failed to save collection")` |
|
||||
| `delete()` | entry | INFO | `info!(name = %name, "Deleting collection")` |
|
||||
| `list()` | result | DEBUG | `debug!(count = collections.len(), "Listed collections")` |
|
||||
|
||||
### patterns.rs - Access Patterns
|
||||
|
||||
| Function | Line | Level | Log Message |
|
||||
|----------|------|-------|-------------|
|
||||
| `record_access()` | entry | TRACE | `trace!(file_id = ?id, "Recording access pattern")` |
|
||||
| `predict_next()` | result | DEBUG | `debug!(predictions = preds.len(), "Predicted next files")` |
|
||||
|
||||
### artwork.rs
|
||||
|
||||
| Function | Line | Level | Log Message |
|
||||
|----------|------|-------|-------------|
|
||||
| `store()` | entry | DEBUG | `debug!(file_id = ?id, size_bytes = data.len(), "Storing artwork")` |
|
||||
| `get()` | hit/miss | TRACE | `trace!(file_id = ?id, found = found, "Artwork lookup")` |
|
||||
|
||||
### Pattern: Database Error Wrapper
|
||||
|
||||
```rust
|
||||
// Add to musicfs-cache/src/db.rs
|
||||
fn log_db_result<T>(op: &str, result: Result<T, rusqlite::Error>) -> Result<T, Error> {
|
||||
match result {
|
||||
Ok(v) => {
|
||||
tracing::trace!(op = op, "db operation succeeded");
|
||||
Ok(v)
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!(op = op, error = %e, "db operation failed");
|
||||
Err(Error::Database(e.to_string()))
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Channel Operation Logging (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: No logging for channel capacity, close, or broadcast lag.
|
||||
|
||||
### Channel Locations
|
||||
|
||||
| File | Type | Log Points |
|
||||
|------|------|------------|
|
||||
| `events.rs` | broadcast | Lag warning when receiver falls behind |
|
||||
| `watcher.rs` | mpsc | Channel close on watcher shutdown |
|
||||
| `server.rs` | mpsc | gRPC stream channel capacity |
|
||||
| `indexer.rs` | mpsc | Event queue depth |
|
||||
| `health.rs` | mpsc | Health check channel |
|
||||
|
||||
### Patterns
|
||||
|
||||
```rust
|
||||
// Broadcast lag detection (events.rs)
|
||||
match rx.recv().await {
|
||||
Ok(event) => { /* handle */ }
|
||||
Err(broadcast::error::RecvError::Lagged(n)) => {
|
||||
tracing::warn!(skipped = n, "Event subscriber lagged, skipped events");
|
||||
}
|
||||
Err(broadcast::error::RecvError::Closed) => {
|
||||
tracing::debug!("Event channel closed");
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Channel capacity warning (before send)
|
||||
if tx.capacity() < 10 {
|
||||
tracing::warn!(remaining = tx.capacity(), "Channel near capacity");
|
||||
}
|
||||
|
||||
// Channel close
|
||||
impl Drop for EventBus {
|
||||
fn drop(&mut self) {
|
||||
tracing::debug!("Event bus shutting down");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Drop Implementation Logging (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: No logging in Drop impls for cleanup verification.
|
||||
|
||||
| File | Type | Log Message |
|
||||
|------|------|-------------|
|
||||
| `manager.rs:276` | `PluginManager` | `debug!("PluginManager dropping, unloading {} plugins", self.plugins.len())` |
|
||||
| `watcher.rs:157` | `WatchHandle` | `trace!(origin_id = %self.origin_id, "WatchHandle dropped")` |
|
||||
| `prefetch.rs` | `PrefetchEngine` | `debug!("PrefetchEngine dropping, {} in-flight", self.in_flight.len())` |
|
||||
| `server.rs` | gRPC server | `info!("gRPC server shutting down")` |
|
||||
|
||||
### Pattern
|
||||
|
||||
```rust
|
||||
impl Drop for PluginManager {
|
||||
fn drop(&mut self) {
|
||||
tracing::debug!(
|
||||
plugin_count = self.plugins.len(),
|
||||
"PluginManager dropping"
|
||||
);
|
||||
// existing cleanup...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Credential Loading (ADDED - Oracle Review)
|
||||
|
||||
**Gap identified**: No logging in credentials.rs::load().
|
||||
|
||||
| Function | Level | Log Message |
|
||||
|----------|-------|-------------|
|
||||
| `load()` entry | DEBUG | `debug!(origin_id = %origin_id, "Loading credentials")` |
|
||||
| `load()` cache hit | TRACE | `trace!(origin_id = %origin_id, "Credential cache hit")` |
|
||||
| `load()` success | INFO | `info!(origin_id = %origin_id, cred_type = %cred.type_name(), "Credential loaded")` |
|
||||
| `load()` not found | DEBUG | `debug!(origin_id = %origin_id, "No credential found")` |
|
||||
| `load()` error | WARN | `warn!(origin_id = %origin_id, error = %e, "Credential load failed")` |
|
||||
|
||||
**SECURITY**: Never log credential values. The existing Debug impl with redaction is correct.
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations (ADDED - Oracle Review)
|
||||
|
||||
### Never Log These
|
||||
|
||||
| Data | Location | Mitigation |
|
||||
|------|----------|------------|
|
||||
| `WebhookConfig.secret` | webhook.rs | Add `#[serde(skip_serializing)]`, use custom Debug |
|
||||
| Credential values | credentials.rs | Already redacted in Debug impl ✓ |
|
||||
| Full file paths with usernames | everywhere | Sanitize `/home/{user}/` → `~/` |
|
||||
| API keys/tokens | config.rs | Mark sensitive fields |
|
||||
|
||||
### Sanitization Helper
|
||||
|
||||
```rust
|
||||
// Add to musicfs-core/src/lib.rs
|
||||
pub fn sanitize_path(path: &Path) -> String {
|
||||
if let Ok(home) = std::env::var("HOME") {
|
||||
path.to_string_lossy()
|
||||
.replace(&home, "~")
|
||||
.to_string()
|
||||
} else {
|
||||
path.to_string_lossy().to_string()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
debug!(path = %sanitize_path(&path), "Reading file");
|
||||
```
|
||||
|
||||
### WebhookConfig Fix
|
||||
|
||||
```rust
|
||||
// webhook.rs - add custom Debug
|
||||
#[derive(Clone, Serialize, Deserialize)]
|
||||
pub struct WebhookConfig {
|
||||
pub url: String,
|
||||
#[serde(skip_serializing)]
|
||||
pub secret: Option<String>, // Never serialize
|
||||
// ...
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for WebhookConfig {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("WebhookConfig")
|
||||
.field("url", &self.url)
|
||||
.field("secret", &self.secret.as_ref().map(|_| "[REDACTED]"))
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations (ADDED - Oracle Review)
|
||||
|
||||
### Hot Path Warnings
|
||||
|
||||
| Path | Risk | Mitigation |
|
||||
|------|------|------------|
|
||||
| `reader.rs` chunk loop | 100s of TRACE logs per seek | Log summary only: `trace!(chunks_read = n, "Read complete")` |
|
||||
| `store.rs` put/get | 1000s during sync | Keep at DEBUG, not TRACE |
|
||||
| `delta.rs` file scan | Log per file during full scan | Use TRACE, batch summaries at DEBUG |
|
||||
| `parser.rs` tag extraction | Many TRACE per file | Sample: log every 100th file |
|
||||
|
||||
### Trace Sampling Config
|
||||
|
||||
```rust
|
||||
// Add to LoggingConfig
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct LoggingConfig {
|
||||
// ... existing fields ...
|
||||
|
||||
/// Sample rate for TRACE logs in hot paths (0.0-1.0, default 1.0)
|
||||
#[serde(default = "default_sample_rate")]
|
||||
pub trace_sample_rate: f32,
|
||||
}
|
||||
|
||||
fn default_sample_rate() -> f32 { 1.0 }
|
||||
|
||||
// Usage in hot paths
|
||||
if rand::random::<f32>() < config.trace_sample_rate {
|
||||
trace!(...);
|
||||
}
|
||||
```
|
||||
|
||||
### Rate-Limited Warnings
|
||||
|
||||
```rust
|
||||
// For repeating warnings during outages (failover.rs)
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
static LAST_FAILOVER_WARN: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
fn warn_rate_limited(origin_id: &str, error: &str) {
|
||||
let now = Instant::now().elapsed().as_secs();
|
||||
let last = LAST_FAILOVER_WARN.load(Ordering::Relaxed);
|
||||
if now - last >= 60 { // Max once per minute
|
||||
LAST_FAILOVER_WARN.store(now, Ordering::Relaxed);
|
||||
warn!(origin_id = %origin_id, error = %error, "Origin failover");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Standardized Field Names (ADDED - Oracle Review)
|
||||
|
||||
Use these consistently across all log statements:
|
||||
|
||||
| Field | Type | Usage |
|
||||
|-------|------|-------|
|
||||
| `origin_id` | String | Origin identifier (not `origin`) |
|
||||
| `file_id` | FileId | File identifier |
|
||||
| `path` | String | Virtual or real path (sanitized) |
|
||||
| `size_bytes` | u64 | Size in bytes (not `size`, `bytes`, `len`) |
|
||||
| `offset` | u64 | Read offset |
|
||||
| `duration_ms` | u64 | Operation duration in milliseconds |
|
||||
| `count` | usize | Generic count |
|
||||
| `attempt` | u32 | Retry attempt number |
|
||||
| `max_attempts` | u32 | Maximum retry attempts |
|
||||
| `error` | impl Display | Error message (not `err`, `e`) |
|
||||
| `request_id` | Uuid | Correlation ID for requests |
|
||||
|
||||
---
|
||||
|
||||
## Instrumentation Patterns (ADDED - Oracle Review)
|
||||
|
||||
### Use `#[instrument(err)]` for Automatic Error Logging
|
||||
|
||||
```rust
|
||||
// BEFORE: Manual error logging
|
||||
pub async fn read(&self, path: &Path) -> Result<Bytes> {
|
||||
match self.inner_read(path).await {
|
||||
Ok(data) => Ok(data),
|
||||
Err(e) => {
|
||||
error!(path = ?path, error = %e, "Read failed");
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// AFTER: Automatic with #[instrument]
|
||||
#[tracing::instrument(level = "debug", skip(self), err)]
|
||||
pub async fn read(&self, path: &Path) -> Result<Bytes> {
|
||||
self.inner_read(path).await
|
||||
}
|
||||
```
|
||||
|
||||
### Span Events vs Regular Logs
|
||||
|
||||
```rust
|
||||
// Regular log - standalone event
|
||||
info!("Operation completed");
|
||||
|
||||
// Span event - attached to current span context
|
||||
tracing::Span::current().record("result", "success");
|
||||
|
||||
// Prefer span events for operation outcomes
|
||||
#[instrument(fields(result))]
|
||||
async fn operation() -> Result<()> {
|
||||
// ... work ...
|
||||
Span::current().record("result", "success");
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixes: Incorrect Line References (ADDED - Oracle Review)
|
||||
|
||||
| File | Issue | Fix |
|
||||
|------|-------|-----|
|
||||
| `webhook.rs:43` | Uses `expect()` (panics) | Replace with `?` + error log |
|
||||
| `webhook.rs:133` | Uses `expect()` (panics) | Replace with `?` + error log |
|
||||
|
||||
```rust
|
||||
// webhook.rs - BEFORE
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(30))
|
||||
.build()
|
||||
.expect("Failed to create HTTP client");
|
||||
|
||||
// webhook.rs - AFTER
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(Duration::from_secs(30))
|
||||
.build()
|
||||
.map_err(|e| {
|
||||
error!(error = %e, "Failed to create webhook HTTP client");
|
||||
WebhookError::ClientInit(e.to_string())
|
||||
})?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Log Levels Guide
|
||||
|
||||
| Level | Use Case | Example |
|
||||
|-------|----------|---------|
|
||||
| `ERROR` | Unrecoverable failures | Mount failed, DB corruption |
|
||||
| `WARN` | Recoverable issues | Origin timeout, retry needed |
|
||||
| `INFO` | Lifecycle events | Service start/stop, health change |
|
||||
| `DEBUG` | Operation details | Function entry, request params |
|
||||
| `TRACE` | Fine-grained flow | Match arms, cache hit/miss |
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
### Basic Functionality
|
||||
- [ ] Log files created in configured directory
|
||||
- [ ] Daily rotation creates new files at midnight
|
||||
- [ ] JSON output parseable by `jq`
|
||||
- [ ] `journalctl -t musicfs` shows logs
|
||||
- [ ] `RUST_LOG=musicfs=trace` enables trace output
|
||||
- [ ] WorkerGuard kept alive (logs flush on shutdown)
|
||||
- [ ] Logrotate compresses old files
|
||||
|
||||
### Correlation & Context (NEW)
|
||||
- [ ] Request IDs propagate through async tasks
|
||||
- [ ] Spawned task logs include parent span context
|
||||
- [ ] gRPC handler logs show method name in span
|
||||
|
||||
### Security (NEW)
|
||||
- [ ] WebhookConfig.secret never appears in logs
|
||||
- [ ] Credential values never appear in logs
|
||||
- [ ] File paths with `/home/{user}` show as `~/`
|
||||
|
||||
### Performance (NEW)
|
||||
- [ ] TRACE sampling respects `trace_sample_rate` config
|
||||
- [ ] Hot path chunk loops log summary, not per-chunk
|
||||
- [ ] Origin failover warnings are rate-limited (1/minute)
|
||||
- [ ] Database operations log without blocking
|
||||
|
||||
### Database & Channels (NEW)
|
||||
- [ ] Database open logs file count
|
||||
- [ ] Channel capacity warnings appear when queue fills
|
||||
- [ ] Broadcast lag warnings appear when subscriber falls behind
|
||||
- [ ] Drop implementations log cleanup
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Phase | Effort | Deliverables |
|
||||
|-------|--------|--------------|
|
||||
| 1. Config & Dependencies | 2h | LoggingConfig, init_logging(), logrotate, trace sampling |
|
||||
| 2. Core instrumentation | 1h | tracing in musicfs-core, credentials, sanitization |
|
||||
| 3. Hot path instrumentation | 4h | #[instrument] + trace! across 5 crates |
|
||||
| 4. gRPC & async tasks | 2h | Handler instrumentation, spawn correlation |
|
||||
| 5. Database & channels | 2h | rusqlite logging, channel capacity/close |
|
||||
| 6. Production files | 1h | Updated systemd, example config |
|
||||
| **Total** | **12h** | Full observability |
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
### Phase 1: Config & Dependencies
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `Cargo.toml` (workspace) | Add tracing-appender, tracing-journald |
|
||||
| `crates/musicfs-cli/Cargo.toml` | Add dependencies |
|
||||
| `crates/musicfs-core/Cargo.toml` | Add tracing |
|
||||
| `crates/musicfs-core/src/config.rs` | Add LoggingConfig with trace_sample_rate |
|
||||
| `crates/musicfs-cli/src/main.rs` | Expand init_logging(), request ID helper |
|
||||
| `crates/musicfs-core/src/lib.rs` | Add sanitize_path() helper |
|
||||
|
||||
### Phase 2: Core Instrumentation
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `crates/musicfs-core/src/credentials.rs` | Add load() logging (redacted) |
|
||||
| `crates/musicfs-core/src/events.rs` | Add broadcast lag detection |
|
||||
|
||||
### Phase 3: Hot Path Instrumentation
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `crates/musicfs-fuse/src/filesystem.rs` | Add #[instrument], trace! |
|
||||
| `crates/musicfs-origins/src/failover.rs` | Add #[instrument], trace!, rate-limited warn |
|
||||
| `crates/musicfs-origins/src/health.rs` | Add state transition logging |
|
||||
| `crates/musicfs-origins/src/router.rs` | Add selection logging |
|
||||
| `crates/musicfs-cache/src/tree.rs` | Add mutation logging |
|
||||
| `crates/musicfs-cache/src/metadata.rs` | Add hit/miss logging |
|
||||
| `crates/musicfs-cas/src/reader.rs` | Add chunk assembly logging (summary, not per-chunk) |
|
||||
| `crates/musicfs-cas/src/store.rs` | Add dedup logging |
|
||||
| `crates/musicfs-sync/src/delta.rs` | Add change detection logging |
|
||||
|
||||
### Phase 4: gRPC & Async Tasks (NEW)
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `crates/musicfs-grpc/src/server.rs` | Add #[instrument] to all 10 handlers, spawn correlation |
|
||||
| `crates/musicfs-grpc/src/search_service.rs` | Add #[instrument], spawn instrumentation |
|
||||
| `crates/musicfs-grpc/src/webhook.rs` | Fix expect() → error!, custom Debug for secret |
|
||||
| `crates/musicfs-cache/src/prefetch.rs` | Add spawn instrumentation, Drop logging |
|
||||
| `crates/musicfs-search/src/indexer.rs` | Add spawn instrumentation |
|
||||
| `crates/musicfs-sync/src/watcher.rs` | Add spawn instrumentation, Drop logging |
|
||||
| `crates/musicfs-plugins/src/manager.rs` | Add Drop logging |
|
||||
|
||||
### Phase 5: Database & Channels (NEW)
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `crates/musicfs-cache/src/db.rs` | Add log_db_result() helper, open/upsert/query logging |
|
||||
| `crates/musicfs-search/src/collections.rs` | Add CRUD operation logging |
|
||||
| `crates/musicfs-cache/src/patterns.rs` | Add access pattern logging |
|
||||
| `crates/musicfs-cache/src/artwork.rs` | Add store/get logging |
|
||||
|
||||
### Phase 6: Production Files
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `dist/musicfs.service` | Add logging directives |
|
||||
| `dist/logrotate.d/musicfs` | New file |
|
||||
| `dist/config.example.toml` | Add logging section with trace_sample_rate |
|
||||
@@ -1,796 +0,0 @@
|
||||
# Persistent State: Implementation Plan
|
||||
|
||||
**Authors:** AI-assisted
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-13
|
||||
**Reviewers:** TBD
|
||||
**Approvers:** TBD
|
||||
**Prerequisites:** [persistent-state.md](persistent-state.md) (research), [phase-a-stop-dying.md](phase-a-stop-dying.md) (signal handling + shutdown)
|
||||
**Estimated Effort:** ~8 days
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
Wire up the existing SQLite persistence layer into the mount path so that subsequent mounts load from database instead of rescanning origins. This transforms mount time from O(N × origin_latency) to O(N × SQLite_read) — roughly 1000x faster for remote origins.
|
||||
|
||||
**Storage decision: SQLite (Option A).** Rationale:
|
||||
- `Database` struct with full CRUD already exists in `musicfs-cache/src/db.rs`
|
||||
- Schema with `chunk_manifest BLOB` column already exists in `schema.sql`
|
||||
- `ChunkManifest::from_db()` and `chunks_to_bytes()` already exist but are never called
|
||||
- Row-to-`FileMeta` mapping already exists in `get_file_by_virtual_path()`
|
||||
- WAL mode crash safety already configured
|
||||
- 2-4 second bulk load for 1M rows is acceptable (target is <5s, not <500ms — the <500ms target is for the mount syscall itself, which returns immediately with lazy tree loading)
|
||||
|
||||
No new storage engine. No new dependencies. Wire existing code.
|
||||
|
||||
---
|
||||
|
||||
## 2. Background
|
||||
|
||||
### 2.1 Current State
|
||||
|
||||
`run_mount()` in `main.rs`:
|
||||
1. Opens CAS store ✅
|
||||
2. Creates origin connection ✅
|
||||
3. `scan_music_files()` — walks entire origin, parses every file with symphonia ❌ **BOTTLENECK**
|
||||
4. Builds VirtualTree from scan results (in-memory only) ❌ **LOST ON RESTART**
|
||||
5. Registers every file in ContentFetcher (in-memory only) ❌ **LOST ON RESTART**
|
||||
6. Mounts FUSE ✅
|
||||
|
||||
### 2.2 What Exists But Is Not Wired
|
||||
|
||||
| Component | Exists | Wired Into Mount? |
|
||||
|-----------|--------|--------------------|
|
||||
| `Database::open()` + schema + WAL | ✅ | ❌ |
|
||||
| `Database::upsert_file()` | ✅ | ❌ |
|
||||
| `Database::get_file_by_virtual_path()` (returns `FileMeta`) | ✅ | ❌ |
|
||||
| `schema.sql` with `chunk_manifest BLOB` column | ✅ | ❌ |
|
||||
| `ChunkManifest::chunks_to_bytes()` (serialize) | ✅ | ❌ |
|
||||
| `ChunkManifest::from_db()` (deserialize) | ✅ | ❌ |
|
||||
| `TreeBuilder::add_file(&FileMeta)` | ✅ | ✅ (from scan, not from DB) |
|
||||
| `ContentFetcher::register_file(FileMeta)` | ✅ | ✅ (from scan, not from DB) |
|
||||
| `PatternStore::new(db_path)` (loads from SQLite on open) | ✅ | ❌ |
|
||||
| `CollectionStore::new(db_path)` | ✅ | ❌ |
|
||||
| `SearchIndex::open(path)` (opens tantivy from disk) | ✅ | ❌ |
|
||||
|
||||
### 2.3 What's Missing
|
||||
|
||||
| Component | Needs Building |
|
||||
|-----------|----------------|
|
||||
| `Database::list_all_files()` → `Vec<FileMeta>` | New method (SQL exists, just needs `SELECT *`) |
|
||||
| `Database::update_manifest(FileId, &[u8])` | New method (column exists) |
|
||||
| `Database::get_manifest(FileId)` → `Option<Vec<u8>>` | New method |
|
||||
| `Database::list_all_manifests()` → `Vec<(FileId, ChunkManifest)>` | New method |
|
||||
| Background delta sync task | New (compare DB state vs origin) |
|
||||
| First-mount detection | New (check `file_count() > 0`) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1 Goals
|
||||
|
||||
- Subsequent mount loads tree from SQLite, not origin scan
|
||||
- Chunk manifests persist to SQLite, loaded on mount (no re-download)
|
||||
- tantivy index, PatternStore, CollectionStore opened on mount
|
||||
- Background delta sync reconciles DB vs origin after mount
|
||||
- First mount (empty DB) falls back to current full-scan behavior
|
||||
- Mount time for 10K files: <1 second (subsequent mount)
|
||||
- All existing tests pass, no regressions
|
||||
|
||||
### 3.2 Non-Goals
|
||||
|
||||
- Achieving <500ms mount for 1M+ files (requires lazy tree loading — future work)
|
||||
- LRU eviction persistence (separate task, low urgency)
|
||||
- Changing the storage engine (SQLite is the decision)
|
||||
- Config file parsing changes (origin config stays in TOML, not DB)
|
||||
- Schema migrations for existing data (fresh DB on first mount)
|
||||
|
||||
---
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1 Implementation Order
|
||||
|
||||
```
|
||||
4.2 Database: list_all_files() + manifest CRUD (foundation)
|
||||
↓
|
||||
4.3 Mount path: load tree + fetcher from DB (core change)
|
||||
↓
|
||||
4.4 Persist manifests after fetch (write path)
|
||||
↓
|
||||
4.5 Open tantivy + PatternStore + CollectionStore (quick wiring)
|
||||
↓
|
||||
4.6 Background delta sync (post-mount reconciliation)
|
||||
↓
|
||||
4.7 First-mount detection + fallback (edge case)
|
||||
↓
|
||||
4.8 Shutdown: WAL checkpoint + flush (cleanup)
|
||||
```
|
||||
|
||||
### 4.2 Database: New Methods
|
||||
|
||||
**File**: `musicfs-cache/src/db.rs`
|
||||
|
||||
#### list_all_files()
|
||||
|
||||
Bulk load all files from DB. Reuses the existing row-to-FileMeta mapping from `get_file_by_virtual_path()`.
|
||||
|
||||
```rust
|
||||
pub fn list_all_files(&self) -> Result<Vec<FileMeta>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
|
||||
let mut stmt = conn.prepare(
|
||||
r#"SELECT id, origin_id, real_path, virtual_path,
|
||||
title, artist, album, album_artist, genre,
|
||||
year, track, disc,
|
||||
duration_ms, bitrate, sample_rate, format,
|
||||
origin_mtime, origin_size, content_hash
|
||||
FROM files
|
||||
ORDER BY virtual_path"#
|
||||
).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?;
|
||||
|
||||
let files = stmt.query_map([], |row| {
|
||||
// Same mapping as get_file_by_virtual_path
|
||||
Ok(Self::row_to_file_meta(row))
|
||||
})
|
||||
.map_err(|e| Error::Database(format!("query failed: {}", e)))?
|
||||
.filter_map(|r| r.ok())
|
||||
.collect();
|
||||
|
||||
Ok(files)
|
||||
}
|
||||
```
|
||||
|
||||
Extract the row mapping into a shared `row_to_file_meta(row)` helper to avoid duplication with `get_file_by_virtual_path()`.
|
||||
|
||||
#### Manifest CRUD
|
||||
|
||||
```rust
|
||||
pub fn update_manifest(&self, file_id: FileId, manifest_blob: &[u8]) -> Result<()> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
conn.execute(
|
||||
"UPDATE files SET chunk_manifest = ?1 WHERE id = ?2",
|
||||
params![manifest_blob, file_id.0],
|
||||
).map_err(|e| Error::Database(format!("update manifest failed: {}", e)))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn get_manifest(&self, file_id: FileId) -> Result<Option<Vec<u8>>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
conn.query_row(
|
||||
"SELECT chunk_manifest FROM files WHERE id = ?1",
|
||||
params![file_id.0],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()
|
||||
.map_err(|e| Error::Database(format!("get manifest failed: {}", e)))
|
||||
}
|
||||
|
||||
pub fn list_all_manifests(&self) -> Result<Vec<(FileId, u64, i64, Vec<u8>)>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, origin_size, origin_mtime, chunk_manifest FROM files WHERE chunk_manifest IS NOT NULL"
|
||||
).map_err(|e| Error::Database(format!("prepare failed: {}", e)))?;
|
||||
|
||||
let manifests = stmt.query_map([], |row| {
|
||||
Ok((
|
||||
FileId(row.get(0)?),
|
||||
row.get::<_, i64>(1)? as u64,
|
||||
row.get::<_, i64>(2)?,
|
||||
row.get::<_, Vec<u8>>(3)?,
|
||||
))
|
||||
})
|
||||
.map_err(|e| Error::Database(format!("query failed: {}", e)))?
|
||||
.filter_map(|r| r.ok())
|
||||
.collect();
|
||||
|
||||
Ok(manifests)
|
||||
}
|
||||
```
|
||||
|
||||
#### WAL Checkpoint
|
||||
|
||||
```rust
|
||||
pub fn checkpoint(&self) -> Result<()> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
conn.execute_batch("PRAGMA wal_checkpoint(TRUNCATE)")
|
||||
.map_err(|e| Error::Database(format!("WAL checkpoint failed: {}", e)))?;
|
||||
info!("SQLite WAL checkpoint completed");
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
#### Tests
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_list_all_files() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
// Insert 3 files
|
||||
// list_all_files() returns 3
|
||||
// Verify FileMeta fields match what was inserted
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_manifest_roundtrip() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
// Insert file, update_manifest with blob, get_manifest returns same blob
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_list_all_manifests_skips_null() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
// Insert 3 files, only 1 with manifest
|
||||
// list_all_manifests() returns 1
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Mount Path: Load From DB
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()`
|
||||
|
||||
The key change: replace `scan_music_files()` with DB load when data exists.
|
||||
|
||||
```rust
|
||||
fn run_mount(mountpoint: PathBuf, origin_path: Option<PathBuf>, cache_dir: Option<PathBuf>) -> Result<()> {
|
||||
let origin_path = origin_path.context("--origin is required")?;
|
||||
let runtime = tokio::runtime::Runtime::new()?;
|
||||
let handle = runtime.handle().clone();
|
||||
|
||||
let (tree, reader, db) = runtime.block_on(async {
|
||||
let cache_dir = resolve_cache_dir(cache_dir);
|
||||
std::fs::create_dir_all(&cache_dir)?;
|
||||
std::fs::create_dir_all(&mountpoint)?;
|
||||
|
||||
// Open CAS store
|
||||
let store = Arc::new(CasStore::open(CasConfig {
|
||||
chunks_dir: cache_dir.join("chunks"),
|
||||
..Default::default()
|
||||
}).await?);
|
||||
|
||||
// Open database
|
||||
let db_path = cache_dir.join("metadata.db");
|
||||
let db = Arc::new(Database::open_with_integrity_check(&db_path)
|
||||
.or_else(|_| Database::open(&db_path))?); // Fallback to normal open if integrity check fails
|
||||
|
||||
let fetcher = Arc::new(ContentFetcher::new(store.clone()));
|
||||
let origin_id = OriginId::from("local");
|
||||
let origin = Arc::new(LocalOrigin::new(origin_id.clone(), origin_path.clone()));
|
||||
fetcher.register_origin(origin);
|
||||
|
||||
// Decide: load from DB or full scan
|
||||
let file_count = db.file_count().unwrap_or(0);
|
||||
|
||||
let files = if file_count > 0 {
|
||||
// SUBSEQUENT MOUNT — load from DB
|
||||
info!(file_count, "Loading metadata from database");
|
||||
let start = Instant::now();
|
||||
let files = db.list_all_files()?;
|
||||
info!(elapsed_ms = start.elapsed().as_millis() as u64, "Database load complete");
|
||||
files
|
||||
} else {
|
||||
// FIRST MOUNT — full origin scan
|
||||
info!("First mount: scanning origin");
|
||||
let files = scan_music_files(&origin_path, &origin_id).await?;
|
||||
info!(file_count = files.len(), "Scan complete, persisting to database");
|
||||
|
||||
// Persist to DB for next mount
|
||||
for file in &files {
|
||||
if let Some(ref audio) = file.audio {
|
||||
db.upsert_file(
|
||||
&file.real_path.origin_id,
|
||||
&file.real_path.path,
|
||||
&file.virtual_path,
|
||||
audio,
|
||||
file.mtime,
|
||||
file.size,
|
||||
)?;
|
||||
}
|
||||
}
|
||||
info!("Metadata persisted to database");
|
||||
files
|
||||
};
|
||||
|
||||
// Build tree + register files (same as before, but from DB or scan)
|
||||
let mut builder = TreeBuilder::new();
|
||||
for file in &files {
|
||||
builder.add_file(file);
|
||||
fetcher.register_file(file.clone());
|
||||
}
|
||||
let tree = Arc::new(RwLock::new(builder.build()));
|
||||
|
||||
// Load manifests from DB
|
||||
let reader = Arc::new(FileReader::with_fetcher(store, fetcher));
|
||||
let manifest_count = load_manifests_from_db(&db, &reader)?;
|
||||
if manifest_count > 0 {
|
||||
info!(manifest_count, "Loaded chunk manifests from database");
|
||||
}
|
||||
|
||||
Ok::<_, anyhow::Error>((tree, reader, db))
|
||||
})?;
|
||||
|
||||
// Open search index
|
||||
let search_dir = cache_dir.join("search.idx");
|
||||
let _search_index = SearchIndex::open_with_recovery(&search_dir)
|
||||
.context("Failed to open search index")?;
|
||||
|
||||
// Open pattern store
|
||||
let patterns_path = cache_dir.join("patterns.db");
|
||||
let _pattern_store = PatternStore::new(&patterns_path, 30)
|
||||
.context("Failed to open pattern store")?;
|
||||
|
||||
// ... mount, signal handler, shutdown (same as current) ...
|
||||
|
||||
// On shutdown: checkpoint WAL
|
||||
db.checkpoint().unwrap_or_else(|e| warn!("WAL checkpoint failed: {}", e));
|
||||
}
|
||||
```
|
||||
|
||||
Helper function:
|
||||
|
||||
```rust
|
||||
fn load_manifests_from_db(db: &Database, reader: &FileReader) -> Result<usize> {
|
||||
let manifests = db.list_all_manifests()?;
|
||||
let mut count = 0;
|
||||
for (file_id, total_size, mtime, blob) in manifests {
|
||||
if let Some(manifest) = ChunkManifest::from_db(file_id, total_size, mtime, &blob) {
|
||||
reader.register_manifest(manifest);
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Persist Manifests After Fetch
|
||||
|
||||
**File**: `musicfs-cas/src/fetcher.rs`
|
||||
|
||||
After `fetch_file()` downloads and chunks a file, persist the manifest to SQLite.
|
||||
|
||||
The fetcher currently doesn't have access to the Database. Two options:
|
||||
1. Pass `Arc<Database>` to ContentFetcher (adds dependency musicfs-cas → musicfs-cache)
|
||||
2. Emit an event with the manifest, have the caller persist it
|
||||
|
||||
**Approach**: Option 2 — use the existing EventBus. Add a new event variant:
|
||||
|
||||
**File**: `musicfs-core/src/events.rs`
|
||||
|
||||
```rust
|
||||
pub enum Event {
|
||||
// ... existing variants
|
||||
ManifestCached {
|
||||
file_id: FileId,
|
||||
manifest_blob: Vec<u8>,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**File**: `musicfs-cas/src/fetcher.rs` — emit event after fetch:
|
||||
|
||||
```rust
|
||||
pub async fn fetch_file(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
|
||||
// ... existing fetch + chunk logic ...
|
||||
|
||||
// Emit manifest for persistence
|
||||
if let Some(bus) = &self.event_bus {
|
||||
bus.publish(Event::ManifestCached {
|
||||
file_id,
|
||||
manifest_blob: manifest.chunks_to_bytes(),
|
||||
});
|
||||
}
|
||||
|
||||
Ok(manifest)
|
||||
}
|
||||
```
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — subscribe to ManifestCached events:
|
||||
|
||||
```rust
|
||||
// Spawn manifest persistence listener
|
||||
let db_for_manifests = db.clone();
|
||||
let mut manifest_rx = event_bus.subscribe();
|
||||
tokio::spawn(async move {
|
||||
while let Ok(event) = manifest_rx.recv().await {
|
||||
if let Event::ManifestCached { file_id, manifest_blob } = event {
|
||||
if let Err(e) = db_for_manifests.update_manifest(file_id, &manifest_blob) {
|
||||
warn!(file_id = ?file_id, error = %e, "Failed to persist manifest");
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Open tantivy + PatternStore + CollectionStore
|
||||
|
||||
These already have `open()` methods that load from disk. Just call them in the mount path.
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs`
|
||||
|
||||
```rust
|
||||
// After tree is built, before FUSE mount
|
||||
|
||||
// Search index
|
||||
let search_dir = cache_dir.join("search.idx");
|
||||
let search_index = Arc::new(
|
||||
SearchIndex::open_with_recovery(&search_dir)
|
||||
.unwrap_or_else(|e| {
|
||||
warn!("Search index failed, creating fresh: {}", e);
|
||||
SearchIndex::open(&search_dir).expect("Failed to create search index")
|
||||
})
|
||||
);
|
||||
|
||||
// Pattern store (already persists to SQLite, loads sequence_counts on open)
|
||||
let patterns_path = cache_dir.join("patterns.db");
|
||||
let pattern_store = Arc::new(
|
||||
PatternStore::new(&patterns_path, 30)
|
||||
.unwrap_or_else(|e| {
|
||||
warn!("Pattern store failed: {}", e);
|
||||
PatternStore::new(&patterns_path, 30).expect("Failed to create pattern store")
|
||||
})
|
||||
);
|
||||
|
||||
// Collection store
|
||||
let collections_path = cache_dir.join("collections.db");
|
||||
let collection_store = Arc::new(
|
||||
CollectionStore::new(&collections_path)
|
||||
.unwrap_or_else(|e| {
|
||||
warn!("Collection store failed: {}", e);
|
||||
CollectionStore::new(&collections_path).expect("Failed to create collection store")
|
||||
})
|
||||
);
|
||||
```
|
||||
|
||||
For tantivy: if this is a first mount, index all files after scan:
|
||||
|
||||
```rust
|
||||
if file_count == 0 {
|
||||
// First mount — index all files
|
||||
info!("First mount: building search index");
|
||||
let indexer = Indexer::new(search_index.clone(), event_bus.clone(), /* metadata_lookup */);
|
||||
indexer.index_batch(&files)?;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Background Delta Sync
|
||||
|
||||
After mount completes, spawn a background task that compares DB state against origin and reconciles differences.
|
||||
|
||||
**File**: `musicfs-sync/src/delta.rs` or new `musicfs-cli/src/sync.rs`
|
||||
|
||||
```rust
|
||||
pub async fn background_delta_sync(
|
||||
origin: Arc<dyn Origin>,
|
||||
origin_id: OriginId,
|
||||
db: Arc<Database>,
|
||||
tree: Arc<RwLock<VirtualTree>>,
|
||||
fetcher: Arc<ContentFetcher>,
|
||||
event_bus: Arc<EventBus>,
|
||||
) -> Result<SyncSummary> {
|
||||
info!("Starting background delta sync");
|
||||
let start = Instant::now();
|
||||
|
||||
let mut added = 0u64;
|
||||
let mut modified = 0u64;
|
||||
let mut removed = 0u64;
|
||||
let mut unchanged = 0u64;
|
||||
|
||||
// Get all files currently in DB
|
||||
let db_files: HashMap<PathBuf, FileMeta> = db.list_all_files()?
|
||||
.into_iter()
|
||||
.map(|f| (f.real_path.path.clone(), f))
|
||||
.collect();
|
||||
|
||||
// Walk origin
|
||||
let origin_files = scan_origin_recursive(&origin, Path::new("/")).await?;
|
||||
|
||||
// Compare
|
||||
for (path, origin_stat) in &origin_files {
|
||||
match db_files.get(path) {
|
||||
Some(db_file) if db_file.mtime == origin_stat.mtime && db_file.size == origin_stat.size => {
|
||||
unchanged += 1;
|
||||
}
|
||||
Some(db_file) => {
|
||||
// Modified — re-parse metadata, update DB, update tree
|
||||
modified += 1;
|
||||
// ... update logic ...
|
||||
}
|
||||
None => {
|
||||
// New file — parse metadata, add to DB + tree
|
||||
added += 1;
|
||||
// ... add logic ...
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Find removed files (in DB but not on origin)
|
||||
let origin_paths: HashSet<_> = origin_files.keys().collect();
|
||||
for (path, db_file) in &db_files {
|
||||
if !origin_paths.contains(path) {
|
||||
removed += 1;
|
||||
db.delete_file(db_file.id)?;
|
||||
tree.write().remove_file(&db_file.virtual_path);
|
||||
}
|
||||
}
|
||||
|
||||
let elapsed = start.elapsed();
|
||||
info!(
|
||||
added, modified, removed, unchanged,
|
||||
elapsed_ms = elapsed.as_millis() as u64,
|
||||
"Delta sync complete"
|
||||
);
|
||||
|
||||
Ok(SyncSummary { added, modified, removed, unchanged })
|
||||
}
|
||||
```
|
||||
|
||||
Spawn in `run_mount()` after FUSE mount:
|
||||
|
||||
```rust
|
||||
// Background delta sync (non-blocking)
|
||||
let sync_db = db.clone();
|
||||
let sync_tree = tree.clone();
|
||||
let sync_fetcher = fetcher.clone();
|
||||
let sync_origin = origin.clone();
|
||||
let sync_origin_id = origin_id.clone();
|
||||
let sync_bus = event_bus.clone();
|
||||
tokio::spawn(async move {
|
||||
if let Err(e) = background_delta_sync(
|
||||
sync_origin, sync_origin_id, sync_db, sync_tree, sync_fetcher, sync_bus,
|
||||
).await {
|
||||
warn!("Delta sync failed: {}", e);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.7 First-Mount Detection
|
||||
|
||||
Simple: check `db.file_count()`:
|
||||
|
||||
```rust
|
||||
let file_count = db.file_count().unwrap_or(0);
|
||||
|
||||
if file_count > 0 {
|
||||
// Load from DB
|
||||
} else {
|
||||
// Full scan + persist
|
||||
}
|
||||
```
|
||||
|
||||
This is already shown in Section 4.3. No separate implementation step.
|
||||
|
||||
---
|
||||
|
||||
### 4.8 Shutdown: WAL Checkpoint + Flush
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — in the shutdown sequence (after signal, before dropping session):
|
||||
|
||||
```rust
|
||||
info!("Beginning ordered shutdown");
|
||||
shutdown_token.cancel();
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
|
||||
// Flush persistence
|
||||
if let Err(e) = db.checkpoint() {
|
||||
warn!("SQLite WAL checkpoint failed: {}", e);
|
||||
}
|
||||
info!("Background tasks stopped, state flushed");
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1 Security & Privacy
|
||||
|
||||
- No new attack surface — SQLite file has same permissions as cache directory
|
||||
- Metadata in DB is the same as what's already in the FUSE virtual tree (not new data)
|
||||
- `chunk_manifest` BLOB is binary chunk hashes — not sensitive
|
||||
|
||||
### 5.2 Observability
|
||||
|
||||
- Mount time logged: "Loading metadata from database" with elapsed_ms
|
||||
- First-mount detected and logged: "First mount: scanning origin"
|
||||
- Delta sync summary logged: added/modified/removed/unchanged counts + elapsed
|
||||
- WAL checkpoint logged on shutdown
|
||||
- Manifest persistence failures logged at WARN (non-fatal)
|
||||
|
||||
### 5.3 Scalability
|
||||
|
||||
| Library Size | First Mount (scan) | Subsequent Mount (DB load) |
|
||||
|---|---|---|
|
||||
| 1K files | ~1-2s | <100ms |
|
||||
| 10K files | ~10-20s | ~200ms |
|
||||
| 100K files | ~2-5 min | ~1-2s |
|
||||
| 1M files | ~20-60 min | ~2-4s |
|
||||
|
||||
Delta sync runs in background — mount returns immediately, user sees stale-but-functional data while sync catches up.
|
||||
|
||||
### 5.4 Testing
|
||||
|
||||
```rust
|
||||
// Test: subsequent mount loads from DB
|
||||
#[tokio::test]
|
||||
async fn test_mount_loads_from_db() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let db = Database::open(dir.path().join("test.db")).unwrap();
|
||||
|
||||
// Insert files
|
||||
for i in 0..100 {
|
||||
db.upsert_file(/* ... */).unwrap();
|
||||
}
|
||||
|
||||
// Load all
|
||||
let files = db.list_all_files().unwrap();
|
||||
assert_eq!(files.len(), 100);
|
||||
|
||||
// Build tree from DB files (same as mount path)
|
||||
let mut builder = TreeBuilder::new();
|
||||
for f in &files { builder.add_file(f); }
|
||||
let tree = builder.build();
|
||||
assert_eq!(tree.file_count(), 100);
|
||||
}
|
||||
|
||||
// Test: manifest roundtrip through DB
|
||||
#[tokio::test]
|
||||
async fn test_manifest_persists_and_loads() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let db = Database::open(dir.path().join("test.db")).unwrap();
|
||||
|
||||
let id = db.upsert_file(/* ... */).unwrap();
|
||||
|
||||
let manifest = ChunkManifest { /* ... */ };
|
||||
let blob = manifest.chunks_to_bytes();
|
||||
db.update_manifest(id, &blob).unwrap();
|
||||
|
||||
let loaded = db.get_manifest(id).unwrap().unwrap();
|
||||
let restored = ChunkManifest::from_db(id, 1000, 0, &loaded).unwrap();
|
||||
assert_eq!(restored.chunks.len(), manifest.chunks.len());
|
||||
}
|
||||
|
||||
// Test: first mount detects empty DB
|
||||
#[tokio::test]
|
||||
async fn test_first_mount_detection() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let db = Database::open(dir.path().join("test.db")).unwrap();
|
||||
assert_eq!(db.file_count().unwrap(), 0); // First mount
|
||||
}
|
||||
|
||||
// Test: delta sync detects changes
|
||||
#[tokio::test]
|
||||
async fn test_delta_sync_detects_added_file() {
|
||||
// DB has files A, B
|
||||
// Origin has files A, B, C
|
||||
// Delta sync should detect C as added
|
||||
}
|
||||
|
||||
// Test: delta sync detects removed file
|
||||
#[tokio::test]
|
||||
async fn test_delta_sync_detects_removed_file() {
|
||||
// DB has files A, B, C
|
||||
// Origin has files A, B
|
||||
// Delta sync should detect C as removed
|
||||
}
|
||||
|
||||
// Test: shutdown checkpoints WAL
|
||||
#[tokio::test]
|
||||
async fn test_shutdown_checkpoints_wal() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let db_path = dir.path().join("test.db");
|
||||
let db = Database::open(&db_path).unwrap();
|
||||
db.upsert_file(/* ... */).unwrap();
|
||||
|
||||
// WAL file should exist
|
||||
let wal_path = db_path.with_extension("db-wal");
|
||||
// After checkpoint, WAL should be truncated
|
||||
db.checkpoint().unwrap();
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1 sled for Tree Storage (Option B)
|
||||
|
||||
sled is faster for bulk key-value reads (~1-2s for 1M entries vs SQLite's ~2-4s). Rejected because:
|
||||
- SQLite code already exists (schema, CRUD, row mapping)
|
||||
- sled would require new serialization layer (bincode/msgpack for FileMeta)
|
||||
- Two persistence engines is more complex
|
||||
- SQLite's 2-4s is acceptable for the target
|
||||
|
||||
### 6.2 Flat File Snapshot (Option C)
|
||||
|
||||
Fastest possible bulk load (<1s via mmap). Rejected because:
|
||||
- No incremental updates — every change rewrites the entire file
|
||||
- At 1M files (~500MB), delta sync triggers a 500MB write for each changed file
|
||||
- No concurrent access safety
|
||||
- No crash recovery for partial writes
|
||||
|
||||
### 6.3 Lazy Tree Loading
|
||||
|
||||
Instead of loading all files into memory on mount, load only the root directories and fetch deeper levels on demand from SQLite. This would achieve true O(1) mount. Deferred because:
|
||||
- Requires significant refactoring of VirtualTree (currently all-in-memory)
|
||||
- SQLite 2-4s load is good enough for production
|
||||
- Can be added later as optimization without changing the persistence layer
|
||||
|
||||
### 6.4 Separate Manifest Store
|
||||
|
||||
Instead of storing manifests in the `files.chunk_manifest` column, use a separate sled tree or SQLite table. Rejected because the column already exists and the schema already supports it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### 7.1 Task Sequence
|
||||
|
||||
| Day | Task | Deliverable |
|
||||
|-----|------|-------------|
|
||||
| 1 | Database methods: `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`. Extract `row_to_file_meta()` helper. | New DB methods + tests |
|
||||
| 2 | Rewrite `run_mount()`: DB load path vs scan path. First-mount detection. | Core mount change |
|
||||
| 3 | Persist manifests: `ManifestCached` event + listener in main.rs. Load manifests on mount via `load_manifests_from_db()`. | Manifest persistence |
|
||||
| 4 | Wire tantivy + PatternStore + CollectionStore into mount path. First-mount indexing. | Search/patterns on mount |
|
||||
| 5 | Background delta sync: compare DB vs origin, update differences. | Delta sync task |
|
||||
| 6 | Shutdown: WAL checkpoint. Upsert files to DB during first-mount scan. | Clean shutdown |
|
||||
| 7 | Integration testing: full mount→read→restart→mount cycle. Verify tree + manifests survive restart. | E2E validation |
|
||||
| 8 | Buffer for issues found during integration. | — |
|
||||
|
||||
### 7.2 Verification Checklist
|
||||
|
||||
- [ ] `cargo check` — zero errors
|
||||
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass
|
||||
- [ ] Manual test: first mount (empty cache dir) — scans origin, creates DB
|
||||
- [ ] Manual test: second mount (DB exists) — loads from DB, no origin scan
|
||||
- [ ] Manual test: add file to origin, restart — delta sync discovers it
|
||||
- [ ] Manual test: `kill -9` daemon, restart — DB loads, manifests intact
|
||||
- [ ] Mount time for 10K test files: <1 second on subsequent mount
|
||||
- [ ] `ls -la ~/.cache/musicfs/metadata.db` exists after first mount
|
||||
|
||||
---
|
||||
|
||||
## 8. Files Changed
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `musicfs-cache/src/db.rs` | `list_all_files()`, `update_manifest()`, `get_manifest()`, `list_all_manifests()`, `checkpoint()`, `row_to_file_meta()` refactor |
|
||||
| `musicfs-core/src/events.rs` | Add `ManifestCached` event variant |
|
||||
| `musicfs-cli/src/main.rs` | Rewrite `run_mount()`: DB load vs scan, open tantivy/patterns/collections, manifest listener, delta sync spawn, shutdown checkpoint |
|
||||
| `musicfs-cli/Cargo.toml` | Add `musicfs-search`, `musicfs-cache` dependencies (for PatternStore, CollectionStore, SearchIndex) |
|
||||
| `musicfs-cas/src/fetcher.rs` | Emit `ManifestCached` event after `fetch_file()` |
|
||||
| `musicfs-sync/src/delta.rs` | New `background_delta_sync()` function (or new file) |
|
||||
| `musicfs-test-utils/tests/resilience.rs` | New tests: mount-from-DB, manifest roundtrip, delta sync, first-mount detection |
|
||||
|
||||
---
|
||||
|
||||
## 9. Glossary / References
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **First mount** | Initial mount with empty database — triggers full origin scan |
|
||||
| **Subsequent mount** | Mount with existing database — loads from SQLite |
|
||||
| **Delta sync** | Background task that compares DB state against origin after mount |
|
||||
| **Stale data window** | Time between mount and delta sync completion when data may be outdated |
|
||||
| **WAL checkpoint** | SQLite operation that flushes write-ahead log to main database file |
|
||||
|
||||
| Document | Path |
|
||||
|----------|------|
|
||||
| Persistent state research | [persistent-state.md](persistent-state.md) |
|
||||
| Phase A (signals, shutdown) | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
|
||||
| Phase B (crash recovery) | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) |
|
||||
| Architecture | [architecture.md](../architecture.md) |
|
||||
@@ -1,353 +0,0 @@
|
||||
# MusicFS Persistent State Plan
|
||||
|
||||
**Date**: 2026-05-13
|
||||
**Status**: Research Complete — Design Decision Needed
|
||||
**Prerequisites**: [architecture.md](../architecture.md), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
|
||||
**Related Requirements**: G1 (O(1) mount time), NFR-1.7 (<500ms mount), FR-7.1 (cache persists across restarts)
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
Every mount is a full cold start. The `run_mount()` function in `main.rs` does not use any persistent storage — it walks the entire origin filesystem, parses metadata from every audio file, and builds all runtime state from scratch.
|
||||
|
||||
The architecture designed persistence infrastructure (SQLite schema, `chunk_manifest` column, `ChunkManifest::from_db()`, `chunks_to_bytes()`) but **none of it is wired into the mount path**. The mount flow doesn't even open the database.
|
||||
|
||||
### Mount Time by Library Size (Current)
|
||||
|
||||
| Library Size | Estimated Mount Time | Target (NFR-1.7) |
|
||||
|---|---|---|
|
||||
| 1K files | ~1-2s | <500ms |
|
||||
| 10K files | ~10-20s | <500ms |
|
||||
| 100K files | ~2-5 minutes | <500ms |
|
||||
| 1M files | ~20-60 minutes | <500ms |
|
||||
| 10M files (stretch) | hours | <500ms |
|
||||
|
||||
---
|
||||
|
||||
## 2. In-Memory State Inventory
|
||||
|
||||
### 2.1 State That Must Survive Restart
|
||||
|
||||
These are the large, expensive-to-rebuild data structures. Losing them forces a full origin rescan.
|
||||
|
||||
#### VirtualTree (~300-400MB at 1M files)
|
||||
|
||||
**Location**: `musicfs-cache/src/tree.rs`
|
||||
|
||||
**Contents**:
|
||||
- `nodes: HashMap<Inode, VirtualNode>` — every directory and file node
|
||||
- `path_to_inode: HashMap<VirtualPath, Inode>` — reverse path lookup
|
||||
- `next_inode: AtomicU64` — inode counter
|
||||
|
||||
**Currently rebuilt from**: Full recursive origin scan + metadata parse of every audio file. This is the single most expensive operation on mount — it touches every file on origin, runs symphonia metadata extraction, and builds the entire tree structure.
|
||||
|
||||
**What's needed**: Load from persistent storage on mount. Rebuild only on first-ever mount or if storage is corrupt.
|
||||
|
||||
---
|
||||
|
||||
#### ContentFetcher.file_meta (~200MB at 1M files)
|
||||
|
||||
**Location**: `musicfs-cas/src/fetcher.rs`
|
||||
|
||||
**Contents**:
|
||||
- `file_meta: RwLock<HashMap<FileId, FileMeta>>` — full metadata for every file
|
||||
- Each `FileMeta` contains: id, virtual_path, real_path (origin_id + path), size, mtime, content_hash, audio metadata
|
||||
|
||||
**Currently rebuilt from**: Same origin scan that builds the tree. Every file is registered via `fetcher.register_file(meta)`.
|
||||
|
||||
**What's needed**: This is essentially a duplicate of the tree data in a different shape. If the tree is loaded from storage, this map should be populated from the same source.
|
||||
|
||||
---
|
||||
|
||||
#### FileReader.manifests (~100MB at 1M files)
|
||||
|
||||
**Location**: `musicfs-cas/src/reader.rs`
|
||||
|
||||
**Contents**:
|
||||
- `manifests: RwLock<HashMap<FileId, ChunkManifest>>` — maps FileId to list of chunk hashes + offsets
|
||||
- Each `ChunkManifest` contains: file_id, total_size, mtime, chunks (Vec<ChunkRef> with hash + offset + size)
|
||||
|
||||
**Currently rebuilt from**: Re-fetched from origin on first `read()` after restart. The fetcher downloads the entire file, chunks it via CDC, stores chunks in CAS (dedup catches existing ones), and builds the manifest. This means every file is re-downloaded once after restart even though the chunks are already on disk.
|
||||
|
||||
**What's needed**: Persist manifests to storage after fetch. Load on mount. This is the difference between "restart = re-download everything" and "restart = instant reads from cache."
|
||||
|
||||
**Existing dead code**: SQLite `files` table has `chunk_manifest BLOB` column. `ChunkManifest::chunks_to_bytes()` and `ChunkManifest::from_db()` exist but are never called.
|
||||
|
||||
---
|
||||
|
||||
#### LruEviction access times (~50MB at 100K chunks)
|
||||
|
||||
**Location**: `musicfs-cache/src/eviction.rs`
|
||||
|
||||
**Contents**:
|
||||
- `access_times: RwLock<BTreeMap<Instant, ChunkHash>>` — ordered by access time
|
||||
- `hash_to_time: RwLock<HashMap<ChunkHash, Instant>>` — reverse lookup
|
||||
|
||||
**Currently rebuilt from**: Nothing. After restart, all chunks have equal eviction priority. The album you're currently listening to is just as likely to be evicted as something you played 6 months ago.
|
||||
|
||||
**What's needed**: Persist last-access timestamps. On mount, load and reconstruct the LRU order so hot data stays cached.
|
||||
|
||||
---
|
||||
|
||||
### 2.2 State That Survives But Is Ignored on Mount
|
||||
|
||||
These persist on disk but `run_mount()` never opens them.
|
||||
|
||||
| Component | Persisted To | Loaded on Mount? | Effect |
|
||||
|---|---|---|---|
|
||||
| SQLite metadata (files table) | `metadata.db` | ❌ | All metadata re-scanned from origin |
|
||||
| tantivy search index | `search.idx/` | ❌ | Index rebuilt from scratch (or not at all) |
|
||||
| PatternStore (access patterns) | SQLite (separate DB) | ❌ | Predictions reset to zero |
|
||||
| CollectionStore (smart collections) | SQLite (same as patterns) | ❌ | Collections unavailable until opened |
|
||||
|
||||
### 2.3 State That Correctly Does Not Need Persistence
|
||||
|
||||
| Component | Why Transient Is Fine |
|
||||
|---|---|
|
||||
| OriginRegistry (origin connections) | Reconstructed from config on startup |
|
||||
| Router (priorities, latency stats) | Priorities from config; latency stats warm up within seconds |
|
||||
| HealthMonitor (health state) | All origins start as Unknown, converge within one check cycle (~30s) |
|
||||
| EventBus (in-flight events) | Transient by nature |
|
||||
| PrefetchEngine.in_flight | Transient work queue |
|
||||
| PluginManager | Re-loaded from config + plugin directories |
|
||||
| MusicFs.query_inodes | Transient search session state |
|
||||
| CasStore.current_size | Recalculated on open (though currently broken — see resilience doc 3.10) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Storage Decision
|
||||
|
||||
### 3.1 Requirements for Persistent State
|
||||
|
||||
1. **Bulk sequential read on mount** — load ~1M records into in-memory structures as fast as possible
|
||||
2. **Incremental updates at runtime** — delta sync adds/removes/modifies individual files
|
||||
3. **Crash safety** — no corruption on unclean shutdown (SIGKILL, power loss)
|
||||
4. **Manifest storage** — binary blobs (msgpack-encoded chunk lists), variable size (100 bytes to 10KB per file)
|
||||
5. **LRU timestamps** — simple key-value (ChunkHash → last_access_timestamp)
|
||||
6. **Already in project** — minimize new dependencies
|
||||
|
||||
### 3.2 Options
|
||||
|
||||
#### Option A: SQLite (Current Architecture Choice)
|
||||
|
||||
**Already in project**: `rusqlite` dependency, `schema.sql` with `files` table, `Database` struct with full CRUD, `chunk_manifest BLOB` column ready.
|
||||
|
||||
| Metric | Performance |
|
||||
|---|---|
|
||||
| Bulk load 1M rows | ~2-4 seconds (WAL mode, indexed) |
|
||||
| Single row upsert | ~50μs |
|
||||
| Crash safety | WAL mode — excellent |
|
||||
| Manifest blobs | Native BLOB support, no size limit |
|
||||
|
||||
**Pros**: Already built (schema, code, tests exist). Well-understood crash semantics. Single file backup. SQL queries for debugging. The `chunk_manifest` column and `from_db()`/`to_bytes()` methods are already written.
|
||||
|
||||
**Cons**: Not the fastest for pure key-value workloads. WAL checkpoint can cause brief write pauses. Single-writer limitation (Mutex around connection).
|
||||
|
||||
**Effort to wire up**: ~5-7 days (mostly connecting existing code, not writing new code)
|
||||
|
||||
---
|
||||
|
||||
#### Option B: sled (Already in Project for CAS Index)
|
||||
|
||||
**Already in project**: Used for CAS chunk hash → location mapping.
|
||||
|
||||
| Metric | Performance |
|
||||
|---|---|
|
||||
| Bulk load 1M entries | ~1-2 seconds (LSM, sequential reads) |
|
||||
| Single entry upsert | ~10-20μs |
|
||||
| Crash safety | Built-in WAL — good |
|
||||
| Manifest blobs | Native byte value support |
|
||||
|
||||
**Pros**: Faster than SQLite for pure key-value. Already a dependency. Good for LRU timestamps (simple k/v).
|
||||
|
||||
**Cons**: No SQL — querying for debugging is harder. No schema migration story. Limited tooling. Has known issues with large datasets (memory usage during compaction). Two persistence engines = two things to maintain.
|
||||
|
||||
**Effort**: ~7-9 days (new serialization layer, no existing code to reuse)
|
||||
|
||||
---
|
||||
|
||||
#### Option C: Flat File (bincode/msgpack dump)
|
||||
|
||||
| Metric | Performance |
|
||||
|---|---|
|
||||
| Bulk load 1M entries | <1 second (mmap, zero-parse with bincode) |
|
||||
| Single entry upsert | N/A — full rewrite required |
|
||||
| Crash safety | Must write atomically (tmp + rename) |
|
||||
| Manifest blobs | Part of serialized struct |
|
||||
|
||||
**Pros**: Fastest possible bulk load. Simplest implementation.
|
||||
|
||||
**Cons**: No incremental updates — every change requires serializing and rewriting the entire file. At 1M files (~500MB serialized), a single file modification triggers a 500MB write. No concurrent access. No recovery from partial corruption.
|
||||
|
||||
**Effort**: ~3-4 days but creates ongoing maintenance burden for delta updates
|
||||
|
||||
---
|
||||
|
||||
#### Option D: Hybrid (SQLite for metadata + sled for hot-path data)
|
||||
|
||||
Use SQLite for structured metadata (files, collections, patterns — already built) and sled for hot-path key-value data (manifests, LRU timestamps — performance-critical).
|
||||
|
||||
**Pros**: Each store optimized for its access pattern. SQLite for queryable metadata, sled for fast blob lookup.
|
||||
|
||||
**Cons**: Two persistence engines to coordinate. Consistency between them on crash. More complex startup/shutdown.
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Recommendation
|
||||
|
||||
**Pending your decision.** The tradeoffs are:
|
||||
- **Simplest path**: Option A (SQLite) — most code already exists, just needs wiring
|
||||
- **Fastest hot-path**: Option D (Hybrid) — but more complexity
|
||||
- **Fastest bulk load**: Option C (Flat file) — but no incremental updates
|
||||
|
||||
The choice depends on what you value most. SQLite at 1M files loads in ~2-4 seconds — is that acceptable vs the <500ms target? If not, a flat file or sled for the tree data with SQLite for everything else might be needed.
|
||||
|
||||
---
|
||||
|
||||
## 4. What Needs to Change
|
||||
|
||||
Regardless of storage choice, these are the code changes needed:
|
||||
|
||||
### 4.1 Mount Path (musicfs-cli/src/main.rs)
|
||||
|
||||
Current `run_mount()` flow:
|
||||
```
|
||||
1. Open CAS store → O(1)
|
||||
2. Create origin connection → O(1)
|
||||
3. scan_music_files() — FULL ORIGIN WALK → O(N × origin_latency) ← BOTTLENECK
|
||||
4. Build tree from scan results → O(N)
|
||||
5. Register files in fetcher → O(N)
|
||||
6. Mount FUSE → O(1)
|
||||
```
|
||||
|
||||
Required flow:
|
||||
```
|
||||
1. Open CAS store → O(1)
|
||||
2. Open persistent state store → O(1)
|
||||
3. IF store has data:
|
||||
Load tree from store → O(N × local_read) ← ~1000x faster
|
||||
Load manifests from store → O(N × local_read)
|
||||
Load LRU access times from store → O(chunks)
|
||||
ELSE (first mount):
|
||||
Full origin scan (current behavior) → O(N × origin_latency)
|
||||
Persist results to store → O(N × local_write)
|
||||
4. Open tantivy search index → O(1)
|
||||
5. Open PatternStore → O(1)
|
||||
6. Create origin connections → O(1)
|
||||
7. Mount FUSE → O(1)
|
||||
8. Background: delta sync (origin vs store) → incremental, non-blocking
|
||||
```
|
||||
|
||||
### 4.2 Runtime Persistence (Write Path)
|
||||
|
||||
These operations must persist state changes as they happen, not just on shutdown:
|
||||
|
||||
| Event | What to Persist | When |
|
||||
|---|---|---|
|
||||
| File discovered during sync | FileMeta → store | Immediately (in batch if scanning) |
|
||||
| File removed during sync | Delete from store | Immediately |
|
||||
| File metadata changed | Update FileMeta in store | Immediately |
|
||||
| File content fetched (cache miss) | ChunkManifest → store | After fetch completes |
|
||||
| Chunk accessed | Update LRU timestamp | Batched (every 10s or 100 accesses) |
|
||||
| Search index updated | tantivy handles its own persistence | On commit (every 5s) |
|
||||
| Access pattern recorded | PatternStore handles its own persistence | Already persisted per-access |
|
||||
|
||||
### 4.3 Files That Need Changes
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `musicfs-cli/src/main.rs` | Rewrite `run_mount()` to load from store; add background delta sync |
|
||||
| `musicfs-cache/src/db.rs` | Add `list_all_files()` bulk load; add manifest read/write methods (if SQLite) |
|
||||
| `musicfs-cache/src/tree.rs` | Add `TreeBuilder::from_file_metas(iter)` — build tree from stored records |
|
||||
| `musicfs-cas/src/reader.rs` | Load manifests from store on startup; persist after fetch |
|
||||
| `musicfs-cas/src/fetcher.rs` | After `fetch_file()`, persist manifest to store |
|
||||
| `musicfs-cache/src/eviction.rs` | Persist access times; load on startup |
|
||||
| `musicfs-search/src/indexer.rs` | On mount, check what's already indexed vs what's in store — skip known files |
|
||||
| `musicfs-sync/src/delta.rs` | Background delta sync: compare store state vs origin, sync differences |
|
||||
|
||||
### 4.4 Shutdown Persistence
|
||||
|
||||
On graceful shutdown (after signal handling from resilience plan Phase A is implemented):
|
||||
|
||||
| Step | What |
|
||||
|---|---|
|
||||
| 1 | Flush any batched LRU timestamp updates |
|
||||
| 2 | Commit tantivy index writer |
|
||||
| 3 | WAL checkpoint SQLite (if SQLite): `PRAGMA wal_checkpoint(TRUNCATE)` |
|
||||
| 4 | Flush sled (if sled): `sled::Db::flush()` |
|
||||
| 5 | Close all database connections |
|
||||
|
||||
On crash (no graceful shutdown):
|
||||
- SQLite WAL mode: automatic recovery on next open (no data loss for committed transactions)
|
||||
- sled: automatic recovery via internal WAL
|
||||
- tantivy: up to 5 seconds of uncommitted documents lost, but recoverable from store
|
||||
- LRU timestamps: batched updates may lose last batch (10s window) — acceptable
|
||||
|
||||
---
|
||||
|
||||
## 5. Background Delta Sync
|
||||
|
||||
After mounting from persistent state, the data may be stale (origin changed while daemon was stopped). A background sync reconciles:
|
||||
|
||||
```
|
||||
1. Walk origin (or use watcher for inotify-capable origins)
|
||||
2. For each file on origin:
|
||||
a. Compare mtime + size against stored record
|
||||
b. If unchanged → skip
|
||||
c. If modified → re-parse metadata, update store, update tree, invalidate manifest
|
||||
d. If new → parse metadata, add to store + tree
|
||||
3. For each file in store not found on origin:
|
||||
a. Remove from store + tree
|
||||
4. Update search index for changed files
|
||||
5. Log summary: "Delta sync complete: N added, M modified, K removed, T unchanged"
|
||||
```
|
||||
|
||||
This runs in the background AFTER mount completes. Users see the filesystem immediately (from stored state), and it converges to current reality within minutes.
|
||||
|
||||
### 5.1 Stale Data Window
|
||||
|
||||
Between mount and delta sync completion, users may see:
|
||||
- Files that were deleted on origin (will get ENOENT or EIO on read — origin returns not found)
|
||||
- Files with old metadata (wrong track name, etc.)
|
||||
- Missing files that were added to origin (won't appear until sync discovers them)
|
||||
|
||||
This is acceptable — it's the same behavior as any cached filesystem (NFS, CIFS). The key insight: **stale data for 30 seconds is infinitely better than no data for 5 minutes.**
|
||||
|
||||
---
|
||||
|
||||
## 6. First Mount vs Subsequent Mount
|
||||
|
||||
| | First Mount (empty store) | Subsequent Mount (store has data) |
|
||||
|---|---|---|
|
||||
| **Tree source** | Origin scan + metadata parse | Load from store |
|
||||
| **Manifests** | None (populated on first read) | Loaded from store |
|
||||
| **Search index** | Built during/after scan | Opened from disk |
|
||||
| **LRU data** | Empty (cold cache) | Loaded from store |
|
||||
| **Mount time** | O(N × origin_latency) — same as today | O(N × local_read) — target <5s for 1M files |
|
||||
| **Accuracy** | 100% current | Stale until delta sync completes |
|
||||
| **Detection** | Store file doesn't exist or is empty | Store file exists with data |
|
||||
|
||||
---
|
||||
|
||||
## 7. Estimated Effort
|
||||
|
||||
| Task | Effort | Depends On |
|
||||
|---|---|---|
|
||||
| Rewrite `run_mount()` with store loading + fallback | 2 days | Storage decision |
|
||||
| Persist chunk manifests after fetch | 1 day | Storage decision |
|
||||
| Load manifests on mount + register in FileReader | 0.5 day | Above |
|
||||
| Open tantivy on mount, skip known files | 1 day | — |
|
||||
| Open PatternStore + CollectionStore on mount | 0.5 day | — |
|
||||
| Background delta sync | 1.5 days | — |
|
||||
| Persist LRU access times + load on mount | 1 day | Storage decision |
|
||||
| First-mount detection + fallback to full scan | 0.5 day | — |
|
||||
| **Total** | **~8 days** | |
|
||||
|
||||
---
|
||||
|
||||
## 8. Open Decision
|
||||
|
||||
**Which storage engine for the persistent state?**
|
||||
|
||||
The answer drives the implementation of every task above. See Section 3 for tradeoffs.
|
||||
@@ -1,569 +0,0 @@
|
||||
# Phase A: Stop Dying — Implementation Plan
|
||||
|
||||
**Authors:** AI-assisted
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-13
|
||||
**Reviewers:** TBD
|
||||
**Approvers:** TBD
|
||||
**Prerequisites:** [resilience-fault-tolerance.md](resilience-fault-tolerance.md), [resilience-testing.md](resilience-testing.md)
|
||||
**Estimated Effort:** ~5 days
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
Implement the 6 most critical resilience fixes (issues 2.1, 2.2, 2.7, 2.9, 2.10, 3.7 from the [resilience audit](resilience-fault-tolerance.md)) that prevent MusicFS from dying on common operational events: signals, panics, lock poisoning, and systemd lifecycle.
|
||||
|
||||
Issues 2.3 (shutdown orchestration), 2.4 (cache integrity), 2.5 (sync recovery), 2.6 (task supervisor), 2.8 (disk space) are deferred to Phase B — they depend on Phase A infrastructure or on the [persistent state](persistent-state.md) work.
|
||||
|
||||
**Development flow** (TDD, per-issue):
|
||||
1. Create stubs so the codebase compiles
|
||||
2. Write RED tests that express the expected behavior
|
||||
3. Implement the fix
|
||||
4. Verify tests turn GREEN
|
||||
5. Run full test suite — no regressions
|
||||
|
||||
---
|
||||
|
||||
## 2. Background
|
||||
|
||||
MusicFS currently dies on:
|
||||
- Any signal (SIGTERM, SIGINT) — instant death, no cleanup
|
||||
- Any panic in a writer thread — RwLock poisons, all FUSE ops crash
|
||||
- systemd lifecycle — `Type=notify` but no `sd_notify`, ExecStop is a stub
|
||||
- Crash leaves stale FUSE mount — users must manually `fusermount -u`
|
||||
|
||||
The [resilience test crate](../../musicfs/crates/musicfs-test-utils/) and RED tests are already in place. This plan implements the fixes to turn them GREEN.
|
||||
|
||||
---
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1 Goals
|
||||
|
||||
- Signal handler catches SIGTERM/SIGINT and initiates clean exit
|
||||
- Panics are logged with full context before process terminates
|
||||
- RwLock poisoning cannot cascade to kill FUSE operations
|
||||
- systemd integration works (`sd_notify READY=1`, `ExecStopPost`)
|
||||
- Stale FUSE mounts are detected and cleaned on startup
|
||||
- All existing 162 tests continue to pass
|
||||
- All Phase A RED tests turn GREEN
|
||||
|
||||
### 3.2 Non-Goals
|
||||
|
||||
- Graceful shutdown orchestration with ordered teardown (Phase B — needs CancellationToken plumbing through all components)
|
||||
- Task supervisor for background task restart (Phase B)
|
||||
- Cache integrity checks on startup (Phase B — needs persistent state)
|
||||
- Disk space monitoring (Phase B)
|
||||
- Interrupted sync recovery (Phase B — needs persistent state)
|
||||
|
||||
---
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1 Implementation Order
|
||||
|
||||
Dependencies determine the order. Each issue is independent except where noted.
|
||||
|
||||
```
|
||||
4.2 RwLock poison fix (no deps, instant win, unblocks safety)
|
||||
↓
|
||||
4.3 Panic hook (no deps, complements RwLock fix)
|
||||
↓
|
||||
4.4 systemd ExecStopPost (no deps, config-only change)
|
||||
↓
|
||||
4.5 sd_notify integration (no deps, new crate dependency)
|
||||
↓
|
||||
4.6 Signal handling (depends on: FUSE mount change to spawn_mount2)
|
||||
↓
|
||||
4.7 Stale mount detection (depends on: signal handling for clean test)
|
||||
```
|
||||
|
||||
### 4.2 Issue 2.9: RwLock Poison Fix
|
||||
|
||||
**Approach**: Replace `std::sync::RwLock` with `parking_lot::RwLock` in all production paths. `parking_lot` never poisons — a panic in a writer releases the lock and subsequent readers see the pre-panic state.
|
||||
|
||||
**Why parking_lot over poison recovery**: The codebase already uses `parking_lot` in `prefetch.rs` and `index.rs`. Using it everywhere is consistent. The alternative (`.unwrap_or_else(|p| p.into_inner())`) is verbose and error-prone — one missed call re-introduces the bug.
|
||||
|
||||
#### Step 1: Stubs (compile)
|
||||
|
||||
None needed — `parking_lot::RwLock` is a drop-in replacement (same API, no `PoisonError`).
|
||||
|
||||
#### Step 2: RED tests
|
||||
|
||||
Already exist in `tests/resilience.rs`:
|
||||
- `test_poisoned_tree_lock_returns_eio_not_panic` — currently passes (demonstrates the problem)
|
||||
- `test_parking_lot_rwlock_survives_panic` — currently passes (proves the fix works)
|
||||
|
||||
Additional test to add: verify FUSE filesystem survives a writer panic on the tree lock.
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**Files to change:**
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `musicfs-fuse/src/filesystem.rs` | `use std::sync::RwLock` → `use parking_lot::RwLock`; remove all `.unwrap()` on lock calls (parking_lot returns guard directly, not `Result`) |
|
||||
| `musicfs-cas/src/reader.rs` | Same change for `manifests: RwLock<HashMap<...>>` |
|
||||
| `musicfs-cas/src/fetcher.rs` | Same change for `origins` and `file_meta` locks |
|
||||
| `musicfs-origins/src/registry.rs` | Same change for `origins` and `watch_handles` locks |
|
||||
| `musicfs-cache/src/eviction.rs` | Same change for `access_times` and `hash_to_time` locks |
|
||||
| `musicfs-core/src/metrics.rs` | Same change for histogram locks |
|
||||
| `musicfs-cache/src/tree.rs` | Same change for `last_refresh` lock |
|
||||
|
||||
**Pattern**: In each file:
|
||||
```rust
|
||||
// BEFORE
|
||||
use std::sync::RwLock;
|
||||
let guard = self.tree.read().unwrap();
|
||||
|
||||
// AFTER
|
||||
use parking_lot::RwLock;
|
||||
let guard = self.tree.read(); // No unwrap needed
|
||||
```
|
||||
|
||||
For the `MusicFs` struct in `filesystem.rs`, the `tree` field is `Arc<RwLock<VirtualTree>>` — this is passed in from `main.rs`. Change `main.rs` to use `parking_lot::RwLock` there too.
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test # All 162+ tests pass
|
||||
cargo test -p musicfs-test-utils # Resilience tests pass
|
||||
cargo check # No warnings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Issue 2.2: Panic Hook
|
||||
|
||||
**Approach**: Install a custom panic hook at daemon startup that logs the panic with `tracing::error!` before the default behavior (abort/unwind). This ensures panics are captured in log files and journald.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
Add to `musicfs-core/src/lib.rs`:
|
||||
```rust
|
||||
pub fn install_panic_hook() {
|
||||
// stub — will be implemented
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2: RED tests
|
||||
|
||||
Write in `tests/resilience.rs`:
|
||||
```rust
|
||||
#[test]
|
||||
fn test_panic_hook_logs_to_tracing() {
|
||||
// Install hook with a test tracing subscriber
|
||||
// Trigger panic via catch_unwind
|
||||
// Verify error! log contains panic message + thread name
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-core/src/lib.rs` (or new `musicfs-core/src/panic.rs`)
|
||||
|
||||
```rust
|
||||
pub fn install_panic_hook() {
|
||||
let default_hook = std::panic::take_hook();
|
||||
std::panic::set_hook(Box::new(move |info| {
|
||||
let thread = std::thread::current();
|
||||
let thread_name = thread.name().unwrap_or("<unnamed>");
|
||||
|
||||
let message = if let Some(s) = info.payload().downcast_ref::<&str>() {
|
||||
s.to_string()
|
||||
} else if let Some(s) = info.payload().downcast_ref::<String>() {
|
||||
s.clone()
|
||||
} else {
|
||||
"unknown panic".to_string()
|
||||
};
|
||||
|
||||
let location = info.location().map(|l| format!("{}:{}:{}", l.file(), l.line(), l.column()))
|
||||
.unwrap_or_else(|| "unknown location".to_string());
|
||||
|
||||
tracing::error!(
|
||||
thread = thread_name,
|
||||
location = %location,
|
||||
"PANIC: {}",
|
||||
message
|
||||
);
|
||||
|
||||
default_hook(info);
|
||||
}));
|
||||
}
|
||||
```
|
||||
|
||||
**Call site**: `musicfs-cli/src/main.rs`, at the very top of `main()`:
|
||||
```rust
|
||||
fn main() -> Result<()> {
|
||||
musicfs_core::install_panic_hook();
|
||||
let cli = Cli::parse();
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-core # Panic hook unit tests
|
||||
cargo test -p musicfs-test-utils # Resilience tests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Issue 3.7 + 2.7: systemd Service Fix + FUSE Cleanup
|
||||
|
||||
**Approach**: Fix the systemd service file and add stale mount detection on startup.
|
||||
|
||||
#### Step 1: No stubs needed (config change)
|
||||
|
||||
#### Step 2: RED tests
|
||||
|
||||
Already exists: `test_systemd_service_has_execstoppost` — currently fails because service file lacks `ExecStopPost`.
|
||||
|
||||
Add test for stale mount detection:
|
||||
```rust
|
||||
#[test]
|
||||
fn test_stale_mount_check_function_exists() {
|
||||
// Verify the function signature exists
|
||||
// (actual mount test needs privileged environment)
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `dist/musicfs.service`
|
||||
|
||||
```diff
|
||||
ExecStop=/usr/bin/musicfs shutdown
|
||||
+ExecStopPost=/usr/bin/fusermount -uz %h/music || true
|
||||
Restart=on-failure
|
||||
```
|
||||
|
||||
Note: `fusermount -uz` is "lazy unmount" — always succeeds even if mount is busy. The `|| true` prevents systemd from treating cleanup failure as a service failure.
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — add stale mount check before mounting:
|
||||
|
||||
```rust
|
||||
fn check_stale_mount(mountpoint: &Path) -> Result<()> {
|
||||
// Check /proc/mounts for existing mount at this path
|
||||
if let Ok(mounts) = std::fs::read_to_string("/proc/mounts") {
|
||||
for line in mounts.lines() {
|
||||
if line.contains(&mountpoint.to_string_lossy().as_ref()) && line.contains("fuse") {
|
||||
warn!("Stale FUSE mount detected at {:?}, attempting cleanup", mountpoint);
|
||||
let status = std::process::Command::new("fusermount")
|
||||
.args(["-uz", &mountpoint.to_string_lossy()])
|
||||
.status();
|
||||
match status {
|
||||
Ok(s) if s.success() => info!("Stale mount cleaned up"),
|
||||
Ok(s) => warn!("fusermount exited with: {}", s),
|
||||
Err(e) => warn!("Failed to run fusermount: {}", e),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
Also fix the `test_systemd_service_has_execstoppost` test path — currently points to wrong location.
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils -- test_systemd # Service file test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Issue 2.10: sd_notify Integration
|
||||
|
||||
**Approach**: Add `sd-notify` crate, call `READY=1` after mount, `STOPPING` on shutdown.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
Add dependency to `musicfs-cli/Cargo.toml`:
|
||||
```toml
|
||||
sd-notify = "0.4"
|
||||
```
|
||||
|
||||
#### Step 2: RED tests
|
||||
|
||||
Write test that mocks the notify socket:
|
||||
```rust
|
||||
#[test]
|
||||
fn test_sd_notify_ready_sent() {
|
||||
// Create Unix datagram socket at $NOTIFY_SOCKET
|
||||
// Call sd_notify::notify(READY=1)
|
||||
// Verify message received on socket
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs`
|
||||
|
||||
After `fs.mount()` succeeds (or more precisely, after `spawn_mount2` — see 4.6):
|
||||
```rust
|
||||
// Notify systemd we're ready
|
||||
if let Err(e) = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]) {
|
||||
debug!("sd_notify not available (not running under systemd): {}", e);
|
||||
}
|
||||
```
|
||||
|
||||
On shutdown path:
|
||||
```rust
|
||||
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]);
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils -- test_sd_notify
|
||||
cargo build -p musicfs-cli # Verify it compiles with new dep
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Issue 2.1: Signal Handling
|
||||
|
||||
**Approach**: Switch from `fuser::mount2` (blocking) to `fuser::spawn_mount2` (background), then listen for signals on the main thread.
|
||||
|
||||
This is the most complex change in Phase A. It restructures the daemon's main loop.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
Change `MusicFs::mount()` signature to return a session handle:
|
||||
|
||||
```rust
|
||||
// BEFORE
|
||||
pub fn mount(self, mountpoint: &Path) -> Result<()> {
|
||||
fuser::mount2(self, mountpoint, &options)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// AFTER (stub — returns BackgroundSession)
|
||||
pub fn spawn_mount(self, mountpoint: &Path) -> Result<fuser::BackgroundSession> {
|
||||
let session = fuser::spawn_mount2(self, mountpoint, &options)?;
|
||||
Ok(session)
|
||||
}
|
||||
```
|
||||
|
||||
Keep old `mount()` temporarily for compatibility.
|
||||
|
||||
#### Step 2: RED tests
|
||||
|
||||
Write in `tests/resilience.rs`:
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_sigterm_triggers_shutdown() {
|
||||
// Spawn daemon as child process
|
||||
// Wait for mount
|
||||
// Send SIGTERM
|
||||
// Verify clean exit within 10s
|
||||
// Verify mountpoint is unmounted
|
||||
}
|
||||
```
|
||||
|
||||
This test requires the signal handler to exist. It will be RED until implementation.
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — rewrite `run_mount()`:
|
||||
|
||||
```rust
|
||||
fn run_mount(mountpoint: PathBuf, origin_path: Option<PathBuf>, cache_dir: Option<PathBuf>) -> Result<()> {
|
||||
let origin_path = origin_path.context("--origin is required")?;
|
||||
let runtime = tokio::runtime::Runtime::new()?;
|
||||
let handle = runtime.handle().clone();
|
||||
|
||||
let (tree, reader) = runtime.block_on(async {
|
||||
// ... existing setup code (unchanged) ...
|
||||
Ok::<_, anyhow::Error>((tree, reader))
|
||||
})?;
|
||||
|
||||
// Check for stale mount before mounting
|
||||
check_stale_mount(&mountpoint)?;
|
||||
|
||||
let fs = MusicFs::with_reader(tree, reader, handle.clone());
|
||||
info!("Mounting filesystem at {:?}", mountpoint);
|
||||
|
||||
// spawn_mount2 returns immediately — FUSE runs in background
|
||||
let session = fs.spawn_mount(&mountpoint)
|
||||
.context("Failed to mount filesystem")?;
|
||||
|
||||
// Notify systemd
|
||||
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Ready]);
|
||||
info!("MusicFS ready, PID {}", std::process::id());
|
||||
|
||||
// Block on signal
|
||||
runtime.block_on(async {
|
||||
let mut sigterm = tokio::signal::unix::signal(
|
||||
tokio::signal::unix::SignalKind::terminate()
|
||||
)?;
|
||||
let mut sigint = tokio::signal::unix::signal(
|
||||
tokio::signal::unix::SignalKind::interrupt()
|
||||
)?;
|
||||
|
||||
tokio::select! {
|
||||
_ = sigterm.recv() => {
|
||||
info!("Received SIGTERM, shutting down");
|
||||
}
|
||||
_ = sigint.recv() => {
|
||||
info!("Received SIGINT, shutting down");
|
||||
}
|
||||
}
|
||||
|
||||
Ok::<_, anyhow::Error>(())
|
||||
})?;
|
||||
|
||||
// Shutdown sequence
|
||||
let _ = sd_notify::notify(false, &[sd_notify::NotifyState::Stopping]);
|
||||
info!("Unmounting filesystem");
|
||||
drop(session); // BackgroundSession::drop() calls unmount
|
||||
info!("Shutdown complete");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
**File**: `musicfs-fuse/src/filesystem.rs` — add `spawn_mount()`:
|
||||
|
||||
```rust
|
||||
pub fn spawn_mount(self, mountpoint: &Path) -> Result<fuser::BackgroundSession> {
|
||||
info!("Mounting MusicFS at {:?}", mountpoint);
|
||||
let options = vec![
|
||||
fuser::MountOption::RO,
|
||||
fuser::MountOption::FSName("musicfs".to_string()),
|
||||
fuser::MountOption::AutoUnmount,
|
||||
fuser::MountOption::AllowOther,
|
||||
];
|
||||
let session = fuser::spawn_mount2(self, mountpoint, &options)
|
||||
.map_err(musicfs_core::Error::Io)?;
|
||||
Ok(session)
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo build -p musicfs-cli
|
||||
cargo test -p musicfs-test-utils -- test_sigterm # Process-level test
|
||||
cargo test # No regressions
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1 Security & Privacy
|
||||
|
||||
- No new attack surface — changes are internal lifecycle management
|
||||
- Panic hook does NOT log sensitive data (only panic message, thread name, location)
|
||||
- `sd_notify` uses existing systemd socket — no new IPC
|
||||
|
||||
### 5.2 Observability
|
||||
|
||||
- Panic hook ensures all panics are captured in logs/journald
|
||||
- Signal handling logs which signal triggered shutdown
|
||||
- sd_notify gives systemd accurate service state
|
||||
- Stale mount detection logs cleanup attempts
|
||||
|
||||
### 5.3 Testing
|
||||
|
||||
All changes follow the TDD flow:
|
||||
1. Stubs compile
|
||||
2. RED tests document expected behavior
|
||||
3. Implementation turns tests GREEN
|
||||
4. Full suite passes (no regressions)
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1 Poison Recovery Instead of parking_lot
|
||||
|
||||
**Alternative**: Keep `std::sync::RwLock`, add `.unwrap_or_else(|p| p.into_inner())` to every lock call.
|
||||
|
||||
**Rejected**: 30+ call sites to change, easy to miss one, and the pattern is verbose. `parking_lot` is already a dependency and is strictly better for this use case (faster, no poison, correct API).
|
||||
|
||||
### 6.2 Keep mount2 (blocking) with Signal Thread
|
||||
|
||||
**Alternative**: Keep `fuser::mount2`, spawn a separate thread for signal handling, use a channel to communicate shutdown.
|
||||
|
||||
**Rejected**: `mount2` consumes `self` and blocks — there's no clean way to interrupt it from another thread. `spawn_mount2` is the canonical solution from the `fuser` crate.
|
||||
|
||||
### 6.3 Defer sd_notify Until Full Shutdown Orchestration
|
||||
|
||||
**Alternative**: Implement sd_notify only after CancellationToken + graceful shutdown are in place.
|
||||
|
||||
**Rejected**: sd_notify `READY=1` is critical now — without it, `Type=notify` in the service file means systemd will timeout and kill the daemon on every start. The shutdown `STOPPING` notification is a bonus but not required for Phase A.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### 7.1 Task Sequence
|
||||
|
||||
| Day | Task | Issue | Effort | Test Approach |
|
||||
|-----|------|-------|--------|---------------|
|
||||
| 1 (morning) | RwLock → parking_lot migration | 2.9 | 2h | Existing GREEN test validates; verify no `.unwrap()` on locks |
|
||||
| 1 (afternoon) | Panic hook | 2.2 | 2h | New test: panic → verify tracing output |
|
||||
| 2 (morning) | systemd ExecStopPost + stale mount check | 3.7 + 2.7 | 2h | Existing RED test → GREEN; new stale mount test |
|
||||
| 2 (afternoon) | sd_notify integration | 2.10 | 2h | New test: mock socket → verify READY=1 |
|
||||
| 3 | Signal handling (spawn_mount2 + signal loop) | 2.1 | 4h | Fork daemon → send SIGTERM → verify exit |
|
||||
| 4 | Integration + regression testing | — | 4h | Full `cargo test`, manual FUSE mount test |
|
||||
| 5 | Buffer for issues found during integration | — | 4h | — |
|
||||
|
||||
### 7.2 Verification Checklist
|
||||
|
||||
After all tasks complete:
|
||||
|
||||
- [ ] `cargo check` — zero errors, zero warnings
|
||||
- [ ] `cargo test` — all 162+ existing tests pass
|
||||
- [ ] `cargo test -p musicfs-test-utils` — all resilience tests pass
|
||||
- [ ] `cargo clippy` — no new warnings
|
||||
- [ ] `grep -r '\.read()\.unwrap()\|\.write()\.unwrap()' crates/` — zero hits in production code (test code is OK)
|
||||
- [ ] `dist/musicfs.service` contains `ExecStopPost`
|
||||
- [ ] Manual test: `musicfs mount`, then `kill -TERM <pid>`, verify clean exit + mount gone
|
||||
- [ ] Manual test: `kill -9 <pid>`, then `musicfs mount` again — no "already mounted" error
|
||||
|
||||
---
|
||||
|
||||
## 8. Files Changed
|
||||
|
||||
| File | Change | Issue |
|
||||
|------|--------|-------|
|
||||
| `musicfs-fuse/src/filesystem.rs` | `std::sync::RwLock` → `parking_lot::RwLock`; add `spawn_mount()` | 2.9, 2.1 |
|
||||
| `musicfs-cas/src/reader.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-cas/src/fetcher.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-origins/src/registry.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-cache/src/eviction.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-cache/src/tree.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-core/src/metrics.rs` | `std::sync::RwLock` → `parking_lot::RwLock` | 2.9 |
|
||||
| `musicfs-core/src/lib.rs` | Add `install_panic_hook()` | 2.2 |
|
||||
| `musicfs-cli/src/main.rs` | Panic hook, signal handler, spawn_mount2, sd_notify, stale mount check | 2.1, 2.2, 2.7, 2.10 |
|
||||
| `musicfs-cli/Cargo.toml` | Add `sd-notify`, `tokio-util` deps | 2.10, 2.1 |
|
||||
| `dist/musicfs.service` | Add `ExecStopPost`, fix `ExecStop` | 3.7 |
|
||||
| `tests/resilience.rs` | Update/add tests for signal, panic hook, sd_notify | all |
|
||||
|
||||
---
|
||||
|
||||
## 9. Glossary / References
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **parking_lot** | Fast, poison-free lock implementation. Already a project dependency. |
|
||||
| **spawn_mount2** | `fuser` API that mounts FUSE in a background thread, returning a `BackgroundSession` handle |
|
||||
| **sd_notify** | systemd notification protocol. `READY=1` signals service started, `STOPPING` signals shutdown. |
|
||||
| **BackgroundSession** | Handle returned by `spawn_mount2`. Dropping it unmounts the filesystem. |
|
||||
|
||||
| Document | Path |
|
||||
|----------|------|
|
||||
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
|
||||
| Resilience testing | [resilience-testing.md](resilience-testing.md) |
|
||||
| Architecture | [architecture.md](../architecture.md) |
|
||||
@@ -1,830 +0,0 @@
|
||||
# Phase B: Crash Recovery — Implementation Plan
|
||||
|
||||
**Authors:** AI-assisted
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-13
|
||||
**Reviewers:** TBD
|
||||
**Approvers:** TBD
|
||||
**Prerequisites:** [phase-a-stop-dying.md](phase-a-stop-dying.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
|
||||
**Estimated Effort:** ~5 days
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
Phase A made the daemon survive signals and panics. Phase B makes it **recover from crashes** — startup integrity checks for all storage layers (SQLite, tantivy, sled), graceful shutdown with ordered teardown of background tasks, disk space pre-checks, and a task supervisor that restarts dead background tasks.
|
||||
|
||||
This covers issues 2.3, 2.4, 2.6, and 2.8 from the [resilience audit](resilience-fault-tolerance.md), deferred from Phase A.
|
||||
|
||||
Issue 2.5 (interrupted sync recovery) is deferred to after [persistent state](persistent-state.md) is wired up — checkpoint/resume requires the DB to be in the mount path.
|
||||
|
||||
**RED tests to turn GREEN** (from current `resilience.rs`):
|
||||
- `test_sqlite_integrity_check_detects_corruption` — currently `todo!()`
|
||||
- `test_tantivy_corruption_triggers_rebuild` — currently `todo!()`
|
||||
- `test_sled_corruption_triggers_repair` — currently `todo!()`
|
||||
- `test_cas_put_handles_enospc` — currently fails (no size pre-check)
|
||||
- `test_tantivy_survives_uncommitted_crash` — currently `todo!()`
|
||||
|
||||
**New tests to write:**
|
||||
- Shutdown orchestration: CancellationToken propagation, ordered teardown, tantivy flush
|
||||
- Task supervisor: panic detection, restart with backoff, status reporting
|
||||
|
||||
---
|
||||
|
||||
## 2. Background
|
||||
|
||||
### 2.1 What Phase A Delivered
|
||||
|
||||
- Signal handling via `spawn_mount2` + tokio signal loop ✅
|
||||
- Panic hook logging via `tracing::error!` ✅
|
||||
- RwLock → `parking_lot` (no more poison cascade) ✅
|
||||
- sd_notify READY/STOPPING ✅
|
||||
- ExecStopPost + stale mount detection ✅
|
||||
|
||||
### 2.2 What's Still Broken After Phase A
|
||||
|
||||
The daemon now **stops cleanly** on signals but:
|
||||
|
||||
1. **Shutdown is unordered** — `drop(session)` unmounts FUSE, but background tasks (health monitor, indexer, watcher, prefetcher) are killed mid-operation by runtime drop. No tantivy flush, no SQLite checkpoint.
|
||||
|
||||
2. **No startup integrity checks** — if the daemon was `kill -9`'d (or OOM-killed, power loss), SQLite/tantivy/sled may have partial writes. Currently these propagate as runtime errors or silent corruption.
|
||||
|
||||
3. **Background tasks are fire-and-forget** — health monitor, watcher, indexer, prefetcher use `tokio::spawn` with no `JoinHandle` stored. If a task panics, it's silently dead.
|
||||
|
||||
4. **CAS accepts oversized writes** — `put()` doesn't check `max_size` before writing. Cache grows unbounded.
|
||||
|
||||
---
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1 Goals
|
||||
|
||||
- Graceful shutdown flushes tantivy, checkpoints SQLite WAL, stops background tasks in order
|
||||
- Corrupted SQLite detected on open via `PRAGMA integrity_check`
|
||||
- Corrupted tantivy index detected and rebuilt from scratch
|
||||
- Corrupted sled index detected and repaired
|
||||
- CAS rejects writes that would exceed `max_size`
|
||||
- Background tasks are supervised — panics detected, critical tasks restarted
|
||||
- All 5 RED tests turn GREEN
|
||||
- All new tests for shutdown + supervisor are GREEN
|
||||
|
||||
### 3.2 Non-Goals
|
||||
|
||||
- Interrupted sync recovery (2.5) — depends on persistent state work
|
||||
- Disk space monitoring daemon (periodic `statvfs`) — Phase C
|
||||
- Connection pooling, config reload, watchdog — Phase C/D
|
||||
- Passthrough mode when cache dies — Phase F
|
||||
|
||||
---
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1 Implementation Order
|
||||
|
||||
```
|
||||
4.2 CAS size pre-check (no deps, simplest fix)
|
||||
↓
|
||||
4.3 SQLite integrity check (no deps)
|
||||
↓
|
||||
4.4 tantivy corruption recovery (no deps)
|
||||
↓
|
||||
4.5 sled corruption recovery (no deps)
|
||||
↓
|
||||
4.6 Graceful shutdown orchestration (depends on: Phase A signal handler)
|
||||
↓
|
||||
4.7 Task supervisor (depends on: 4.6 CancellationToken)
|
||||
```
|
||||
|
||||
### 4.2 Issue 2.8: CAS Size Pre-Check
|
||||
|
||||
**Problem**: `CasStore::put()` writes data without checking if it would exceed `max_size`. The existing test `test_cas_put_handles_enospc` creates a store with `max_size: 100` and writes 1000 bytes — currently succeeds when it should fail.
|
||||
|
||||
#### Step 1: Stubs — none needed
|
||||
|
||||
#### Step 2: RED test — already exists
|
||||
|
||||
```rust
|
||||
// Currently FAILS — this is what we need to fix
|
||||
#[tokio::test]
|
||||
async fn test_cas_put_handles_enospc() {
|
||||
let store = CasStore::open(CasConfig { max_size: 100, ... }).await.unwrap();
|
||||
let large_data = vec![0u8; 1000];
|
||||
let result = store.put(&large_data).await;
|
||||
assert!(result.is_err());
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cas/src/store.rs` — add size check at top of `put()`:
|
||||
|
||||
```rust
|
||||
pub async fn put(&self, data: &[u8]) -> Result<ChunkHash, CasError> {
|
||||
let hash = ChunkHash::from_bytes(data);
|
||||
let path = self.chunk_path(&hash);
|
||||
|
||||
if path.exists() {
|
||||
trace!(hash = %hash, size_bytes = data.len(), "dedup hit");
|
||||
return Ok(hash);
|
||||
}
|
||||
|
||||
// NEW: Pre-check size limit
|
||||
if self.config.max_size > 0 {
|
||||
let new_size = self.current_size.load(Ordering::SeqCst) + data.len() as u64;
|
||||
if new_size > self.config.max_size {
|
||||
warn!(
|
||||
current_size = self.current_size.load(Ordering::SeqCst),
|
||||
chunk_size = data.len(),
|
||||
max_size = self.config.max_size,
|
||||
"CAS store full, rejecting write"
|
||||
);
|
||||
return Err(CasError::StoreFull {
|
||||
current: self.current_size.load(Ordering::SeqCst),
|
||||
max: self.config.max_size,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ... rest of put() unchanged
|
||||
}
|
||||
```
|
||||
|
||||
Also add new error variant:
|
||||
|
||||
```rust
|
||||
pub enum CasError {
|
||||
// ... existing variants
|
||||
#[error("Store full: {current} / {max} bytes")]
|
||||
StoreFull { current: u64, max: u64 },
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_cas_put_handles_enospc
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Issue 2.4 (part 1): SQLite Integrity Check
|
||||
|
||||
**Problem**: `Database::open()` runs schema but no integrity check. After crash, corrupt pages serve bad data silently.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
Add to `musicfs-cache/src/db.rs`:
|
||||
|
||||
```rust
|
||||
pub fn open_with_integrity_check(path: &Path) -> Result<Self> {
|
||||
todo!()
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2: RED test — already exists as `todo!()`
|
||||
|
||||
Replace the `todo!()` with a real test:
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_sqlite_integrity_check_detects_corruption() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let db_path = dir.path().join("test.db");
|
||||
|
||||
// Create valid DB with data
|
||||
{
|
||||
let db = Database::open(&db_path).unwrap();
|
||||
db.upsert_file(
|
||||
&OriginId::from("test"),
|
||||
Path::new("/test.flac"),
|
||||
&VirtualPath::new("/Test.flac"),
|
||||
&AudioMeta::default(),
|
||||
UNIX_EPOCH,
|
||||
1000,
|
||||
).unwrap();
|
||||
}
|
||||
|
||||
// Corrupt the file
|
||||
let mut data = std::fs::read(&db_path).unwrap();
|
||||
let mid = data.len() / 2;
|
||||
data[mid..mid+100].fill(0xFF);
|
||||
std::fs::write(&db_path, &data).unwrap();
|
||||
|
||||
// open_with_integrity_check should detect corruption
|
||||
let result = Database::open_with_integrity_check(&db_path);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cache/src/db.rs`
|
||||
|
||||
```rust
|
||||
pub fn open_with_integrity_check(path: &Path) -> Result<Self> {
|
||||
debug!(?path, "Opening database with integrity check");
|
||||
|
||||
let conn = Connection::open(path)
|
||||
.map_err(|e| Error::Database(format!("open failed: {}", e)))?;
|
||||
|
||||
// Quick integrity check — verifies page-level consistency
|
||||
let integrity: String = conn
|
||||
.query_row("PRAGMA integrity_check(1)", [], |row| row.get(0))
|
||||
.map_err(|e| Error::Database(format!("integrity check failed: {}", e)))?;
|
||||
|
||||
if integrity != "ok" {
|
||||
warn!(path = ?path, result = %integrity, "Database integrity check failed");
|
||||
return Err(Error::DatabaseCorrupted(format!(
|
||||
"integrity check failed: {}", integrity
|
||||
)));
|
||||
}
|
||||
|
||||
conn.execute_batch(SCHEMA)
|
||||
.map_err(|e| Error::Database(format!("schema init failed: {}", e)))?;
|
||||
|
||||
let db = Self { conn: Arc::new(Mutex::new(conn)) };
|
||||
let count = db.file_count().unwrap_or(0);
|
||||
info!(path = ?path, file_count = count, "Database opened (integrity verified)");
|
||||
Ok(db)
|
||||
}
|
||||
```
|
||||
|
||||
Also add the error variant to `musicfs-core/src/error.rs`:
|
||||
|
||||
```rust
|
||||
pub enum Error {
|
||||
// ... existing
|
||||
#[error("Database corrupted: {0}")]
|
||||
DatabaseCorrupted(String),
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_sqlite_integrity
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Issue 2.4 (part 2): tantivy Corruption Recovery
|
||||
|
||||
**Problem**: If tantivy `meta.json` or segment files are corrupted, `Index::open_in_dir()` panics or returns an error. No recovery path — daemon crashes.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
Add to `musicfs-search/src/index.rs`:
|
||||
|
||||
```rust
|
||||
pub fn open_with_recovery(index_path: &Path) -> Result<Self, SearchError> {
|
||||
todo!()
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2: RED test — replace `todo!()` with real test
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_tantivy_corruption_triggers_rebuild() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let index_path = dir.path().join("search_idx");
|
||||
|
||||
// Create valid index with data
|
||||
{
|
||||
let index = SearchIndex::open(&index_path).unwrap();
|
||||
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
|
||||
index.commit().unwrap();
|
||||
}
|
||||
|
||||
// Corrupt meta.json
|
||||
std::fs::write(index_path.join("meta.json"), b"corrupted").unwrap();
|
||||
|
||||
// open_with_recovery should detect corruption and rebuild empty
|
||||
let index = SearchIndex::open_with_recovery(&index_path).unwrap();
|
||||
let results = index.search("a", 10).unwrap();
|
||||
assert_eq!(results.len(), 0); // Rebuilt empty but functional
|
||||
}
|
||||
```
|
||||
|
||||
Also replace the tantivy crash test `todo!()`:
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_tantivy_survives_uncommitted_crash() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let index_path = dir.path().join("search_idx");
|
||||
|
||||
{
|
||||
let index = SearchIndex::open(&index_path).unwrap();
|
||||
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
|
||||
index.commit().unwrap();
|
||||
// Write without commit, then "crash" (drop without commit)
|
||||
index.index_file(&make_file_meta(2, "/b.flac", 1000)).unwrap();
|
||||
// mem::forget would leak, just drop naturally
|
||||
}
|
||||
|
||||
let index = SearchIndex::open(&index_path).unwrap();
|
||||
let results = index.search("a", 10).unwrap();
|
||||
assert_eq!(results.len(), 1); // Committed doc survives
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-search/src/index.rs`
|
||||
|
||||
```rust
|
||||
pub fn open_with_recovery(index_path: &Path) -> Result<Self, SearchError> {
|
||||
match Self::open(index_path) {
|
||||
Ok(index) => {
|
||||
// Verify index is functional with a simple search
|
||||
match index.reader.searcher().num_docs() {
|
||||
docs => {
|
||||
info!(docs, "Search index opened successfully");
|
||||
Ok(index)
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(
|
||||
error = %e,
|
||||
path = ?index_path,
|
||||
"Search index corrupted, rebuilding from scratch"
|
||||
);
|
||||
// Delete corrupted index
|
||||
if index_path.exists() {
|
||||
std::fs::remove_dir_all(index_path)
|
||||
.map_err(|e| SearchError::Io(e))?;
|
||||
}
|
||||
// Create fresh index
|
||||
Self::open(index_path)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_tantivy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Issue 3.5: sled Corruption Recovery
|
||||
|
||||
**Problem**: `sled::open()` on a corrupted DB returns `sled::Error::Corruption` which propagates as `CasError::Sled` and crashes the daemon on startup.
|
||||
|
||||
#### Step 1: Stubs — none needed, modify existing `open()`
|
||||
|
||||
#### Step 2: RED test — replace `todo!()`
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_sled_corruption_triggers_repair() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let chunks_dir = dir.path().join("chunks");
|
||||
let config = CasConfig { chunks_dir: chunks_dir.clone(), max_size: 10_000_000, shard_levels: 2 };
|
||||
|
||||
// Create valid store with data
|
||||
{
|
||||
let store = CasStore::open(config.clone()).await.unwrap();
|
||||
store.put(b"test data").await.unwrap();
|
||||
}
|
||||
|
||||
// Corrupt sled index files
|
||||
let sled_dir = chunks_dir.join("index.sled");
|
||||
if sled_dir.exists() {
|
||||
for entry in std::fs::read_dir(&sled_dir).unwrap() {
|
||||
let entry = entry.unwrap();
|
||||
if entry.metadata().unwrap().is_file() {
|
||||
std::fs::write(entry.path(), b"corrupted").unwrap();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Re-open should recover (repair or recreate)
|
||||
let result = CasStore::open(config).await;
|
||||
assert!(result.is_ok(), "sled should recover from corruption");
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cas/src/store.rs` — modify `open()`:
|
||||
|
||||
```rust
|
||||
pub async fn open(config: CasConfig) -> Result<Self, CasError> {
|
||||
fs::create_dir_all(&config.chunks_dir).await?;
|
||||
|
||||
let index_path = config.chunks_dir.join("index.sled");
|
||||
let index = match sled::open(&index_path) {
|
||||
Ok(db) => db,
|
||||
Err(e) => {
|
||||
warn!(error = %e, path = ?index_path, "sled index corrupted, attempting recovery");
|
||||
|
||||
// Try repair
|
||||
match sled::Config::new().path(&index_path).repair(true).open() {
|
||||
Ok(db) => {
|
||||
info!("sled index repaired successfully");
|
||||
db
|
||||
}
|
||||
Err(repair_err) => {
|
||||
warn!(error = %repair_err, "sled repair failed, recreating index");
|
||||
// Delete and recreate
|
||||
if index_path.exists() {
|
||||
std::fs::remove_dir_all(&index_path)
|
||||
.map_err(|e| CasError::Io(e))?;
|
||||
}
|
||||
sled::open(&index_path)?
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
let current_size = Self::calculate_size(&config.chunks_dir).await;
|
||||
|
||||
Ok(Self {
|
||||
config,
|
||||
index,
|
||||
current_size: AtomicU64::new(current_size),
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_sled_corruption
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Issue 2.3: Graceful Shutdown Orchestration
|
||||
|
||||
**Problem**: On signal, `drop(session)` unmounts FUSE, then `drop(runtime)` kills all tokio tasks abruptly. No tantivy flush, no SQLite WAL checkpoint, no ordered task shutdown.
|
||||
|
||||
**Approach**: `CancellationToken` from `tokio_util` propagated to all background tasks. Signal triggers token cancellation, then ordered shutdown.
|
||||
|
||||
#### Step 1: Add dependency
|
||||
|
||||
```toml
|
||||
# musicfs-cli/Cargo.toml
|
||||
tokio-util = { version = "0.7", features = ["rt"] }
|
||||
```
|
||||
|
||||
#### Step 2: Tests
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_shutdown_cancels_background_tasks() {
|
||||
let token = CancellationToken::new();
|
||||
let stopped = Arc::new(AtomicBool::new(false));
|
||||
let stopped_clone = stopped.clone();
|
||||
let token_clone = token.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
token_clone.cancelled().await;
|
||||
stopped_clone.store(true, Ordering::SeqCst);
|
||||
});
|
||||
|
||||
assert!(!stopped.load(Ordering::SeqCst));
|
||||
token.cancel();
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
assert!(stopped.load(Ordering::SeqCst));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_shutdown_flushes_tantivy() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let index = SearchIndex::open(dir.path().join("idx")).unwrap();
|
||||
|
||||
index.index_file(&make_file_meta(1, "/a.flac", 1000)).unwrap();
|
||||
// Graceful shutdown should commit
|
||||
index.commit().unwrap();
|
||||
|
||||
let index2 = SearchIndex::open(dir.path().join("idx")).unwrap();
|
||||
assert_eq!(index2.search("a", 10).unwrap().len(), 1);
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs` — restructure the signal loop:
|
||||
|
||||
The current code:
|
||||
```rust
|
||||
// Wait for signal
|
||||
runtime.block_on(async { ... signal select ... })?;
|
||||
// Drop session, exit
|
||||
```
|
||||
|
||||
Change to:
|
||||
```rust
|
||||
let shutdown_token = CancellationToken::new();
|
||||
|
||||
// TODO: Pass token to health monitor, watcher, indexer, prefetcher
|
||||
// (requires their start() methods to accept CancellationToken)
|
||||
// For now, we just use it for the shutdown sequence
|
||||
|
||||
runtime.block_on(async {
|
||||
// ... signal select ...
|
||||
|
||||
// Ordered shutdown
|
||||
info!("Beginning ordered shutdown");
|
||||
shutdown_token.cancel();
|
||||
|
||||
// Wait briefly for tasks to notice cancellation
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
|
||||
// Flush search index if available
|
||||
// (requires SearchIndex to be accessible — currently not wired in main.rs)
|
||||
|
||||
info!("Background tasks stopped");
|
||||
})?;
|
||||
```
|
||||
|
||||
**Note**: Full CancellationToken propagation through health monitor, watcher, indexer, and prefetcher `start()` methods requires changing their signatures. The current `mpsc::channel<()>` stop mechanism in each task should be replaced with or supplemented by the token. This can be done incrementally — start by adding the token to `run_mount()`, then wire it into each task as they're touched.
|
||||
|
||||
For this phase, the minimum viable change is:
|
||||
1. Create the token in `run_mount()`
|
||||
2. Cancel it on signal
|
||||
3. Add a brief sleep for tasks to notice
|
||||
4. The existing `drop(session)` and runtime drop handle cleanup
|
||||
|
||||
Full per-task CancellationToken wiring is tracked as follow-up work.
|
||||
|
||||
---
|
||||
|
||||
### 4.7 Issue 2.6: Task Supervisor
|
||||
|
||||
**Problem**: 13 `tokio::spawn()` calls with no `JoinHandle` stored. Dead tasks go unnoticed.
|
||||
|
||||
**Approach**: New `TaskSupervisor` struct in `musicfs-core` that stores handles, checks liveness, and restarts critical tasks.
|
||||
|
||||
#### Step 1: Stubs
|
||||
|
||||
**File**: `musicfs-core/src/supervisor.rs` (new file)
|
||||
|
||||
```rust
|
||||
pub struct TaskSupervisor { ... }
|
||||
|
||||
pub enum TaskStatus {
|
||||
Running,
|
||||
Failed { error: String, at: Instant },
|
||||
Restarting { attempt: u32 },
|
||||
Stopped,
|
||||
}
|
||||
|
||||
impl TaskSupervisor {
|
||||
pub fn new() -> Self;
|
||||
pub fn spawn_supervised(&self, name: &str, future: impl Future) -> ();
|
||||
pub fn spawn_critical(&self, name: &str, factory: impl Fn() -> impl Future) -> ();
|
||||
pub fn task_status(&self, name: &str) -> TaskStatus;
|
||||
pub fn check_all(&self) -> Vec<(String, TaskStatus)>;
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2: Tests
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_supervisor_detects_task_completion() {
|
||||
let supervisor = TaskSupervisor::new();
|
||||
supervisor.spawn_supervised("fast", async { /* returns immediately */ });
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
// Task completed normally — should be Stopped, not Failed
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_supervisor_detects_panic() {
|
||||
let supervisor = TaskSupervisor::new();
|
||||
supervisor.spawn_supervised("panicker", async {
|
||||
panic!("boom");
|
||||
});
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
assert!(matches!(supervisor.task_status("panicker"), TaskStatus::Failed { .. }));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_supervisor_restarts_critical_task() {
|
||||
let count = Arc::new(AtomicU32::new(0));
|
||||
let c = count.clone();
|
||||
|
||||
let supervisor = TaskSupervisor::new();
|
||||
supervisor.spawn_critical("restartable", move || {
|
||||
let c = c.clone();
|
||||
async move {
|
||||
let n = c.fetch_add(1, Ordering::SeqCst);
|
||||
if n == 0 { panic!("first run fails"); }
|
||||
// Second run: stay alive
|
||||
loop { tokio::time::sleep(Duration::from_secs(60)).await; }
|
||||
}
|
||||
});
|
||||
|
||||
tokio::time::sleep(Duration::from_secs(2)).await;
|
||||
assert_eq!(count.load(Ordering::SeqCst), 2);
|
||||
assert!(matches!(supervisor.task_status("restartable"), TaskStatus::Running));
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-core/src/supervisor.rs`
|
||||
|
||||
```rust
|
||||
use parking_lot::RwLock;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
use tokio::task::JoinHandle;
|
||||
use tracing::{error, info, warn};
|
||||
|
||||
pub struct TaskSupervisor {
|
||||
tasks: Arc<RwLock<HashMap<String, TaskEntry>>>,
|
||||
}
|
||||
|
||||
struct TaskEntry {
|
||||
handle: JoinHandle<()>,
|
||||
status: TaskStatus,
|
||||
restart_count: u32,
|
||||
last_restart: Option<Instant>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum TaskStatus {
|
||||
Running,
|
||||
Failed { error: String, at: Instant },
|
||||
Restarting { attempt: u32 },
|
||||
Stopped,
|
||||
}
|
||||
|
||||
impl TaskSupervisor {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
tasks: Arc::new(RwLock::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn spawn_supervised<F>(&self, name: &str, future: F)
|
||||
where
|
||||
F: std::future::Future<Output = ()> + Send + 'static,
|
||||
{
|
||||
let tasks = self.tasks.clone();
|
||||
let name_owned = name.to_string();
|
||||
|
||||
let handle = tokio::spawn(async move {
|
||||
future.await;
|
||||
});
|
||||
|
||||
// Monitor the handle
|
||||
let tasks_monitor = self.tasks.clone();
|
||||
let name_monitor = name.to_string();
|
||||
let monitor_handle = handle;
|
||||
|
||||
self.tasks.write().insert(
|
||||
name_owned,
|
||||
TaskEntry {
|
||||
handle: monitor_handle,
|
||||
status: TaskStatus::Running,
|
||||
restart_count: 0,
|
||||
last_restart: None,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
pub fn task_status(&self, name: &str) -> TaskStatus {
|
||||
let mut tasks = self.tasks.write();
|
||||
if let Some(entry) = tasks.get_mut(name) {
|
||||
if entry.handle.is_finished() {
|
||||
entry.status = TaskStatus::Failed {
|
||||
error: "Task exited".into(),
|
||||
at: Instant::now(),
|
||||
};
|
||||
}
|
||||
entry.status.clone()
|
||||
} else {
|
||||
TaskStatus::Stopped
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: The full `spawn_critical` with automatic restart requires a task factory (`Fn() -> Future`) pattern. The supervisor spawns a monitor task that awaits the `JoinHandle`, and on failure, calls the factory again with exponential backoff (1s→5s→30s, max 5 restarts). This is the most complex piece — the detailed implementation is in the test code above.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1 Security & Privacy
|
||||
|
||||
- `PRAGMA integrity_check` is read-only — no risk to data
|
||||
- sled repair may lose recently-written entries — acceptable for a cache
|
||||
- tantivy rebuild deletes index entirely — no sensitive data exposure (metadata only)
|
||||
|
||||
### 5.2 Observability
|
||||
|
||||
- SQLite integrity check result logged at INFO (ok) or WARN (failed)
|
||||
- sled repair attempts logged at WARN
|
||||
- tantivy rebuild logged at WARN with file count before/after
|
||||
- CAS `StoreFull` error logged at WARN with current/max sizes
|
||||
- Task supervisor logs all state transitions (started, failed, restarting, stopped)
|
||||
|
||||
### 5.3 Testing
|
||||
|
||||
| Test | Status Before | Status After | Issue |
|
||||
|------|---------------|--------------|-------|
|
||||
| `test_cas_put_handles_enospc` | ❌ FAILED | ✅ GREEN | 2.8 |
|
||||
| `test_sqlite_integrity_check_detects_corruption` | ❌ todo!() | ✅ GREEN | 2.4 |
|
||||
| `test_tantivy_corruption_triggers_rebuild` | ❌ todo!() | ✅ GREEN | 2.4 |
|
||||
| `test_tantivy_survives_uncommitted_crash` | ❌ todo!() | ✅ GREEN | 5.2 |
|
||||
| `test_sled_corruption_triggers_repair` | ❌ todo!() | ✅ GREEN | 3.5 |
|
||||
| `test_shutdown_cancels_background_tasks` | NEW | ✅ GREEN | 2.3 |
|
||||
| `test_shutdown_flushes_tantivy` | NEW | ✅ GREEN | 2.3 |
|
||||
| `test_supervisor_detects_panic` | NEW | ✅ GREEN | 2.6 |
|
||||
| `test_supervisor_restarts_critical_task` | NEW | ✅ GREEN | 2.6 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1 Full `PRAGMA integrity_check` vs Quick Check
|
||||
|
||||
`PRAGMA integrity_check` scans every page — slow for large DBs (seconds for 1M rows). `PRAGMA integrity_check(1)` stops after the first error — fast enough for startup. We use the quick variant.
|
||||
|
||||
### 6.2 tantivy Repair vs Rebuild
|
||||
|
||||
tantivy has no built-in repair. If `meta.json` is corrupt or segments are missing, the only option is delete + recreate. This is acceptable because the search index can be rebuilt from SQLite metadata (once persistent state is wired up). For now, rebuild produces an empty index.
|
||||
|
||||
### 6.3 sled Repair vs Recreate
|
||||
|
||||
sled has `Config::repair(true)` which attempts to recover. If repair fails, we delete and recreate. After recreation, the index is empty but chunk files still exist on disk — a future reconciliation pass can rebuild the index from chunk files (Phase F).
|
||||
|
||||
### 6.4 Custom Supervisor vs `tokio-graceful` Crate
|
||||
|
||||
`tokio-graceful` provides shutdown coordination but not task restart. Our needs are specific (restart with backoff, status reporting, critical vs non-critical distinction). A custom `TaskSupervisor` is simpler and avoids a dependency for ~100 lines of code.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### 7.1 Task Sequence
|
||||
|
||||
| Day | Task | Issue | Effort | Test |
|
||||
|-----|------|-------|--------|------|
|
||||
| 1 (morning) | CAS size pre-check + `StoreFull` error variant | 2.8 | 1h | `test_cas_put_handles_enospc` → GREEN |
|
||||
| 1 (afternoon) | SQLite `open_with_integrity_check` + `DatabaseCorrupted` error | 2.4 | 2h | `test_sqlite_integrity_check` → GREEN |
|
||||
| 2 (morning) | tantivy `open_with_recovery` (detect + delete + recreate) | 2.4 | 2h | `test_tantivy_corruption` + `test_tantivy_survives_uncommitted_crash` → GREEN |
|
||||
| 2 (afternoon) | sled recovery in `CasStore::open` (repair + fallback recreate) | 3.5 | 2h | `test_sled_corruption` → GREEN |
|
||||
| 3 | Graceful shutdown with CancellationToken | 2.3 | 4h | `test_shutdown_cancels_background_tasks`, `test_shutdown_flushes_tantivy` → GREEN |
|
||||
| 4 | Task supervisor implementation | 2.6 | 4h | `test_supervisor_detects_panic`, `test_supervisor_restarts` → GREEN |
|
||||
| 5 | Integration + regression testing | — | 4h | Full `cargo test`, verify no regressions |
|
||||
|
||||
### 7.2 Verification Checklist
|
||||
|
||||
After all tasks:
|
||||
|
||||
- [ ] `cargo check` — zero errors, zero warnings
|
||||
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all tests pass (exclude pre-existing grpc issue)
|
||||
- [ ] `cargo test -p musicfs-test-utils --test resilience` — 5 previously-RED tests now GREEN
|
||||
- [ ] `cargo clippy` — no new warnings
|
||||
- [ ] Remaining RED tests are only for Phases C-F (health timeout, parallel checks, fd exhaustion, chunk auto-repair, passthrough mode)
|
||||
|
||||
---
|
||||
|
||||
## 8. Files Changed
|
||||
|
||||
| File | Change | Issue |
|
||||
|------|--------|-------|
|
||||
| `musicfs-cas/src/store.rs` | Size pre-check in `put()`, `StoreFull` error, sled recovery in `open()` | 2.8, 3.5 |
|
||||
| `musicfs-cache/src/db.rs` | `open_with_integrity_check()` with `PRAGMA integrity_check(1)` | 2.4 |
|
||||
| `musicfs-core/src/error.rs` | Add `DatabaseCorrupted(String)` variant | 2.4 |
|
||||
| `musicfs-search/src/index.rs` | `open_with_recovery()` — detect, delete, recreate | 2.4 |
|
||||
| `musicfs-core/src/supervisor.rs` | NEW — `TaskSupervisor`, `TaskStatus`, spawn/monitor/restart | 2.6 |
|
||||
| `musicfs-core/src/lib.rs` | Re-export supervisor module | 2.6 |
|
||||
| `musicfs-cli/src/main.rs` | CancellationToken creation, ordered shutdown sequence | 2.3 |
|
||||
| `musicfs-cli/Cargo.toml` | Add `tokio-util` dependency | 2.3 |
|
||||
| `musicfs-test-utils/tests/resilience.rs` | Replace `todo!()` stubs with real tests, add supervisor tests | all |
|
||||
|
||||
---
|
||||
|
||||
## 9. Glossary / References
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **CancellationToken** | `tokio_util::sync::CancellationToken` — cooperative cancellation signal for async tasks |
|
||||
| **PRAGMA integrity_check** | SQLite command that verifies page-level data consistency |
|
||||
| **sled repair** | sled's built-in recovery mode that attempts to reconstruct a corrupted database |
|
||||
| **TaskSupervisor** | New struct that monitors `JoinHandle`s and restarts failed tasks with backoff |
|
||||
| **StoreFull** | New `CasError` variant returned when a write would exceed `max_size` |
|
||||
|
||||
| Document | Path |
|
||||
|----------|------|
|
||||
| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
|
||||
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
|
||||
| Resilience testing | [resilience-testing.md](resilience-testing.md) |
|
||||
| Persistent state | [persistent-state.md](persistent-state.md) |
|
||||
@@ -1,598 +0,0 @@
|
||||
# Phase C: Production Hardening — Implementation Plan
|
||||
|
||||
**Authors:** AI-assisted
|
||||
**Status:** Draft
|
||||
**Last Updated:** 2026-05-13
|
||||
**Reviewers:** TBD
|
||||
**Approvers:** TBD
|
||||
**Prerequisites:** [phase-b-crash-recovery.md](phase-b-crash-recovery.md) (completed), [resilience-fault-tolerance.md](resilience-fault-tolerance.md)
|
||||
**Estimated Effort:** ~4 days
|
||||
|
||||
---
|
||||
|
||||
[TOC]
|
||||
|
||||
---
|
||||
|
||||
## 1. Abstract
|
||||
|
||||
Phase C merges the practical items from Phases C and D of the resilience audit into a single implementation pass. It fixes the remaining 6 RED tests and addresses production-critical issues: health check hangs that block all origin monitoring, unbounded FUSE reads that can freeze the filesystem, broken CAS size accounting that disables eviction, and concurrent mount protection.
|
||||
|
||||
**Deferred items** (depend on unimplemented features or low urgency): interrupted sync recovery (needs persistent state), SIGHUP config reload, connection pooling (S3/SFTP are stubs), event bus backpressure, FUSE session recovery, offline mode state machine, DNS failure handling, stale-data awareness.
|
||||
|
||||
**RED tests to turn GREEN:**
|
||||
- `test_local_origin_health_check_has_timeout` (D1)
|
||||
- `test_health_checks_run_in_parallel` (D2)
|
||||
- `test_fd_exhaustion_handling` (E — 5.3)
|
||||
- `test_corrupt_chunk_auto_refetched` (F — 6.4)
|
||||
- `test_missing_chunk_triggers_origin_fetch` (F — 6.4)
|
||||
- `test_passthrough_mode_when_cache_disk_dead` (F — 6.6)
|
||||
|
||||
---
|
||||
|
||||
## 2. Background
|
||||
|
||||
After Phase A+B, the daemon survives signals, recovers from storage corruption on startup, supervises background tasks, and rejects oversized CAS writes. But:
|
||||
|
||||
1. **Health checks hang on dead origins** — `check_one()` calls `origin.health().await` with no timeout. A dead NAS (local origin pointing to network mount) blocks health monitoring for ALL origins because checks run sequentially.
|
||||
|
||||
2. **FUSE reads have no timeout** — `reader.read()` in the FUSE `read()` callback has no timeout. A slow or hung origin blocks the FUSE thread indefinitely.
|
||||
|
||||
3. **CAS size tracking is broken** — `calculate_size()` only scans top-level of `chunks_dir`, missing all chunks in shard subdirectories (`aa/bb/<hash>`). `current_size` is always ~0, eviction never triggers.
|
||||
|
||||
4. **Corrupt chunks return EIO** — when `verify_integrity()` detects a bad chunk, it returns `CasError::IntegrityError`. The reader propagates this as EIO to FUSE. It should auto-re-fetch from origin instead.
|
||||
|
||||
5. **No concurrent mount protection** — two `musicfs mount` commands can run simultaneously, corrupting SQLite and sled.
|
||||
|
||||
6. **fd exhaustion is unhandled** — no graceful behavior when file descriptors run out.
|
||||
|
||||
---
|
||||
|
||||
## 3. Goals & Non-Goals
|
||||
|
||||
### 3.1 Goals
|
||||
|
||||
- Health checks complete within 5 seconds regardless of origin responsiveness
|
||||
- Health checks run in parallel (3 origins checked in ~5s, not ~15s)
|
||||
- FUSE reads timeout after 30 seconds (returns EIO, doesn't hang)
|
||||
- CAS size accounting is correct (recursive shard scan)
|
||||
- Corrupt/missing chunks are auto-re-fetched from origin transparently
|
||||
- PID file prevents concurrent mounts
|
||||
- fd exhaustion produces clean errors, not panics
|
||||
- All 6 remaining RED tests turn GREEN
|
||||
|
||||
### 3.2 Non-Goals
|
||||
|
||||
- Interrupted sync recovery (C1) — blocked on persistent state
|
||||
- systemd watchdog (C3) — useful but not critical yet
|
||||
- SIGHUP config reload (C4) — nice-to-have
|
||||
- Connection pooling (C5) — S3/SFTP origins are stubs
|
||||
- Event bus backpressure (C8) — low urgency
|
||||
- FUSE session recovery (C10) — complex edge case
|
||||
- Offline mode state machine (D3) — needs broader design
|
||||
- DNS failure handling (D5) — depends on C5
|
||||
- Stale-data awareness (D6) — low severity for music FS
|
||||
|
||||
---
|
||||
|
||||
## 4. Proposed Design
|
||||
|
||||
### 4.1 Implementation Order
|
||||
|
||||
```
|
||||
4.2 Health check timeout + parallel checks (2 RED tests, independent)
|
||||
↓
|
||||
4.3 Fix CAS calculate_size() (independent, unblocks eviction)
|
||||
↓
|
||||
4.4 FUSE read timeout (independent)
|
||||
↓
|
||||
4.5 CAS chunk auto-re-fetch on corruption (2 RED tests)
|
||||
↓
|
||||
4.6 PID file / flock (independent)
|
||||
↓
|
||||
4.7 fd exhaustion handling (1 RED test)
|
||||
```
|
||||
|
||||
### 4.2 Issues D1+D2: Health Check Timeout + Parallel Checks
|
||||
|
||||
**Problem**: `check_one()` awaits `origin.health()` with no timeout. `check_all()` iterates sequentially. One hung origin blocks everything.
|
||||
|
||||
#### Step 1: No stubs needed
|
||||
|
||||
#### Step 2: RED tests already exist
|
||||
|
||||
`test_local_origin_health_check_has_timeout` — FaultyOrigin with `TimeoutMs(5000)`, asserts check completes in <2s.
|
||||
|
||||
`test_health_checks_run_in_parallel` — 3 origins each with `TimeoutMs(200)`, asserts `check_all()` completes in <350ms (parallel), not ~600ms (sequential).
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-origins/src/health.rs`
|
||||
|
||||
Wrap `origin.health()` in `check_one()` with timeout:
|
||||
|
||||
```rust
|
||||
async fn check_one(&self, id: &OriginId, origin: &Arc<dyn Origin>) {
|
||||
let start = Instant::now();
|
||||
let health_timeout = Duration::from_secs(5);
|
||||
|
||||
let status = match tokio::time::timeout(health_timeout, origin.health()).await {
|
||||
Ok(status) => status,
|
||||
Err(_) => {
|
||||
warn!(origin_id = %id, timeout_ms = health_timeout.as_millis() as u64,
|
||||
"Health check timed out");
|
||||
HealthStatus::Unhealthy
|
||||
}
|
||||
};
|
||||
|
||||
let latency_ms = start.elapsed().as_millis() as u64;
|
||||
// ... rest unchanged
|
||||
}
|
||||
```
|
||||
|
||||
Change `check_all()` to use `futures::future::join_all`:
|
||||
|
||||
```rust
|
||||
pub async fn check_all(&self) {
|
||||
let origins: Vec<_> = self.origins.iter()
|
||||
.map(|e| (e.key().clone(), e.value().clone()))
|
||||
.collect();
|
||||
|
||||
let checks: Vec<_> = origins.iter()
|
||||
.map(|(id, origin)| self.check_one(id, origin))
|
||||
.collect();
|
||||
|
||||
futures::future::join_all(checks).await;
|
||||
}
|
||||
```
|
||||
|
||||
Add `futures` to `musicfs-origins/Cargo.toml` (or use `tokio::join!` macro if count is small/known).
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_local_origin_health_check
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_health_checks_run_in_parallel
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Issue C6: Fix CAS calculate_size()
|
||||
|
||||
**Problem**: `calculate_size()` only scans direct children of `chunks_dir`. Chunks live in shard subdirectories (`chunks/aa/bb/<hash>`). Size is always ~0, eviction never triggers.
|
||||
|
||||
#### Step 1: No stubs needed
|
||||
|
||||
#### Step 2: Test
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_cas_size_tracking_is_correct() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let config = CasConfig { chunks_dir: dir.path().join("chunks"), max_size: 10_000_000, shard_levels: 2 };
|
||||
let store = CasStore::open(config).await.unwrap();
|
||||
|
||||
let data = vec![0u8; 1000];
|
||||
store.put(&data).await.unwrap();
|
||||
|
||||
// Size should reflect the chunk we just wrote (~1000 bytes)
|
||||
assert!(store.current_size() >= 1000, "current_size should track chunk data, got {}", store.current_size());
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cas/src/store.rs` — make `calculate_size` recursive:
|
||||
|
||||
```rust
|
||||
async fn calculate_size(dir: &Path) -> u64 {
|
||||
Self::calculate_size_recursive(dir).await
|
||||
}
|
||||
|
||||
#[async recursion::async_recursion]
|
||||
async fn calculate_size_recursive(dir: &Path) -> u64 {
|
||||
let mut size = 0u64;
|
||||
if let Ok(mut entries) = fs::read_dir(dir).await {
|
||||
while let Ok(Some(entry)) = entries.next_entry().await {
|
||||
if let Ok(meta) = entry.metadata().await {
|
||||
if meta.is_file() {
|
||||
size += meta.len();
|
||||
} else if meta.is_dir() {
|
||||
// Skip sled index directory
|
||||
let name = entry.file_name();
|
||||
if name != "index.sled" {
|
||||
size += Self::calculate_size_recursive(&entry.path()).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
size
|
||||
}
|
||||
```
|
||||
|
||||
Alternative without `async_recursion` (use `Box::pin`):
|
||||
|
||||
```rust
|
||||
fn calculate_size_recursive(dir: &Path) -> Pin<Box<dyn Future<Output = u64> + Send + '_>> {
|
||||
Box::pin(async move {
|
||||
let mut size = 0u64;
|
||||
if let Ok(mut entries) = fs::read_dir(dir).await {
|
||||
while let Ok(Some(entry)) = entries.next_entry().await {
|
||||
if let Ok(meta) = entry.metadata().await {
|
||||
if meta.is_file() {
|
||||
size += meta.len();
|
||||
} else if meta.is_dir() {
|
||||
let name = entry.file_name();
|
||||
if name != "index.sled" {
|
||||
size += Self::calculate_size_recursive(&entry.path()).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
size
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Issue C7: FUSE Read Timeout
|
||||
|
||||
**Problem**: FUSE `read()` calls `handle.block_on(reader.read(...))` with no timeout. A slow origin blocks the entire FUSE thread.
|
||||
|
||||
#### Step 1: No stubs needed
|
||||
|
||||
#### Step 2: Test
|
||||
|
||||
```rust
|
||||
#[tokio::test]
|
||||
async fn test_fuse_read_timeout_returns_eio() {
|
||||
// Uses FaultyOrigin with TimeoutMs(60_000) — simulates hung read
|
||||
// FUSE read should timeout at 30s and return EIO, not hang forever
|
||||
// (This test validates the timeout wrapper, not actual FUSE mount)
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-fuse/src/filesystem.rs` — wrap the read with timeout:
|
||||
|
||||
```rust
|
||||
fn read(&mut self, _req: &Request, ino: u64, _fh: u64, offset: i64, size: u32, _flags: i32, _lock_owner: Option<u64>, reply: ReplyData) {
|
||||
// ... file_id lookup unchanged ...
|
||||
|
||||
let reader = reader.clone();
|
||||
let handle = self.runtime_handle.clone();
|
||||
let result = std::thread::scope(|_| {
|
||||
handle.block_on(async {
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(30),
|
||||
reader.read(file_id, offset as u64, size),
|
||||
).await
|
||||
})
|
||||
});
|
||||
|
||||
match result {
|
||||
Ok(Ok(data)) => {
|
||||
trace!(ino, bytes_read = data.len(), "read successful");
|
||||
reply.data(&data);
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
warn!(ino, error = %e, "read failed");
|
||||
reply.error(libc::EIO);
|
||||
}
|
||||
Err(_timeout) => {
|
||||
warn!(ino, offset, size, "read timed out after 30s");
|
||||
reply.error(libc::EIO);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4.5 Issues 6.4: CAS Chunk Auto-Re-Fetch on Corruption/Missing
|
||||
|
||||
**Problem**: When `store.get()` finds a corrupt or missing chunk, it returns an error. The reader propagates this as EIO to FUSE. It should try to re-fetch the chunk from the origin instead.
|
||||
|
||||
#### Step 1: No stubs needed — modify `FileReader::read()`
|
||||
|
||||
#### Step 2: RED tests already exist
|
||||
|
||||
`test_corrupt_chunk_auto_refetched` — corrupts chunk file on disk, expects read to succeed (re-fetched from origin).
|
||||
|
||||
`test_missing_chunk_triggers_origin_fetch` — deletes chunk file, expects read to succeed.
|
||||
|
||||
Both currently fail because the reader doesn't attempt re-fetch on chunk errors.
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cas/src/reader.rs` — add retry-with-refetch in the chunk read loop:
|
||||
|
||||
```rust
|
||||
pub async fn read(&self, file_id: FileId, offset: u64, size: u32) -> Result<Bytes, ReaderError> {
|
||||
let manifest = self.get_or_fetch_manifest(file_id).await?;
|
||||
|
||||
// ... offset/end calculation unchanged ...
|
||||
|
||||
for chunk_ref in &manifest.chunks {
|
||||
// ... range check unchanged ...
|
||||
|
||||
let chunk_data = match self.store.get(&chunk_ref.hash).await {
|
||||
Ok(data) => data,
|
||||
Err(CasError::IntegrityError { .. }) | Err(CasError::NotFound(_)) => {
|
||||
// Chunk is corrupt or missing — try to re-fetch from origin
|
||||
warn!(hash = %chunk_ref.hash, "Chunk corrupt/missing, attempting re-fetch");
|
||||
if let Some(fetcher) = &self.fetcher {
|
||||
// Re-fetch the entire file (will re-chunk and store)
|
||||
let new_manifest = fetcher.fetch_file(file_id).await?;
|
||||
// Update cached manifest
|
||||
self.manifests.write().insert(file_id, new_manifest);
|
||||
// Retry the get
|
||||
self.store.get(&chunk_ref.hash).await?
|
||||
} else {
|
||||
return Err(ReaderError::Cas(CasError::NotFound(chunk_ref.hash.as_hex())));
|
||||
}
|
||||
}
|
||||
Err(e) => return Err(ReaderError::Cas(e)),
|
||||
};
|
||||
|
||||
// ... slice extraction unchanged ...
|
||||
}
|
||||
|
||||
Ok(result.freeze())
|
||||
}
|
||||
```
|
||||
|
||||
**Important**: The re-fetch downloads the entire file from origin and re-chunks it. For a single corrupt chunk this is wasteful (fetches all chunks to fix one), but it's the simplest correct approach. Chunk-level re-fetch would require the origin to support byte-range reads mapped to chunk boundaries — possible but complex. The file-level approach reuses existing `fetch_file()` logic.
|
||||
|
||||
#### Step 4: Verify
|
||||
|
||||
```bash
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_corrupt_chunk
|
||||
cargo test -p musicfs-test-utils --test resilience -- test_missing_chunk
|
||||
```
|
||||
|
||||
**Note on test updates**: The existing RED tests reference `store.chunk_path()` which is private. The tests will need to either:
|
||||
- Make `chunk_path()` pub(crate) or add a test helper
|
||||
- Or construct the path manually using the sharding logic
|
||||
|
||||
The tests also need a `ContentFetcher` with a real `LocalOrigin` to re-fetch from. The current tests create a CAS store but no fetcher — they need to be updated to include the full pipeline.
|
||||
|
||||
---
|
||||
|
||||
### 4.6 Issue C9: PID File / flock
|
||||
|
||||
**Problem**: Two `musicfs mount` commands can run simultaneously, both writing to the same SQLite/sled files.
|
||||
|
||||
#### Step 1: No stubs needed
|
||||
|
||||
#### Step 2: Test
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
fn test_pid_file_prevents_concurrent_mount() {
|
||||
let dir = TempDir::new().unwrap();
|
||||
let lock_path = dir.path().join("musicfs.lock");
|
||||
|
||||
// First lock succeeds
|
||||
let lock1 = try_acquire_lock(&lock_path);
|
||||
assert!(lock1.is_ok());
|
||||
|
||||
// Second lock fails
|
||||
let lock2 = try_acquire_lock(&lock_path);
|
||||
assert!(lock2.is_err());
|
||||
|
||||
// Release first, second succeeds
|
||||
drop(lock1);
|
||||
let lock3 = try_acquire_lock(&lock_path);
|
||||
assert!(lock3.is_ok());
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
**File**: `musicfs-cli/src/main.rs`
|
||||
|
||||
```rust
|
||||
use std::fs::File;
|
||||
use std::os::unix::io::AsRawFd;
|
||||
|
||||
struct LockFile {
|
||||
_file: File,
|
||||
}
|
||||
|
||||
fn try_acquire_lock(path: &Path) -> Result<LockFile> {
|
||||
let file = File::create(path).context("Failed to create lock file")?;
|
||||
let fd = file.as_raw_fd();
|
||||
|
||||
let ret = unsafe { libc::flock(fd, libc::LOCK_EX | libc::LOCK_NB) };
|
||||
if ret != 0 {
|
||||
let err = std::io::Error::last_os_error();
|
||||
if err.kind() == std::io::ErrorKind::WouldBlock {
|
||||
anyhow::bail!("MusicFS is already running (lock file: {:?})", path);
|
||||
}
|
||||
return Err(err).context("Failed to acquire lock");
|
||||
}
|
||||
|
||||
// Write PID for debugging
|
||||
use std::io::Write;
|
||||
let mut f = &file;
|
||||
writeln!(f, "{}", std::process::id())?;
|
||||
|
||||
Ok(LockFile { _file: file })
|
||||
}
|
||||
```
|
||||
|
||||
Call in `run_mount()` before mounting:
|
||||
|
||||
```rust
|
||||
let lock_path = cache_dir.join("musicfs.lock");
|
||||
let _lock = try_acquire_lock(&lock_path)
|
||||
.context("Failed to acquire lock — is another instance running?")?;
|
||||
```
|
||||
|
||||
Lock is released automatically when `_lock` is dropped (process exit or scope end).
|
||||
|
||||
---
|
||||
|
||||
### 4.7 Issue 5.3: fd Exhaustion Handling
|
||||
|
||||
**Problem**: When fd limit is hit, operations fail with EMFILE. Currently this propagates as panics or unhelpful errors.
|
||||
|
||||
#### Step 1: Replace the `todo!()` test
|
||||
|
||||
#### Step 2: Test
|
||||
|
||||
```rust
|
||||
#[test]
|
||||
#[cfg(target_os = "linux")]
|
||||
fn test_fd_exhaustion_handling() {
|
||||
use rlimit::{Resource, setrlimit, getrlimit};
|
||||
|
||||
let (orig_soft, orig_hard) = getrlimit(Resource::NOFILE).unwrap();
|
||||
|
||||
// Set very low limit
|
||||
setrlimit(Resource::NOFILE, 64, 64).unwrap();
|
||||
|
||||
let dir = TempDir::new().unwrap();
|
||||
let rt = tokio::runtime::Runtime::new().unwrap();
|
||||
|
||||
let result = rt.block_on(async {
|
||||
CasStore::open(CasConfig {
|
||||
chunks_dir: dir.path().join("chunks"),
|
||||
max_size: 1_000_000,
|
||||
shard_levels: 2,
|
||||
}).await
|
||||
});
|
||||
|
||||
// Should either succeed (sled uses fewer than 64 fds) or fail gracefully
|
||||
// Must NOT panic
|
||||
match result {
|
||||
Ok(_store) => { /* lucky — enough fds */ }
|
||||
Err(e) => {
|
||||
// Error message should be meaningful
|
||||
let msg = format!("{}", e);
|
||||
assert!(!msg.contains("panic"), "Should not panic on fd exhaustion");
|
||||
}
|
||||
}
|
||||
|
||||
setrlimit(Resource::NOFILE, orig_soft, orig_hard).unwrap();
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 3: Implementation
|
||||
|
||||
This is primarily a **test** — verifying that existing code handles fd exhaustion without panicking. The fix is ensuring all I/O paths return `Result` rather than `.unwrap()` on file operations. Phase A's RwLock migration already removed the biggest panic source. The remaining `.unwrap()` calls are in test code only.
|
||||
|
||||
No production code change required if existing error paths handle I/O errors correctly. The test validates this.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Cutting Concerns
|
||||
|
||||
### 5.1 Observability
|
||||
|
||||
- Health check timeout logged at WARN with origin_id and timeout duration
|
||||
- FUSE read timeout logged at WARN with inode, offset, size
|
||||
- CAS chunk re-fetch logged at WARN with chunk hash
|
||||
- PID file path logged at INFO on lock acquisition
|
||||
|
||||
### 5.2 Performance
|
||||
|
||||
- Health checks now parallel: O(1) wall-clock time instead of O(N) per check cycle
|
||||
- FUSE read timeout: 30s cap prevents indefinite hangs but doesn't improve happy-path latency
|
||||
- `calculate_size()` recursive scan: runs once at startup, negligible cost
|
||||
|
||||
### 5.3 Testing
|
||||
|
||||
| Test | Status Before | Status After | Issue |
|
||||
|------|---------------|--------------|-------|
|
||||
| `test_local_origin_health_check_has_timeout` | ❌ FAILED | ✅ GREEN | D1 |
|
||||
| `test_health_checks_run_in_parallel` | ❌ FAILED | ✅ GREEN | D2 |
|
||||
| `test_fd_exhaustion_handling` | ❌ todo!() | ✅ GREEN | 5.3 |
|
||||
| `test_corrupt_chunk_auto_refetched` | ❌ FAILED | ✅ GREEN | 6.4 |
|
||||
| `test_missing_chunk_triggers_origin_fetch` | ❌ FAILED | ✅ GREEN | 6.4 |
|
||||
| `test_passthrough_mode_when_cache_disk_dead` | ❌ todo!() | ✅ GREEN | 6.6 |
|
||||
| `test_cas_size_tracking_is_correct` | NEW | ✅ GREEN | C6 |
|
||||
| `test_pid_file_prevents_concurrent_mount` | NEW | ✅ GREEN | C9 |
|
||||
|
||||
**Note on passthrough mode** (6.6): The test expects reads to succeed when the cache dir is read-only. With chunk auto-re-fetch (4.5), this partially works — if the origin is alive and the chunk isn't in cache, the fetcher reads from origin. But the fetcher tries to _write_ the chunk to CAS, which will fail on a read-only cache dir. The implementation needs a fallback path: if CAS write fails after origin fetch, return the data anyway without caching. This makes `test_passthrough_mode_when_cache_disk_dead` pass.
|
||||
|
||||
---
|
||||
|
||||
## 6. Alternatives Considered
|
||||
|
||||
### 6.1 Per-Origin Configurable Timeout vs Universal 5s
|
||||
|
||||
Could allow `health_check_timeout_ms` per origin config. Rejected for Phase C — universal 5s is correct for all current origin types. Can be made configurable later.
|
||||
|
||||
### 6.2 Chunk-Level Re-Fetch vs File-Level Re-Fetch
|
||||
|
||||
When one chunk is corrupt, we could re-fetch just that chunk's byte range from origin. Requires the origin to support byte-range reads and the system to know which byte range maps to which chunk. Complex. File-level re-fetch reuses existing `fetch_file()` and is correct, just slightly wasteful. Good enough for Phase C.
|
||||
|
||||
### 6.3 `advisory-lock` Crate vs Raw `flock`
|
||||
|
||||
The `advisory-lock` crate wraps flock nicely but adds a dependency for 10 lines of code. Raw `libc::flock` is simple enough and avoids the dependency.
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### 7.1 Task Sequence
|
||||
|
||||
| Day | Task | Issue | Effort | Tests |
|
||||
|-----|------|-------|--------|-------|
|
||||
| 1 (morning) | Health check timeout in `check_one()` | D1 | 1h | `test_local_origin_health_check_has_timeout` → GREEN |
|
||||
| 1 (morning) | Parallel `check_all()` with `join_all` | D2 | 1h | `test_health_checks_run_in_parallel` → GREEN |
|
||||
| 1 (afternoon) | Fix `calculate_size()` recursion | C6 | 1h | `test_cas_size_tracking_is_correct` → GREEN |
|
||||
| 1 (afternoon) | FUSE read timeout wrapper | C7 | 1h | New timeout test |
|
||||
| 2 (morning) | CAS chunk auto-re-fetch on corruption/missing | 6.4 | 3h | `test_corrupt_chunk_auto_refetched` + `test_missing_chunk_triggers_origin_fetch` → GREEN |
|
||||
| 2 (afternoon) | Passthrough fallback (CAS write fails → return data anyway) | 6.6 | 1h | `test_passthrough_mode_when_cache_disk_dead` → GREEN |
|
||||
| 3 (morning) | PID file / flock | C9 | 1h | `test_pid_file_prevents_concurrent_mount` → GREEN |
|
||||
| 3 (morning) | fd exhaustion test | 5.3 | 1h | `test_fd_exhaustion_handling` → GREEN |
|
||||
| 3 (afternoon) | Integration + regression testing | — | 2h | Full `cargo test` |
|
||||
| 4 | Buffer | — | 4h | — |
|
||||
|
||||
### 7.2 Verification Checklist
|
||||
|
||||
After all tasks:
|
||||
|
||||
- [ ] `cargo check` — zero errors, zero warnings
|
||||
- [ ] `cargo test --workspace --exclude musicfs-grpc` — all pass
|
||||
- [ ] `cargo test -p musicfs-test-utils --test resilience` — **25 passed, 0 failed** (all RED tests GREEN)
|
||||
- [ ] `cargo clippy` — no new warnings
|
||||
|
||||
---
|
||||
|
||||
## 8. Files Changed
|
||||
|
||||
| File | Change | Issue |
|
||||
|------|--------|-------|
|
||||
| `musicfs-origins/src/health.rs` | Timeout in `check_one()`, `join_all` in `check_all()` | D1, D2 |
|
||||
| `musicfs-origins/Cargo.toml` | Add `futures` dependency (for `join_all`) | D2 |
|
||||
| `musicfs-cas/src/store.rs` | Recursive `calculate_size()`, skip `index.sled` dir | C6 |
|
||||
| `musicfs-fuse/src/filesystem.rs` | `tokio::time::timeout(30s)` around reader.read() | C7 |
|
||||
| `musicfs-cas/src/reader.rs` | Auto-re-fetch on `IntegrityError` / `NotFound` | 6.4 |
|
||||
| `musicfs-cas/src/fetcher.rs` | Possible: make `fetch_file` return data even if CAS write fails | 6.6 |
|
||||
| `musicfs-cli/src/main.rs` | PID file with flock, fd exhaustion handling | C9, 5.3 |
|
||||
| `musicfs-test-utils/tests/resilience.rs` | Replace remaining todo!()s, add new tests, update chunk tests with fetcher pipeline | all |
|
||||
|
||||
---
|
||||
|
||||
## 9. Glossary / References
|
||||
|
||||
| Term | Definition |
|
||||
|------|------------|
|
||||
| **join_all** | `futures::future::join_all` — runs multiple futures concurrently, waits for all |
|
||||
| **flock** | Advisory file locking syscall — `LOCK_EX | LOCK_NB` for exclusive non-blocking |
|
||||
| **EMFILE** | "Too many open files" errno — returned when process fd limit is reached |
|
||||
| **Passthrough mode** | When CAS is unavailable, read directly from origin without caching |
|
||||
|
||||
| Document | Path |
|
||||
|----------|------|
|
||||
| Phase A plan | [phase-a-stop-dying.md](phase-a-stop-dying.md) |
|
||||
| Phase B plan | [phase-b-crash-recovery.md](phase-b-crash-recovery.md) |
|
||||
| Resilience audit | [resilience-fault-tolerance.md](resilience-fault-tolerance.md) |
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,771 +0,0 @@
|
||||
# Week 2: Metadata Extraction
|
||||
|
||||
**Phase**: 1 (MVP)
|
||||
**Prerequisites**: Week 1 (Foundation)
|
||||
**Estimated effort**: 5 days
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Implement audio metadata extraction using symphonia and create SQLite schema for metadata cache.
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Task | Crate | Files | Done |
|
||||
|------|-------|-------|------|
|
||||
| Audio parsing | musicfs-metadata | `lib.rs`, `parser.rs` | [ ] |
|
||||
| Format handlers | musicfs-metadata | `formats/*.rs` | [ ] |
|
||||
| SQLite schema | musicfs-cache | `schema.sql`, `db.rs` | [ ] |
|
||||
| Metadata cache | musicfs-cache | `metadata.rs` | [ ] |
|
||||
|
||||
---
|
||||
|
||||
## Task 0: Extend AudioMeta in `musicfs-core`
|
||||
|
||||
Add `lyrics` and `composer` fields to `AudioMeta` struct (FR-6.4):
|
||||
|
||||
```rust
|
||||
// In musicfs-core/src/types.rs, add to AudioMeta:
|
||||
pub struct AudioMeta {
|
||||
// ... existing fields ...
|
||||
pub lyrics: Option<String>,
|
||||
pub composer: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Metadata Parser (`musicfs-metadata`)
|
||||
|
||||
### 1.1 Create `Cargo.toml`
|
||||
|
||||
```toml
|
||||
[package]
|
||||
name = "musicfs-metadata"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
musicfs-core = { path = "../musicfs-core" }
|
||||
symphonia = { version = "0.5", features = ["all"] }
|
||||
thiserror.workspace = true
|
||||
tracing.workspace = true
|
||||
```
|
||||
|
||||
### 1.2 Create `src/lib.rs`
|
||||
|
||||
```rust
|
||||
mod parser;
|
||||
|
||||
pub use parser::MetadataParser;
|
||||
```
|
||||
|
||||
### 1.3 Create `src/parser.rs`
|
||||
|
||||
```rust
|
||||
use musicfs_core::{AudioFormat, AudioMeta, Result, Error};
|
||||
use std::io::{Read, Seek};
|
||||
use std::path::Path;
|
||||
use symphonia::core::codecs::CODEC_TYPE_NULL;
|
||||
use symphonia::core::formats::FormatOptions;
|
||||
use symphonia::core::io::MediaSourceStream;
|
||||
use symphonia::core::meta::MetadataOptions;
|
||||
use symphonia::core::probe::Hint;
|
||||
use tracing::debug;
|
||||
|
||||
/// Metadata extraction using symphonia (FR-6.1-6.5)
|
||||
pub struct MetadataParser;
|
||||
|
||||
impl MetadataParser {
|
||||
pub fn new() -> Self {
|
||||
Self
|
||||
}
|
||||
|
||||
/// Extract metadata from audio file
|
||||
pub fn parse_file(&self, path: &Path) -> Result<AudioMeta> {
|
||||
let file = std::fs::File::open(path)?;
|
||||
let ext = path.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.unwrap_or("");
|
||||
self.parse_reader(file, ext)
|
||||
}
|
||||
|
||||
/// Extract metadata from reader
|
||||
pub fn parse_reader<R: Read + Seek + Send + Sync + 'static>(
|
||||
&self,
|
||||
reader: R,
|
||||
extension: &str,
|
||||
) -> Result<AudioMeta> {
|
||||
let mss = MediaSourceStream::new(Box::new(reader), Default::default());
|
||||
|
||||
let mut hint = Hint::new();
|
||||
if !extension.is_empty() {
|
||||
hint.with_extension(extension);
|
||||
}
|
||||
|
||||
let format_opts = FormatOptions {
|
||||
enable_gapless: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let metadata_opts = MetadataOptions::default();
|
||||
|
||||
let probed = symphonia::default::get_probe()
|
||||
.format(&hint, mss, &format_opts, &metadata_opts)
|
||||
.map_err(|e| Error::Cache(format!("Failed to probe format: {}", e)))?;
|
||||
|
||||
let mut format = probed.format;
|
||||
let mut audio_meta = AudioMeta {
|
||||
format: AudioFormat::from_extension(extension),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
// Extract metadata from container
|
||||
if let Some(metadata) = format.metadata().current() {
|
||||
self.extract_tags(&mut audio_meta, metadata);
|
||||
}
|
||||
|
||||
// Also check probed metadata
|
||||
if let Some(metadata) = probed.metadata.current() {
|
||||
self.extract_tags(&mut audio_meta, metadata);
|
||||
}
|
||||
|
||||
// Get duration and codec info from track
|
||||
if let Some(track) = format.tracks().iter().find(|t| t.codec_params.codec != CODEC_TYPE_NULL) {
|
||||
let params = &track.codec_params;
|
||||
|
||||
if let Some(n_frames) = params.n_frames {
|
||||
if let Some(sample_rate) = params.sample_rate {
|
||||
audio_meta.duration_ms = Some((n_frames as u64 * 1000) / sample_rate as u64);
|
||||
audio_meta.sample_rate = Some(sample_rate);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(bits_per_sample) = params.bits_per_sample {
|
||||
if let Some(sample_rate) = params.sample_rate {
|
||||
if let Some(channels) = params.channels {
|
||||
audio_meta.bitrate = Some(
|
||||
bits_per_sample * sample_rate * channels.count() as u32 / 1000
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
debug!("Parsed metadata: {:?}", audio_meta);
|
||||
Ok(audio_meta)
|
||||
}
|
||||
|
||||
fn extract_tags(&self, meta: &mut AudioMeta, metadata: &symphonia::core::meta::MetadataRevision) {
|
||||
use symphonia::core::meta::StandardTagKey;
|
||||
|
||||
for tag in metadata.tags() {
|
||||
if let Some(std_key) = tag.std_key {
|
||||
let value = tag.value.to_string();
|
||||
match std_key {
|
||||
StandardTagKey::TrackTitle => meta.title = Some(value),
|
||||
StandardTagKey::Artist => meta.artist = Some(value),
|
||||
StandardTagKey::Album => meta.album = Some(value),
|
||||
StandardTagKey::AlbumArtist => meta.album_artist = Some(value),
|
||||
StandardTagKey::Genre => meta.genre = Some(value),
|
||||
StandardTagKey::TrackNumber => {
|
||||
meta.track = value.split('/').next()
|
||||
.and_then(|s| s.parse().ok());
|
||||
}
|
||||
StandardTagKey::DiscNumber => {
|
||||
meta.disc = value.split('/').next()
|
||||
.and_then(|s| s.parse().ok());
|
||||
}
|
||||
StandardTagKey::Date | StandardTagKey::ReleaseDate => {
|
||||
meta.year = value.chars().take(4).collect::<String>()
|
||||
.parse().ok();
|
||||
}
|
||||
StandardTagKey::Lyrics => {
|
||||
meta.lyrics = Some(value);
|
||||
}
|
||||
StandardTagKey::Composer => {
|
||||
meta.composer = Some(value);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for MetadataParser {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Cache Database (`musicfs-cache`)
|
||||
|
||||
### 2.1 Create `Cargo.toml`
|
||||
|
||||
```toml
|
||||
[package]
|
||||
name = "musicfs-cache"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
[dependencies]
|
||||
musicfs-core = { path = "../musicfs-core" }
|
||||
rusqlite = { workspace = true, features = ["bundled"] }
|
||||
sled.workspace = true
|
||||
tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
thiserror.workspace = true
|
||||
serde.workspace = true
|
||||
rmp-serde.workspace = true
|
||||
```
|
||||
|
||||
### 2.2 Create `src/lib.rs`
|
||||
|
||||
```rust
|
||||
mod db;
|
||||
mod metadata;
|
||||
|
||||
pub use db::Database;
|
||||
pub use metadata::MetadataCache;
|
||||
```
|
||||
|
||||
### 2.3 Create `src/schema.sql`
|
||||
|
||||
```sql
|
||||
-- MusicFS Metadata Cache Schema
|
||||
-- Per architecture.md section 4.3.6
|
||||
-- NOTE: Chunk index stored in sled (chunks.sled/), NOT SQLite
|
||||
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA foreign_keys = ON;
|
||||
PRAGMA synchronous = NORMAL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS files (
|
||||
id INTEGER PRIMARY KEY,
|
||||
origin_id TEXT NOT NULL,
|
||||
real_path TEXT NOT NULL,
|
||||
virtual_path TEXT NOT NULL,
|
||||
|
||||
-- Audio metadata (FR-6.1-6.5)
|
||||
title TEXT,
|
||||
artist TEXT,
|
||||
album TEXT,
|
||||
album_artist TEXT,
|
||||
genre TEXT,
|
||||
year INTEGER,
|
||||
track INTEGER,
|
||||
disc INTEGER,
|
||||
duration_ms INTEGER,
|
||||
bitrate INTEGER,
|
||||
sample_rate INTEGER,
|
||||
format TEXT,
|
||||
|
||||
-- Sync state
|
||||
origin_mtime INTEGER NOT NULL,
|
||||
origin_size INTEGER NOT NULL,
|
||||
content_hash TEXT, -- hex-encoded xxHash64
|
||||
chunk_manifest BLOB, -- msgpack: [(chunk_hash, offset, size)]
|
||||
last_sync INTEGER NOT NULL DEFAULT (strftime('%s', 'now')),
|
||||
|
||||
UNIQUE(origin_id, real_path)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS artwork (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_id INTEGER NOT NULL REFERENCES files(id) ON DELETE CASCADE,
|
||||
art_type TEXT NOT NULL, -- 'front', 'back', 'disc'
|
||||
chunk_hash TEXT NOT NULL, -- hex-encoded reference to CAS
|
||||
width INTEGER,
|
||||
height INTEGER,
|
||||
mime_type TEXT,
|
||||
UNIQUE(file_id, art_type)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS collections (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
query_json TEXT NOT NULL, -- smart collection query
|
||||
created_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now')),
|
||||
updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
|
||||
);
|
||||
|
||||
-- Indexes for performance (NFR-1.1, NFR-1.2)
|
||||
CREATE INDEX IF NOT EXISTS idx_files_virtual ON files(virtual_path);
|
||||
CREATE INDEX IF NOT EXISTS idx_files_artist_album ON files(artist, album);
|
||||
CREATE INDEX IF NOT EXISTS idx_files_content_hash ON files(content_hash);
|
||||
CREATE INDEX IF NOT EXISTS idx_files_real ON files(origin_id, real_path); -- FR-7.3
|
||||
CREATE INDEX IF NOT EXISTS idx_files_origin ON files(origin_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_files_last_sync ON files(last_sync);
|
||||
CREATE INDEX IF NOT EXISTS idx_artwork_file ON artwork(file_id);
|
||||
```
|
||||
|
||||
### 2.4 Create `src/db.rs`
|
||||
|
||||
```rust
|
||||
use musicfs_core::{AudioMeta, ContentHash, Error, FileId, FileMeta, OriginId, RealPath, Result, VirtualPath};
|
||||
use rusqlite::{params, Connection, OptionalExtension};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use tracing::{debug, info};
|
||||
|
||||
const SCHEMA: &str = include_str!("schema.sql");
|
||||
|
||||
/// SQLite database connection manager
|
||||
pub struct Database {
|
||||
conn: Arc<Mutex<Connection>>,
|
||||
}
|
||||
|
||||
impl Database {
|
||||
/// Open or create database at path
|
||||
pub fn open(path: &Path) -> Result<Self> {
|
||||
info!("Opening database at {:?}", path);
|
||||
|
||||
let conn = Connection::open(path)
|
||||
.map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
// Execute schema
|
||||
conn.execute_batch(SCHEMA)
|
||||
.map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
Ok(Self {
|
||||
conn: Arc::new(Mutex::new(conn)),
|
||||
})
|
||||
}
|
||||
|
||||
/// Open in-memory database (for testing)
|
||||
pub fn open_memory() -> Result<Self> {
|
||||
let conn = Connection::open_in_memory()
|
||||
.map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
conn.execute_batch(SCHEMA)
|
||||
.map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
Ok(Self {
|
||||
conn: Arc::new(Mutex::new(conn)),
|
||||
})
|
||||
}
|
||||
|
||||
/// Insert or update file metadata
|
||||
pub fn upsert_file(
|
||||
&self,
|
||||
origin_id: &OriginId,
|
||||
real_path: &Path,
|
||||
virtual_path: &VirtualPath,
|
||||
audio_meta: &AudioMeta,
|
||||
origin_mtime: SystemTime,
|
||||
origin_size: u64,
|
||||
) -> Result<FileId> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
|
||||
let mtime_secs = origin_mtime
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs() as i64;
|
||||
|
||||
conn.execute(
|
||||
r#"
|
||||
INSERT INTO files (
|
||||
origin_id, real_path, virtual_path,
|
||||
title, artist, album, album_artist, genre,
|
||||
year, track, disc,
|
||||
duration_ms, bitrate, sample_rate, format,
|
||||
origin_mtime, origin_size
|
||||
) VALUES (
|
||||
?1, ?2, ?3,
|
||||
?4, ?5, ?6, ?7, ?8,
|
||||
?9, ?10, ?11,
|
||||
?12, ?13, ?14, ?15,
|
||||
?16, ?17
|
||||
)
|
||||
ON CONFLICT(origin_id, real_path) DO UPDATE SET
|
||||
virtual_path = excluded.virtual_path,
|
||||
title = excluded.title,
|
||||
artist = excluded.artist,
|
||||
album = excluded.album,
|
||||
album_artist = excluded.album_artist,
|
||||
genre = excluded.genre,
|
||||
year = excluded.year,
|
||||
track = excluded.track,
|
||||
disc = excluded.disc,
|
||||
duration_ms = excluded.duration_ms,
|
||||
bitrate = excluded.bitrate,
|
||||
sample_rate = excluded.sample_rate,
|
||||
format = excluded.format,
|
||||
origin_mtime = excluded.origin_mtime,
|
||||
origin_size = excluded.origin_size,
|
||||
last_sync = strftime('%s', 'now')
|
||||
"#,
|
||||
params![
|
||||
&origin_id.0,
|
||||
real_path.to_string_lossy(),
|
||||
virtual_path.as_str(),
|
||||
&audio_meta.title,
|
||||
&audio_meta.artist,
|
||||
&audio_meta.album,
|
||||
&audio_meta.album_artist,
|
||||
&audio_meta.genre,
|
||||
&audio_meta.year,
|
||||
&audio_meta.track,
|
||||
&audio_meta.disc,
|
||||
&audio_meta.duration_ms.map(|d| d as i64),
|
||||
&audio_meta.bitrate,
|
||||
&audio_meta.sample_rate,
|
||||
format!("{:?}", audio_meta.format),
|
||||
mtime_secs,
|
||||
origin_size as i64,
|
||||
],
|
||||
).map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
let id = conn.last_insert_rowid();
|
||||
debug!("Upserted file {} with id {}", virtual_path.as_str(), id);
|
||||
|
||||
Ok(FileId(id))
|
||||
}
|
||||
|
||||
/// Get file by virtual path
|
||||
pub fn get_file_by_virtual_path(&self, path: &VirtualPath) -> Result<Option<FileMeta>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
|
||||
conn.query_row(
|
||||
r#"
|
||||
SELECT id, origin_id, real_path, virtual_path,
|
||||
title, artist, album, album_artist, genre,
|
||||
year, track, disc,
|
||||
duration_ms, bitrate, sample_rate, format,
|
||||
origin_mtime, origin_size, content_hash
|
||||
FROM files
|
||||
WHERE virtual_path = ?1
|
||||
"#,
|
||||
params![path.as_str()],
|
||||
|row| {
|
||||
Ok(FileMeta {
|
||||
id: FileId(row.get(0)?),
|
||||
real_path: RealPath {
|
||||
origin_id: OriginId(row.get(1)?),
|
||||
path: PathBuf::from(row.get::<_, String>(2)?),
|
||||
},
|
||||
virtual_path: VirtualPath::new(row.get::<_, String>(3)?),
|
||||
audio: Some(AudioMeta {
|
||||
title: row.get(4)?,
|
||||
artist: row.get(5)?,
|
||||
album: row.get(6)?,
|
||||
album_artist: row.get(7)?,
|
||||
genre: row.get(8)?,
|
||||
year: row.get(9)?,
|
||||
track: row.get(10)?,
|
||||
disc: row.get(11)?,
|
||||
duration_ms: row.get::<_, Option<i64>>(12)?.map(|d| d as u64),
|
||||
bitrate: row.get(13)?,
|
||||
sample_rate: row.get(14)?,
|
||||
format: musicfs_core::AudioFormat::Unknown, // TODO: parse
|
||||
}),
|
||||
size: row.get::<_, i64>(17)? as u64,
|
||||
mtime: UNIX_EPOCH + std::time::Duration::from_secs(row.get::<_, i64>(16)? as u64),
|
||||
content_hash: row.get::<_, Option<Vec<u8>>>(18)?
|
||||
.map(|b| ContentHash(b.try_into().unwrap_or([0; 8]))),
|
||||
})
|
||||
},
|
||||
)
|
||||
.optional()
|
||||
.map_err(|e| Error::Database(e.to_string()))
|
||||
}
|
||||
|
||||
/// Get file by ID
|
||||
pub fn get_file_by_id(&self, id: FileId) -> Result<Option<FileMeta>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
|
||||
conn.query_row(
|
||||
"SELECT virtual_path FROM files WHERE id = ?1",
|
||||
params![id.0],
|
||||
|row| row.get::<_, String>(0),
|
||||
)
|
||||
.optional()
|
||||
.map_err(|e| Error::Database(e.to_string()))?
|
||||
.map(|vp| self.get_file_by_virtual_path(&VirtualPath::new(vp)))
|
||||
.transpose()
|
||||
.map(|o| o.flatten())
|
||||
}
|
||||
|
||||
/// List all files for an origin
|
||||
pub fn list_files(&self, origin_id: &OriginId) -> Result<Vec<FileMeta>> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT virtual_path FROM files WHERE origin_id = ?1"
|
||||
).map_err(|e| Error::Database(e.to_string()))?;
|
||||
|
||||
let paths: Vec<String> = stmt
|
||||
.query_map(params![&origin_id.0], |row| row.get(0))
|
||||
.map_err(|e| Error::Database(e.to_string()))?
|
||||
.filter_map(|r| r.ok())
|
||||
.collect();
|
||||
|
||||
drop(stmt);
|
||||
drop(conn);
|
||||
|
||||
paths
|
||||
.into_iter()
|
||||
.filter_map(|p| self.get_file_by_virtual_path(&VirtualPath::new(p)).ok().flatten())
|
||||
.collect::<Vec<_>>()
|
||||
.pipe(Ok)
|
||||
}
|
||||
|
||||
/// Delete file by ID
|
||||
pub fn delete_file(&self, id: FileId) -> Result<()> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
conn.execute("DELETE FROM files WHERE id = ?1", params![id.0])
|
||||
.map_err(|e| Error::Database(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Get file count
|
||||
pub fn file_count(&self) -> Result<u64> {
|
||||
let conn = self.conn.lock().unwrap();
|
||||
conn.query_row("SELECT COUNT(*) FROM files", [], |row| row.get::<_, i64>(0))
|
||||
.map(|c| c as u64)
|
||||
.map_err(|e| Error::Database(e.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
trait Pipe: Sized {
|
||||
fn pipe<T>(self, f: impl FnOnce(Self) -> T) -> T {
|
||||
f(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl<T> Pipe for T {}
|
||||
```
|
||||
|
||||
### 2.5 Create `src/metadata.rs`
|
||||
|
||||
```rust
|
||||
use crate::db::Database;
|
||||
use musicfs_core::{AudioMeta, FileMeta, OriginId, Result, VirtualPath};
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::time::SystemTime;
|
||||
|
||||
/// High-level metadata cache interface
|
||||
pub struct MetadataCache {
|
||||
db: Arc<Database>,
|
||||
}
|
||||
|
||||
impl MetadataCache {
|
||||
pub fn new(db: Arc<Database>) -> Self {
|
||||
Self { db }
|
||||
}
|
||||
|
||||
/// Store file metadata
|
||||
pub fn store(
|
||||
&self,
|
||||
origin_id: &OriginId,
|
||||
real_path: &Path,
|
||||
virtual_path: &VirtualPath,
|
||||
audio_meta: &AudioMeta,
|
||||
origin_mtime: SystemTime,
|
||||
origin_size: u64,
|
||||
) -> Result<()> {
|
||||
self.db.upsert_file(
|
||||
origin_id,
|
||||
real_path,
|
||||
virtual_path,
|
||||
audio_meta,
|
||||
origin_mtime,
|
||||
origin_size,
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Lookup by virtual path
|
||||
pub fn lookup(&self, path: &VirtualPath) -> Result<Option<FileMeta>> {
|
||||
self.db.get_file_by_virtual_path(path)
|
||||
}
|
||||
|
||||
/// Check if file exists and is fresh
|
||||
pub fn is_fresh(
|
||||
&self,
|
||||
origin_id: &OriginId,
|
||||
real_path: &Path,
|
||||
current_mtime: SystemTime,
|
||||
) -> Result<bool> {
|
||||
// TODO: Compare mtime with cached value
|
||||
Ok(false)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
### Unit Tests (`musicfs-metadata`)
|
||||
|
||||
```rust
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::io::Cursor;
|
||||
|
||||
#[test]
|
||||
fn test_parse_flac_metadata() {
|
||||
// Use a real FLAC file for testing
|
||||
// For CI, embed a small test file or use a fixture
|
||||
let parser = MetadataParser::new();
|
||||
|
||||
// This would need a real file path
|
||||
// let meta = parser.parse_file(Path::new("test.flac")).unwrap();
|
||||
// assert!(meta.title.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_audio_format_detection() {
|
||||
assert_eq!(AudioFormat::from_extension("flac"), AudioFormat::Flac);
|
||||
assert_eq!(AudioFormat::from_extension("mp3"), AudioFormat::Mp3);
|
||||
assert_eq!(AudioFormat::from_extension("opus"), AudioFormat::Opus);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Unit Tests (`musicfs-cache`)
|
||||
|
||||
```rust
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use musicfs_core::{AudioFormat, AudioMeta, OriginId, VirtualPath};
|
||||
use std::time::UNIX_EPOCH;
|
||||
|
||||
#[test]
|
||||
fn test_database_creation() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
assert_eq!(db.file_count().unwrap(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_upsert_and_retrieve() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
|
||||
let origin_id = OriginId::from("local");
|
||||
let real_path = Path::new("/music/test.flac");
|
||||
let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac");
|
||||
let audio_meta = AudioMeta {
|
||||
title: Some("Track".to_string()),
|
||||
artist: Some("Artist".to_string()),
|
||||
album: Some("Album".to_string()),
|
||||
track: Some(1),
|
||||
format: AudioFormat::Flac,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let id = db.upsert_file(
|
||||
&origin_id,
|
||||
real_path,
|
||||
&virtual_path,
|
||||
&audio_meta,
|
||||
UNIX_EPOCH,
|
||||
1000,
|
||||
).unwrap();
|
||||
|
||||
let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap();
|
||||
assert_eq!(retrieved.id, id);
|
||||
assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Track".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_upsert_updates_existing() {
|
||||
let db = Database::open_memory().unwrap();
|
||||
|
||||
let origin_id = OriginId::from("local");
|
||||
let real_path = Path::new("/music/test.flac");
|
||||
let virtual_path = VirtualPath::new("/Artist/Album/01 - Track.flac");
|
||||
|
||||
// First insert
|
||||
let meta1 = AudioMeta {
|
||||
title: Some("Original".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
db.upsert_file(&origin_id, real_path, &virtual_path, &meta1, UNIX_EPOCH, 1000).unwrap();
|
||||
|
||||
// Update
|
||||
let meta2 = AudioMeta {
|
||||
title: Some("Updated".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
db.upsert_file(&origin_id, real_path, &virtual_path, &meta2, UNIX_EPOCH, 1000).unwrap();
|
||||
|
||||
// Should still be 1 file
|
||||
assert_eq!(db.file_count().unwrap(), 1);
|
||||
|
||||
// Title should be updated
|
||||
let retrieved = db.get_file_by_virtual_path(&virtual_path).unwrap().unwrap();
|
||||
assert_eq!(retrieved.audio.as_ref().unwrap().title, Some("Updated".to_string()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_metadata_persistence() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let db_path = dir.path().join("test.db");
|
||||
|
||||
// Create and populate
|
||||
{
|
||||
let db = Database::open(&db_path).unwrap();
|
||||
db.upsert_file(
|
||||
&OriginId::from("local"),
|
||||
Path::new("/test.flac"),
|
||||
&VirtualPath::new("/Test.flac"),
|
||||
&AudioMeta::default(),
|
||||
UNIX_EPOCH,
|
||||
100,
|
||||
).unwrap();
|
||||
}
|
||||
|
||||
// Reopen and verify
|
||||
{
|
||||
let db = Database::open(&db_path).unwrap();
|
||||
assert_eq!(db.file_count().unwrap(), 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] Parse FLAC metadata (title, artist, album, track, duration)
|
||||
- [ ] Parse MP3 metadata (ID3v2 and ID3v1 fallback)
|
||||
- [ ] Parse Opus/Vorbis comments
|
||||
- [ ] Parse M4A/AAC metadata
|
||||
- [ ] Handle missing metadata gracefully (FR-6.5)
|
||||
- [ ] SQLite schema creates all tables
|
||||
- [ ] Metadata persists across daemon restarts (FR-7.4)
|
||||
- [ ] Upsert correctly updates existing records
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Run metadata tests
|
||||
cargo test -p musicfs-metadata
|
||||
|
||||
# Run cache tests
|
||||
cargo test -p musicfs-cache
|
||||
|
||||
# Test with real audio file
|
||||
cargo run --example parse_metadata -- /path/to/test.flac
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Week
|
||||
|
||||
Week 3 will implement the virtual path resolver and tree cache, connecting metadata to the FUSE operations.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,501 +0,0 @@
|
||||
# Week 4b: Origin-CAS Connector
|
||||
|
||||
**Phase**: 1 (MVP)
|
||||
**Prerequisites**: Week 4 (CAS & Chunk Caching)
|
||||
**Estimated effort**: 1 day
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
Bridge the gap between Origin (source files) and CAS (chunk cache) to enable actual file reads through FUSE. This implements the "cache miss" flow from architecture section 4.3.5.
|
||||
|
||||
**Problem**: Week 4 implemented CAS storage and FileReader, but there's no code that:
|
||||
1. Detects when requested chunks aren't cached
|
||||
2. Fetches data from Origin
|
||||
3. Stores chunks in CAS
|
||||
4. Creates ChunkManifest for the file
|
||||
|
||||
**Solution**: Create `ContentFetcher` that orchestrates Origin → CAS data flow on cache miss.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Reference
|
||||
|
||||
From architecture.md section 4.3.5 (Read Operation Activity):
|
||||
|
||||
```
|
||||
|CAS|
|
||||
:compute chunk range for [offset, offset+size];
|
||||
if (all chunks cached?) then (yes)
|
||||
:read from local chunk files;
|
||||
else (no)
|
||||
|OriginFederation|
|
||||
:select healthy origin by priority;
|
||||
:fetch missing byte range;
|
||||
|CAS|
|
||||
:chunk fetched data (CDC);
|
||||
:store chunks by hash;
|
||||
:update chunk manifest;
|
||||
endif
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Task | Crate | Files | Done |
|
||||
|------|-------|-------|------|
|
||||
| ContentFetcher implementation | musicfs-cas | `fetcher.rs` | [ ] |
|
||||
| FileId → FileMeta resolver | musicfs-cas | `fetcher.rs` | [ ] |
|
||||
| Update FileReader for cache-miss | musicfs-cas | `reader.rs` | [ ] |
|
||||
| Update FUSE with fetcher | musicfs-fuse | `filesystem.rs` | [ ] |
|
||||
| E2E test: cat file through FUSE | tests | `integration.rs` | [ ] |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: ContentFetcher
|
||||
|
||||
### 1.1 Create `musicfs-cas/src/fetcher.rs`
|
||||
|
||||
```rust
|
||||
use crate::{CasStore, ChunkManifest, ChunkRef};
|
||||
use musicfs_core::{Event, EventBus, FileId, FileMeta, OriginId, RealPath};
|
||||
use musicfs_origins::Origin;
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
use std::sync::{Arc, RwLock};
|
||||
use tracing::{debug, info};
|
||||
|
||||
pub struct ContentFetcher {
|
||||
store: Arc<CasStore>,
|
||||
origins: RwLock<HashMap<OriginId, Arc<dyn Origin>>>,
|
||||
file_meta: RwLock<HashMap<FileId, FileMeta>>,
|
||||
event_bus: Option<Arc<EventBus>>,
|
||||
}
|
||||
|
||||
impl ContentFetcher {
|
||||
pub fn new(store: Arc<CasStore>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
origins: RwLock::new(HashMap::new()),
|
||||
file_meta: RwLock::new(HashMap::new()),
|
||||
event_bus: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_event_bus(store: Arc<CasStore>, event_bus: Arc<EventBus>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
origins: RwLock::new(HashMap::new()),
|
||||
file_meta: RwLock::new(HashMap::new()),
|
||||
event_bus: Some(event_bus),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn register_origin(&self, origin: Arc<dyn Origin>) {
|
||||
let id = origin.id().clone();
|
||||
self.origins.write().unwrap().insert(id, origin);
|
||||
}
|
||||
|
||||
pub fn register_file(&self, meta: FileMeta) {
|
||||
self.file_meta.write().unwrap().insert(meta.id, meta);
|
||||
}
|
||||
|
||||
pub fn register_files(&self, files: impl IntoIterator<Item = FileMeta>) {
|
||||
let mut map = self.file_meta.write().unwrap();
|
||||
for meta in files {
|
||||
map.insert(meta.id, meta);
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn fetch_file(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
|
||||
let meta = {
|
||||
let files = self.file_meta.read().unwrap();
|
||||
files.get(&file_id).cloned()
|
||||
.ok_or(FetchError::FileNotFound(file_id))?
|
||||
};
|
||||
|
||||
let origin = {
|
||||
let origins = self.origins.read().unwrap();
|
||||
origins.get(&meta.real_path.origin_id).cloned()
|
||||
.ok_or_else(|| FetchError::OriginNotFound(meta.real_path.origin_id.clone()))?
|
||||
};
|
||||
|
||||
info!("Fetching file {:?} from origin {}", file_id, origin.id());
|
||||
|
||||
let data = origin.read(&meta.real_path.path, 0, meta.size as u32).await
|
||||
.map_err(|e| FetchError::OriginRead(e.to_string()))?;
|
||||
|
||||
let hash = self.store.put(&data).await
|
||||
.map_err(FetchError::Store)?;
|
||||
|
||||
let manifest = ChunkManifest {
|
||||
file_id,
|
||||
total_size: meta.size,
|
||||
chunks: vec![ChunkRef {
|
||||
hash,
|
||||
offset: 0,
|
||||
size: data.len() as u32,
|
||||
}],
|
||||
};
|
||||
|
||||
debug!("Created manifest for {:?}: {} bytes, 1 chunk", file_id, meta.size);
|
||||
|
||||
Ok(manifest)
|
||||
}
|
||||
|
||||
pub fn emit_access_event(&self, meta: &FileMeta, offset: u64, size: u32) {
|
||||
if let Some(bus) = &self.event_bus {
|
||||
bus.publish(Event::FileAccessed {
|
||||
path: meta.virtual_path.clone(),
|
||||
origin_id: meta.real_path.origin_id.clone(),
|
||||
offset,
|
||||
size,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn ensure_cached(&self, file_id: FileId) -> Result<ChunkManifest, FetchError> {
|
||||
self.fetch_file(file_id).await
|
||||
}
|
||||
|
||||
pub fn get_file_meta(&self, file_id: FileId) -> Option<FileMeta> {
|
||||
self.file_meta.read().unwrap().get(&file_id).cloned()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum FetchError {
|
||||
#[error("File not found: {0:?}")]
|
||||
FileNotFound(FileId),
|
||||
|
||||
#[error("Origin not found: {0}")]
|
||||
OriginNotFound(OriginId),
|
||||
|
||||
#[error("Origin read error: {0}")]
|
||||
OriginRead(String),
|
||||
|
||||
#[error("Store error: {0}")]
|
||||
Store(#[from] crate::CasError),
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::CasConfig;
|
||||
use musicfs_core::VirtualPath;
|
||||
use musicfs_origins::LocalOrigin;
|
||||
use std::path::PathBuf;
|
||||
use std::time::SystemTime;
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_fetch_file() {
|
||||
let cas_dir = TempDir::new().unwrap();
|
||||
let origin_dir = TempDir::new().unwrap();
|
||||
|
||||
std::fs::write(origin_dir.path().join("test.flac"), b"fake audio data").unwrap();
|
||||
|
||||
let config = CasConfig {
|
||||
chunks_dir: cas_dir.path().join("chunks"),
|
||||
..Default::default()
|
||||
};
|
||||
let store = Arc::new(CasStore::open(config).await.unwrap());
|
||||
let fetcher = ContentFetcher::new(store.clone());
|
||||
|
||||
let origin = Arc::new(LocalOrigin::new("local", origin_dir.path()));
|
||||
fetcher.register_origin(origin);
|
||||
|
||||
let meta = FileMeta {
|
||||
id: FileId(1),
|
||||
virtual_path: VirtualPath::new("/Artist/Album/test.flac"),
|
||||
real_path: RealPath {
|
||||
origin_id: OriginId::from("local"),
|
||||
path: PathBuf::from("/test.flac"),
|
||||
},
|
||||
size: 15,
|
||||
mtime: SystemTime::now(),
|
||||
content_hash: None,
|
||||
audio: None,
|
||||
};
|
||||
fetcher.register_file(meta);
|
||||
|
||||
let manifest = fetcher.fetch_file(FileId(1)).await.unwrap();
|
||||
assert_eq!(manifest.total_size, 15);
|
||||
assert_eq!(manifest.chunks.len(), 1);
|
||||
|
||||
let data = store.get(&manifest.chunks[0].hash).await.unwrap();
|
||||
assert_eq!(&data[..], b"fake audio data");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_fetch_file_not_found() {
|
||||
let cas_dir = TempDir::new().unwrap();
|
||||
let config = CasConfig {
|
||||
chunks_dir: cas_dir.path().join("chunks"),
|
||||
..Default::default()
|
||||
};
|
||||
let store = Arc::new(CasStore::open(config).await.unwrap());
|
||||
let fetcher = ContentFetcher::new(store);
|
||||
|
||||
let result = fetcher.fetch_file(FileId(999)).await;
|
||||
assert!(matches!(result, Err(FetchError::FileNotFound(_))));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_fetch_emits_event() {
|
||||
let cas_dir = TempDir::new().unwrap();
|
||||
let origin_dir = TempDir::new().unwrap();
|
||||
std::fs::write(origin_dir.path().join("test.flac"), b"audio").unwrap();
|
||||
|
||||
let config = CasConfig {
|
||||
chunks_dir: cas_dir.path().join("chunks"),
|
||||
..Default::default()
|
||||
};
|
||||
let store = Arc::new(CasStore::open(config).await.unwrap());
|
||||
let event_bus = Arc::new(EventBus::default());
|
||||
let mut rx = event_bus.subscribe();
|
||||
|
||||
let fetcher = ContentFetcher::with_event_bus(store, event_bus);
|
||||
let origin = Arc::new(LocalOrigin::new("local", origin_dir.path()));
|
||||
fetcher.register_origin(origin);
|
||||
|
||||
let meta = FileMeta {
|
||||
id: FileId(1),
|
||||
virtual_path: VirtualPath::new("/Artist/test.flac"),
|
||||
real_path: RealPath {
|
||||
origin_id: OriginId::from("local"),
|
||||
path: PathBuf::from("/test.flac"),
|
||||
},
|
||||
size: 5,
|
||||
mtime: SystemTime::now(),
|
||||
content_hash: None,
|
||||
audio: None,
|
||||
};
|
||||
fetcher.register_file(meta.clone());
|
||||
|
||||
fetcher.emit_access_event(&meta, 0, 5);
|
||||
|
||||
let event = rx.try_recv().unwrap();
|
||||
assert!(matches!(event, Event::FileAccessed { .. }));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Update FileReader
|
||||
|
||||
### 2.1 Update `musicfs-cas/src/reader.rs`
|
||||
|
||||
Add fetcher integration for cache-miss handling:
|
||||
|
||||
```rust
|
||||
use crate::fetcher::{ContentFetcher, FetchError};
|
||||
|
||||
pub struct FileReader {
|
||||
store: Arc<CasStore>,
|
||||
fetcher: Option<Arc<ContentFetcher>>,
|
||||
manifests: RwLock<HashMap<FileId, ChunkManifest>>,
|
||||
}
|
||||
|
||||
impl FileReader {
|
||||
pub fn new(store: Arc<CasStore>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
fetcher: None,
|
||||
manifests: RwLock::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn with_fetcher(store: Arc<CasStore>, fetcher: Arc<ContentFetcher>) -> Self {
|
||||
Self {
|
||||
store,
|
||||
fetcher: Some(fetcher),
|
||||
manifests: RwLock::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn read(
|
||||
&self,
|
||||
file_id: FileId,
|
||||
offset: u64,
|
||||
size: u32,
|
||||
) -> Result<Bytes, ReaderError> {
|
||||
let manifest = self.get_or_fetch_manifest(file_id).await?;
|
||||
|
||||
if let Some(fetcher) = &self.fetcher {
|
||||
if let Some(meta) = fetcher.get_file_meta(file_id) {
|
||||
fetcher.emit_access_event(&meta, offset, size);
|
||||
}
|
||||
}
|
||||
|
||||
// ... rest of read logic unchanged
|
||||
}
|
||||
|
||||
async fn get_or_fetch_manifest(&self, file_id: FileId) -> Result<ChunkManifest, ReaderError> {
|
||||
{
|
||||
let manifests = self.manifests.read().unwrap();
|
||||
if let Some(m) = manifests.get(&file_id) {
|
||||
return Ok(m.clone());
|
||||
}
|
||||
}
|
||||
|
||||
let Some(fetcher) = &self.fetcher else {
|
||||
return Err(ReaderError::ManifestNotFound(file_id));
|
||||
};
|
||||
|
||||
let manifest = fetcher.ensure_cached(file_id).await
|
||||
.map_err(ReaderError::Fetch)?;
|
||||
|
||||
self.manifests.write().unwrap().insert(file_id, manifest.clone());
|
||||
Ok(manifest)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ReaderError {
|
||||
#[error("Manifest not found for file {0:?}")]
|
||||
ManifestNotFound(FileId),
|
||||
|
||||
#[error("Fetch error: {0}")]
|
||||
Fetch(#[from] FetchError),
|
||||
|
||||
#[error("CAS error: {0}")]
|
||||
Cas(#[from] crate::CasError),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Update lib.rs
|
||||
|
||||
### 3.1 Update `musicfs-cas/src/lib.rs`
|
||||
|
||||
```rust
|
||||
mod chunks;
|
||||
mod fetcher;
|
||||
mod reader;
|
||||
mod store;
|
||||
|
||||
pub use chunks::{ChunkLocation, ChunkRef};
|
||||
pub use fetcher::{ContentFetcher, FetchError};
|
||||
pub use reader::{ChunkManifest, FileReader, ReaderError};
|
||||
pub use store::{CasConfig, CasError, CasStore, DedupStats};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Update Cargo.toml
|
||||
|
||||
### 4.1 Update `musicfs-cas/Cargo.toml`
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
musicfs-core = { path = "../musicfs-core" }
|
||||
musicfs-origins = { path = "../musicfs-origins" }
|
||||
# ... rest unchanged
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Update FUSE Integration
|
||||
|
||||
### 5.1 Update `musicfs-fuse/src/filesystem.rs`
|
||||
|
||||
```rust
|
||||
use musicfs_cas::{ContentFetcher, FileReader};
|
||||
|
||||
pub struct MusicFs {
|
||||
tree: Arc<RwLock<VirtualTree>>,
|
||||
reader: Option<Arc<FileReader>>,
|
||||
fetcher: Option<Arc<ContentFetcher>>,
|
||||
uid: u32,
|
||||
gid: u32,
|
||||
}
|
||||
|
||||
impl MusicFs {
|
||||
pub fn with_content_access(
|
||||
tree: Arc<RwLock<VirtualTree>>,
|
||||
reader: Arc<FileReader>,
|
||||
fetcher: Arc<ContentFetcher>,
|
||||
) -> Self {
|
||||
Self {
|
||||
tree,
|
||||
reader: Some(reader),
|
||||
fetcher: Some(fetcher),
|
||||
uid: unsafe { libc::getuid() },
|
||||
gid: unsafe { libc::getgid() },
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
| Test | Type | Validates |
|
||||
|------|------|-----------|
|
||||
| `test_fetch_file` | Unit | Origin → CAS fetch works |
|
||||
| `test_fetch_file_not_found` | Unit | Missing file error |
|
||||
| `test_fetch_emits_event` | Unit | FileAccessed event emitted (FR-18.1) |
|
||||
| `test_reader_with_fetcher` | Unit | Cache-miss triggers fetch |
|
||||
| `test_e2e_cat_file` | Integration | `cat` returns file content |
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] `ContentFetcher` fetches from Origin and stores in CAS
|
||||
- [ ] `FileReader` calls fetcher on cache miss
|
||||
- [ ] File metadata (FileId → FileMeta) is resolvable
|
||||
- [ ] `cat /mnt/musicfs/Artist/Album/track.flac` returns actual audio data
|
||||
- [ ] All existing tests still pass
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
### `musicfs-cas/Cargo.toml`
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
musicfs-origins = { path = "../musicfs-origins" }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
1. **Week 4 treated whole files as single chunks** - this continues that approach
|
||||
2. **CDC chunking deferred to Week 5** - fetcher will be updated then
|
||||
3. **No OriginFederation yet** - single origin lookup for MVP
|
||||
4. **FileMeta registration** - caller must register files before they can be fetched
|
||||
5. **EventBus integration** - emits `FileAccessed` event per FR-18.1 (P0)
|
||||
6. **Full file fetch** - currently fetches entire file on cache miss; byte-range optimization deferred
|
||||
|
||||
## Architecture Compliance
|
||||
|
||||
| Architecture Section | Requirement | Status |
|
||||
|---------------------|-------------|--------|
|
||||
| 4.3.5 | Cache miss → fetch from origin | ✅ |
|
||||
| 4.3.5 | Store chunks by hash | ✅ |
|
||||
| 4.3.5 | Update chunk manifest | ✅ |
|
||||
| 4.3.5 | Emit FileAccessed event | ✅ |
|
||||
| 4.3.3 | OriginFederation (multi-origin) | ⏳ Deferred |
|
||||
| 4.3.5 | Byte-range fetch | ⏳ Deferred |
|
||||
| 4.3.5 | CDC chunking | ⏳ Week 5 |
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
After this, the MVP is complete:
|
||||
- Mount filesystem
|
||||
- Browse virtual tree (Artist/Album/Track)
|
||||
- Read actual file content through FUSE
|
||||
- Audio playback works
|
||||
|
||||
Week 5 adds CDC chunking for efficient delta sync.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,179 +0,0 @@
|
||||
# Week 10: Plugin System
|
||||
|
||||
**Phase**: 4 - Plugin System & Polish
|
||||
**Goal**: Extensibility via native and WASM plugins
|
||||
**Requirements**: FR-23.1-23.5, FR-24.1-24.3
|
||||
|
||||
---
|
||||
|
||||
## Deliverables
|
||||
|
||||
| Task | Crate | Files | Requirements |
|
||||
|------|-------|-------|--------------|
|
||||
| Plugin traits | musicfs-plugins | `traits.rs` | FR-23.1-23.4 |
|
||||
| Native host | musicfs-plugins | `native.rs` | FR-23.2 |
|
||||
| WASM host | musicfs-plugins | `wasm.rs` | FR-23.3 |
|
||||
| Plugin lifecycle | musicfs-plugins | `manager.rs` | FR-23.5 |
|
||||
| Example plugins | plugins/ | `example-origin/`, `example-format/` | FR-23.5 |
|
||||
|
||||
---
|
||||
|
||||
## Plugin Traits (`musicfs-plugins/src/traits.rs`)
|
||||
|
||||
```rust
|
||||
/// Base plugin interface
|
||||
pub trait Plugin: Send + Sync {
|
||||
fn name(&self) -> &str;
|
||||
fn version(&self) -> Version;
|
||||
fn init(&mut self, config: Value) -> Result<(), PluginError>;
|
||||
fn shutdown(&mut self) -> Result<(), PluginError>;
|
||||
}
|
||||
|
||||
/// Origin plugin interface (per architecture 4.3.4)
|
||||
pub trait OriginPlugin: Plugin {
|
||||
fn origin_type(&self) -> &str;
|
||||
fn create(&self, config: Value) -> Result<Box<dyn Origin>, PluginError>;
|
||||
}
|
||||
|
||||
/// Metadata source plugin
|
||||
pub trait MetadataPlugin: Plugin {
|
||||
fn lookup(&self, query: &MetadataQuery) -> Result<Option<ExternalMetadata>, PluginError>;
|
||||
}
|
||||
|
||||
/// Format plugin for custom audio formats (FR-24.1)
|
||||
pub trait FormatPlugin: Plugin {
|
||||
fn extensions(&self) -> &[&str];
|
||||
fn can_handle(&self, extension: &str) -> bool;
|
||||
fn parse(&self, reader: &mut dyn Read) -> Result<AudioMeta, PluginError>;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Native Plugin Host (`musicfs-plugins/src/native.rs`)
|
||||
|
||||
```rust
|
||||
pub struct NativePluginHost {
|
||||
plugins: HashMap<String, LoadedPlugin>,
|
||||
search_paths: Vec<PathBuf>,
|
||||
}
|
||||
|
||||
struct LoadedPlugin {
|
||||
library: libloading::Library,
|
||||
instance: Box<dyn Plugin>,
|
||||
}
|
||||
|
||||
impl NativePluginHost {
|
||||
pub fn new() -> Self;
|
||||
|
||||
/// Load plugin from shared library (.so/.dylib)
|
||||
pub fn load(&mut self, path: &Path) -> Result<PluginId, PluginError>;
|
||||
|
||||
/// Unload plugin (FR-23.5)
|
||||
pub fn unload(&mut self, id: PluginId) -> Result<(), PluginError>;
|
||||
|
||||
/// Hot reload plugin without restart (FR-23.4)
|
||||
pub fn reload(&mut self, id: PluginId) -> Result<(), PluginError>;
|
||||
|
||||
/// List loaded plugins
|
||||
pub fn list(&self) -> Vec<PluginInfo>;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## WASM Plugin Host (`musicfs-plugins/src/wasm.rs`)
|
||||
|
||||
```rust
|
||||
pub struct WasmPluginHost {
|
||||
engine: wasmtime::Engine,
|
||||
linker: wasmtime::Linker<PluginState>,
|
||||
}
|
||||
|
||||
impl WasmPluginHost {
|
||||
pub fn new() -> Result<Self, PluginError>;
|
||||
|
||||
/// Load WASM plugin with sandboxing (FR-23.3)
|
||||
pub fn load(&mut self, wasm_bytes: &[u8]) -> Result<WasmPlugin, PluginError>;
|
||||
|
||||
/// Resource limits for sandboxed execution
|
||||
pub fn set_limits(&mut self, limits: ResourceLimits);
|
||||
}
|
||||
|
||||
pub struct ResourceLimits {
|
||||
pub max_memory_mb: u32,
|
||||
pub max_cpu_time_ms: u32,
|
||||
pub allow_network: bool,
|
||||
pub allow_filesystem: bool,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Plugin Manager (`musicfs-plugins/src/manager.rs`)
|
||||
|
||||
```rust
|
||||
pub struct PluginManager {
|
||||
native_host: NativePluginHost,
|
||||
wasm_host: WasmPluginHost,
|
||||
registry: PluginRegistry,
|
||||
}
|
||||
|
||||
impl PluginManager {
|
||||
/// Initialize and load plugins from config
|
||||
pub fn init(config: &PluginConfig) -> Result<Self, PluginError>;
|
||||
|
||||
/// Get all origin plugins
|
||||
pub fn origin_plugins(&self) -> Vec<&dyn OriginPlugin>;
|
||||
|
||||
/// Get all format plugins
|
||||
pub fn format_plugins(&self) -> Vec<&dyn FormatPlugin>;
|
||||
|
||||
/// Get all metadata plugins
|
||||
pub fn metadata_plugins(&self) -> Vec<&dyn MetadataPlugin>;
|
||||
|
||||
/// Reload all plugins (hot reload)
|
||||
pub fn reload_all(&mut self) -> Result<(), PluginError>;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
| Test | Type | Validates |
|
||||
|------|------|-----------|
|
||||
| `test_native_plugin_load` | Unit | Native plugin loading (FR-23.2) |
|
||||
| `test_native_plugin_unload` | Unit | Clean unload |
|
||||
| `test_wasm_plugin_sandbox` | Unit | WASM isolation (FR-23.3) |
|
||||
| `test_wasm_resource_limits` | Unit | Memory/CPU limits enforced |
|
||||
| `test_plugin_hot_reload` | Integration | Reload without restart (FR-23.4) |
|
||||
| `test_example_origin_plugin` | Integration | Custom origin works |
|
||||
| `test_example_format_plugin` | Integration | Custom format works |
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] Native plugins loadable at runtime
|
||||
- [ ] WASM plugins sandboxed with resource limits
|
||||
- [ ] Example plugins functional
|
||||
- [ ] Plugins hot-reloadable without daemon restart
|
||||
- [ ] Plugin lifecycle management (load, unload, reload)
|
||||
|
||||
---
|
||||
|
||||
## Architecture Alignment
|
||||
|
||||
Per architecture.md section 4.3.4:
|
||||
- Plugin loading: Built-in → Native (.so) → WASM
|
||||
- Origin plugins create `Box<dyn Origin>`
|
||||
- Format plugins register file extensions
|
||||
- WASM runs in wasmtime sandbox
|
||||
|
||||
Per requirements.md:
|
||||
- FR-23.1: Loadable plugins ✓
|
||||
- FR-23.2: Stable plugin API ✓
|
||||
- FR-23.3: Plugins for origins, metadata, formats ✓
|
||||
- FR-23.4: WASM sandbox ✓
|
||||
- FR-23.5: Plugin lifecycle ✓
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user