Implements all 6 critical resilience fixes from phase-a-stop-dying.md:
- Issue 2.9: Migrate std::sync::RwLock → parking_lot::RwLock (7 files)
Prevents lock poisoning cascade on writer panic
- Issue 2.2: Add install_panic_hook() to log panics via tracing
Ensures panics are captured in logs/journald before process death
- Issue 3.7: Add ExecStopPost to systemd service
Cleans up stale FUSE mounts on service stop
- Issue 2.7: Add check_stale_mount() detection on startup
Auto-cleans leftover mounts from previous crashes
- Issue 2.10: Integrate sd_notify for systemd lifecycle
Sends READY=1 after mount, STOPPING on shutdown
- Issue 2.1: Add signal handling with spawn_mount
Catches SIGTERM/SIGINT for clean shutdown instead of instant death
All 7 Phase A tests pass:
- test_poisoned_tree_lock_returns_eio_not_panic
- test_parking_lot_rwlock_survives_panic
- test_panic_hook_logs_to_tracing
- test_systemd_service_has_execstoppost
- test_stale_mount_check_function_exists
- test_sd_notify_ready_sent
- test_sigterm_triggers_shutdown
Phase 1 of resilience testing design doc implementation:
- New musicfs-test-utils crate with FaultyOrigin, FaultyCasStore, fixtures
- Failpoints instrumented in musicfs-cas/store.rs
- 16 resilience tests (13 RED for missing features, 3 GREEN for existing)
- 3 Docker/Toxiproxy network tests (RED until docker-compose up)
- docker-compose.yml for Toxiproxy + MinIO + SFTP test infrastructure
Tests properly fail-first (TDD): check_all() sequential, no health timeout,
missing corruption detection, no passthrough mode, etc.
Alexander