Add benchmark suite and results
Benchmark harness (benchmarks/run_benchmarks.py): - Mount time, readdir, stat latency, file open/read, memory usage - ENOENT lookup (missing file) benchmark per Oracle review - Uses synthetic FLAC files from test infrastructure Results: ALL BENCHMARKS BLOCKED BY BUGS - Bug #2 (directory tree building) crashes mount with any content - FSNode.adddir() assumes parent dirs exist, fails with KeyError - Bug #1 (nested methods) would block FUSE ops even if mount worked beetfs is non-functional for real-world use until both bugs fixed.
This commit is contained in:
Alexander
@@ -0,0 +1,75 @@
|
||||
{
|
||||
"timestamp": "2026-05-12T14:42:37.343765",
|
||||
"results": [
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "mount_time",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "readdir",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "stat_latency",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "enoent_lookup",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "file_open",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "read_throughput",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
},
|
||||
{
|
||||
"mean_ms": null,
|
||||
"runs": 0,
|
||||
"name": "memory_usage",
|
||||
"memory_kb": null,
|
||||
"error": "Mount process died: Traceback (most recent call last):\n File \"/tmp/nix-shell.VlFHpy/nix-shell.rhvctI/beetfs_bench_JfIl36/mount.py\", line 40, in <module>\n beetFs.directory_structure.adddir(sub_elements, level_subbed[level])\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 414, in adddir\n node = self.getnode(elements, root=root)\n File \"/home/fujin/Code/agregators/music-agregator/beetfs/beetsplug/beetFs.py\", line 403, in getnode\n return self.getnode(elements, root=root.dirs[topdir])\nKeyError: u'Bench Artist'\n",
|
||||
"min_ms": null,
|
||||
"metadata": {},
|
||||
"max_ms": null
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,624 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
beetfs Benchmark Suite
|
||||
Measures mount time, metadata ops, file I/O, and memory usage.
|
||||
"""
|
||||
from __future__ import print_function
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import json
|
||||
import tempfile
|
||||
import shutil
|
||||
import subprocess
|
||||
import signal
|
||||
import resource
|
||||
import datetime
|
||||
|
||||
# Add project paths
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'beetsplug'))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), 'tests'))
|
||||
|
||||
from conftest import create_synthetic_flac
|
||||
|
||||
|
||||
class BenchmarkResult(object):
|
||||
"""Stores benchmark results."""
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
self.timings = []
|
||||
self.memory_kb = None
|
||||
self.error = None
|
||||
self.metadata = {}
|
||||
|
||||
def add_timing(self, seconds):
|
||||
self.timings.append(seconds)
|
||||
|
||||
@property
|
||||
def mean(self):
|
||||
if not self.timings:
|
||||
return None
|
||||
return sum(self.timings) / len(self.timings)
|
||||
|
||||
@property
|
||||
def min_time(self):
|
||||
return min(self.timings) if self.timings else None
|
||||
|
||||
@property
|
||||
def max_time(self):
|
||||
return max(self.timings) if self.timings else None
|
||||
|
||||
def to_dict(self):
|
||||
return {
|
||||
'name': self.name,
|
||||
'mean_ms': self.mean * 1000 if self.mean else None,
|
||||
'min_ms': self.min_time * 1000 if self.min_time else None,
|
||||
'max_ms': self.max_time * 1000 if self.max_time else None,
|
||||
'runs': len(self.timings),
|
||||
'memory_kb': self.memory_kb,
|
||||
'error': self.error,
|
||||
'metadata': self.metadata
|
||||
}
|
||||
|
||||
|
||||
class BeetFSBenchmark(object):
|
||||
"""Benchmark harness for beetfs."""
|
||||
|
||||
def __init__(self, output_dir):
|
||||
self.output_dir = output_dir
|
||||
self.results = []
|
||||
self.temp_dir = None
|
||||
self.mount_dir = None
|
||||
self.music_dir = None
|
||||
self.db_path = None
|
||||
self.mount_process = None
|
||||
|
||||
def setup(self, num_tracks=10, track_size_mb=5):
|
||||
"""Create test environment with synthetic tracks."""
|
||||
self.temp_dir = tempfile.mkdtemp(prefix='beetfs_bench_')
|
||||
self.mount_dir = os.path.join(self.temp_dir, 'mount')
|
||||
self.music_dir = os.path.join(self.temp_dir, 'music')
|
||||
self.db_path = os.path.join(self.temp_dir, 'library.db')
|
||||
self.config_dir = os.path.join(self.temp_dir, 'config')
|
||||
|
||||
os.makedirs(self.mount_dir)
|
||||
os.makedirs(self.music_dir)
|
||||
os.makedirs(self.config_dir)
|
||||
|
||||
# Create beets config
|
||||
config_path = os.path.join(self.config_dir, 'config.yaml')
|
||||
with open(config_path, 'w') as f:
|
||||
f.write('directory: {}\n'.format(self.music_dir))
|
||||
f.write('library: {}\n'.format(self.db_path))
|
||||
f.write('plugins: []\n')
|
||||
|
||||
os.environ['BEETSDIR'] = self.config_dir
|
||||
|
||||
# Create synthetic FLAC files
|
||||
print("Creating {} synthetic tracks ({} MB each)...".format(num_tracks, track_size_mb))
|
||||
track_paths = []
|
||||
for i in range(num_tracks):
|
||||
artist = 'Bench Artist'
|
||||
album = 'Bench Album'
|
||||
title = 'Track {:03d}'.format(i + 1)
|
||||
filename = '{:02d} - {} - {}.flac'.format(i + 1, artist, title)
|
||||
track_path = os.path.join(self.music_dir, artist, album, filename)
|
||||
self._makedirs(os.path.dirname(track_path))
|
||||
create_synthetic_flac(track_path, duration_sec=track_size_mb * 10,
|
||||
artist=artist, title=title, album=album, track=str(i + 1))
|
||||
track_paths.append(track_path)
|
||||
|
||||
# Import into beets library
|
||||
print("Importing tracks into beets library...")
|
||||
from beets import config
|
||||
from beets.library import Library
|
||||
config.read(user=False)
|
||||
config['directory'].set(self.music_dir)
|
||||
config['library'].set(self.db_path)
|
||||
|
||||
lib = Library(self.db_path)
|
||||
from beets.library import Item
|
||||
for i, path in enumerate(track_paths):
|
||||
item = Item(
|
||||
path=path,
|
||||
artist=u'Bench Artist',
|
||||
album=u'Bench Album',
|
||||
title=u'Track {:03d}'.format(i + 1),
|
||||
track=i + 1,
|
||||
year=2024,
|
||||
genre=u'Benchmark',
|
||||
format='flac'
|
||||
)
|
||||
lib.add(item)
|
||||
lib._close()
|
||||
|
||||
return len(track_paths)
|
||||
|
||||
def _makedirs(self, path):
|
||||
"""Python 2 compatible makedirs."""
|
||||
if not os.path.exists(path):
|
||||
os.makedirs(path)
|
||||
|
||||
def teardown(self):
|
||||
"""Clean up test environment."""
|
||||
self.unmount()
|
||||
if self.temp_dir and os.path.exists(self.temp_dir):
|
||||
shutil.rmtree(self.temp_dir, ignore_errors=True)
|
||||
|
||||
def mount(self):
|
||||
"""Mount beetfs and return time taken."""
|
||||
# Create mount script
|
||||
beetfs_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
beetsplug = os.path.join(beetfs_root, 'beetsplug')
|
||||
|
||||
mount_script = '''
|
||||
import sys
|
||||
sys.path.insert(0, "{beetfs_root}")
|
||||
sys.path.insert(0, "{beetsplug}")
|
||||
|
||||
import os
|
||||
import re
|
||||
os.environ["BEETSDIR"] = "{config_dir}"
|
||||
|
||||
from beets import config
|
||||
from beets.library import Library
|
||||
|
||||
config.read(user=False)
|
||||
config["directory"] = "{music_dir}"
|
||||
config["library"] = "{db_path}"
|
||||
|
||||
lib = Library("{db_path}")
|
||||
|
||||
import beetFs
|
||||
import fuse
|
||||
|
||||
fuse.fuse_python_api = (0, 2)
|
||||
|
||||
beetFs.library = lib
|
||||
beetFs.structure_depth = 4
|
||||
beetFs.structure_split = [0, 1, 2, 3]
|
||||
beetFs.directory_structure = beetFs.FSNode({{}}, {{}})
|
||||
|
||||
for item in lib.items():
|
||||
mapping = beetFs.template_mapping(lib, item)
|
||||
path_str = beetFs.PATH_FORMAT
|
||||
for key, val in mapping.items():
|
||||
if val is not None:
|
||||
clean_val = re.sub(r"[\\\\/:]|^\\.", "_", unicode(val))
|
||||
path_str = path_str.replace("$" + key, clean_val)
|
||||
elements = path_str.split("/")
|
||||
sub_elements = elements[0:beetFs.structure_depth-1]
|
||||
for level in range(len(sub_elements)):
|
||||
level_subbed = sub_elements[0:level+1]
|
||||
beetFs.directory_structure.adddir(sub_elements, level_subbed[level])
|
||||
beetFs.directory_structure.addfile(
|
||||
sub_elements,
|
||||
elements[beetFs.structure_depth-1],
|
||||
item.id
|
||||
)
|
||||
|
||||
fs = beetFs.beetFileSystem(
|
||||
version="%prog " + fuse.__version__,
|
||||
usage="beetfs benchmark",
|
||||
dash_s_do="setsingle"
|
||||
)
|
||||
|
||||
fs.parser.add_option(mountopt="root", metavar="PATH", default="{music_dir}",
|
||||
help="music library root path")
|
||||
fs.parse(args=["{mount_dir}"], errex=1)
|
||||
fs.flags = 0
|
||||
fs.multithreaded = False
|
||||
fs.fuse_args.setmod("foreground")
|
||||
fs.fuse_args.add("fsname=beetfs")
|
||||
fs.fuse_args.add("nonempty")
|
||||
fs.lib = lib
|
||||
|
||||
fs.main()
|
||||
'''.format(
|
||||
beetfs_root=beetfs_root,
|
||||
beetsplug=beetsplug,
|
||||
config_dir=self.config_dir,
|
||||
music_dir=self.music_dir,
|
||||
db_path=self.db_path,
|
||||
mount_dir=self.mount_dir
|
||||
)
|
||||
|
||||
script_path = os.path.join(self.temp_dir, 'mount.py')
|
||||
with open(script_path, 'w') as f:
|
||||
f.write(mount_script)
|
||||
|
||||
start_time = time.time()
|
||||
self.mount_process = subprocess.Popen(
|
||||
[sys.executable, script_path],
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE
|
||||
)
|
||||
|
||||
# Wait for mount
|
||||
timeout = 30
|
||||
poll_interval = 0.05
|
||||
elapsed = 0
|
||||
while elapsed < timeout:
|
||||
if os.path.ismount(self.mount_dir):
|
||||
mount_time = time.time() - start_time
|
||||
return mount_time
|
||||
time.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
# Check if process died
|
||||
if self.mount_process.poll() is not None:
|
||||
stdout, stderr = self.mount_process.communicate()
|
||||
raise RuntimeError("Mount process died: {}".format(stderr.decode('utf-8', errors='replace')))
|
||||
|
||||
raise RuntimeError("Mount timeout after {} seconds".format(timeout))
|
||||
|
||||
def unmount(self):
|
||||
"""Unmount beetfs."""
|
||||
if os.path.ismount(self.mount_dir):
|
||||
subprocess.call(['fusermount', '-u', self.mount_dir])
|
||||
time.sleep(0.5)
|
||||
|
||||
if self.mount_process and self.mount_process.poll() is None:
|
||||
self.mount_process.terminate()
|
||||
try:
|
||||
self.mount_process.wait(timeout=5)
|
||||
except:
|
||||
self.mount_process.kill()
|
||||
|
||||
def get_memory_usage(self):
|
||||
"""Get current process memory usage in KB."""
|
||||
if self.mount_process and self.mount_process.poll() is None:
|
||||
try:
|
||||
with open('/proc/{}/status'.format(self.mount_process.pid)) as f:
|
||||
for line in f:
|
||||
if line.startswith('VmRSS:'):
|
||||
return int(line.split()[1])
|
||||
except:
|
||||
pass
|
||||
return None
|
||||
|
||||
# ========================
|
||||
# BENCHMARK METHODS
|
||||
# ========================
|
||||
|
||||
def bench_mount_time(self, runs=5):
|
||||
"""Benchmark mount time."""
|
||||
result = BenchmarkResult('mount_time')
|
||||
print("\n=== Mount Time Benchmark ({} runs) ===".format(runs))
|
||||
|
||||
for i in range(runs):
|
||||
try:
|
||||
mount_time = self.mount()
|
||||
result.add_timing(mount_time)
|
||||
print(" Run {}: {:.3f}s".format(i + 1, mount_time))
|
||||
result.memory_kb = self.get_memory_usage()
|
||||
self.unmount()
|
||||
time.sleep(0.5)
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" Run {}: ERROR - {}".format(i + 1, e))
|
||||
break
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_stat_latency(self, runs=50):
|
||||
"""Benchmark single stat() call latency."""
|
||||
result = BenchmarkResult('stat_latency')
|
||||
print("\n=== Stat Latency Benchmark ({} runs) ===".format(runs))
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(1) # Let mount settle
|
||||
|
||||
# Find a file to stat
|
||||
test_path = None
|
||||
for root, dirs, files in os.walk(self.mount_dir):
|
||||
if files:
|
||||
test_path = os.path.join(root, files[0])
|
||||
break
|
||||
|
||||
if not test_path:
|
||||
result.error = "No files found in mount"
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
result.metadata['test_path'] = test_path
|
||||
|
||||
for i in range(runs):
|
||||
start = time.time()
|
||||
try:
|
||||
os.stat(test_path)
|
||||
elapsed = time.time() - start
|
||||
result.add_timing(elapsed)
|
||||
except OSError as e:
|
||||
result.error = "stat failed: {} (errno {})".format(e.strerror, e.errno)
|
||||
print(" ERROR: {}".format(result.error))
|
||||
break
|
||||
|
||||
if result.timings:
|
||||
print(" Mean: {:.3f}ms, Min: {:.3f}ms, Max: {:.3f}ms".format(
|
||||
result.mean * 1000, result.min_time * 1000, result.max_time * 1000))
|
||||
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_readdir(self, runs=20):
|
||||
"""Benchmark directory listing."""
|
||||
result = BenchmarkResult('readdir')
|
||||
print("\n=== Readdir Benchmark ({} runs) ===".format(runs))
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(1)
|
||||
|
||||
for i in range(runs):
|
||||
start = time.time()
|
||||
try:
|
||||
entries = os.listdir(self.mount_dir)
|
||||
elapsed = time.time() - start
|
||||
result.add_timing(elapsed)
|
||||
if i == 0:
|
||||
result.metadata['entry_count'] = len(entries)
|
||||
except OSError as e:
|
||||
result.error = "listdir failed: {} (errno {})".format(e.strerror, e.errno)
|
||||
print(" ERROR: {}".format(result.error))
|
||||
break
|
||||
|
||||
if result.timings:
|
||||
print(" Mean: {:.3f}ms, Entries: {}".format(
|
||||
result.mean * 1000, result.metadata.get('entry_count', 'N/A')))
|
||||
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_file_open(self, runs=10):
|
||||
"""Benchmark file open latency."""
|
||||
result = BenchmarkResult('file_open')
|
||||
print("\n=== File Open Benchmark ({} runs) ===".format(runs))
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(1)
|
||||
|
||||
# Find a file to open
|
||||
test_path = None
|
||||
for root, dirs, files in os.walk(self.mount_dir):
|
||||
if files:
|
||||
test_path = os.path.join(root, files[0])
|
||||
break
|
||||
|
||||
if not test_path:
|
||||
result.error = "No files found in mount"
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
result.metadata['test_path'] = test_path
|
||||
|
||||
for i in range(runs):
|
||||
# Clear page cache between runs (requires sudo, skip if not available)
|
||||
try:
|
||||
subprocess.call(['sync'])
|
||||
except:
|
||||
pass
|
||||
|
||||
start = time.time()
|
||||
try:
|
||||
f = open(test_path, 'rb')
|
||||
f.read(1) # Trigger actual open
|
||||
f.close()
|
||||
elapsed = time.time() - start
|
||||
result.add_timing(elapsed)
|
||||
except (IOError, OSError) as e:
|
||||
result.error = "open failed: {}".format(e)
|
||||
print(" ERROR: {}".format(result.error))
|
||||
break
|
||||
|
||||
if result.timings:
|
||||
print(" Mean: {:.3f}ms, Min: {:.3f}ms, Max: {:.3f}ms".format(
|
||||
result.mean * 1000, result.min_time * 1000, result.max_time * 1000))
|
||||
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_read_throughput(self):
|
||||
"""Benchmark read throughput."""
|
||||
result = BenchmarkResult('read_throughput')
|
||||
print("\n=== Read Throughput Benchmark ===")
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(1)
|
||||
|
||||
# Find a file to read
|
||||
test_path = None
|
||||
for root, dirs, files in os.walk(self.mount_dir):
|
||||
if files:
|
||||
test_path = os.path.join(root, files[0])
|
||||
break
|
||||
|
||||
if not test_path:
|
||||
result.error = "No files found in mount"
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
result.metadata['test_path'] = test_path
|
||||
|
||||
# Read entire file and measure throughput
|
||||
start = time.time()
|
||||
try:
|
||||
with open(test_path, 'rb') as f:
|
||||
data = f.read()
|
||||
elapsed = time.time() - start
|
||||
|
||||
file_size = len(data)
|
||||
throughput_mbps = (file_size / (1024 * 1024)) / elapsed if elapsed > 0 else 0
|
||||
|
||||
result.add_timing(elapsed)
|
||||
result.metadata['file_size_bytes'] = file_size
|
||||
result.metadata['throughput_mbps'] = throughput_mbps
|
||||
|
||||
print(" File size: {:.2f} MB, Time: {:.3f}s, Throughput: {:.2f} MB/s".format(
|
||||
file_size / (1024 * 1024), elapsed, throughput_mbps))
|
||||
|
||||
except (IOError, OSError) as e:
|
||||
result.error = "read failed: {}".format(e)
|
||||
print(" ERROR: {}".format(result.error))
|
||||
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_memory_usage(self):
|
||||
"""Benchmark memory usage."""
|
||||
result = BenchmarkResult('memory_usage')
|
||||
print("\n=== Memory Usage Benchmark ===")
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(2)
|
||||
|
||||
# Measure idle memory
|
||||
idle_mem = self.get_memory_usage()
|
||||
result.metadata['idle_memory_kb'] = idle_mem
|
||||
print(" Idle memory: {} KB".format(idle_mem))
|
||||
|
||||
# Open a file and measure
|
||||
test_path = None
|
||||
for root, dirs, files in os.walk(self.mount_dir):
|
||||
if files:
|
||||
test_path = os.path.join(root, files[0])
|
||||
break
|
||||
|
||||
if test_path:
|
||||
try:
|
||||
with open(test_path, 'rb') as f:
|
||||
f.read()
|
||||
after_read_mem = self.get_memory_usage()
|
||||
result.metadata['after_read_memory_kb'] = after_read_mem
|
||||
print(" After file read: {} KB".format(after_read_mem))
|
||||
if idle_mem and after_read_mem:
|
||||
print(" Memory increase: {} KB".format(after_read_mem - idle_mem))
|
||||
except (IOError, OSError) as e:
|
||||
result.error = "read failed: {}".format(e)
|
||||
|
||||
result.memory_kb = self.get_memory_usage()
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def bench_enoent_lookup(self, runs=50):
|
||||
"""Benchmark ENOENT lookup (missing file) latency."""
|
||||
result = BenchmarkResult('enoent_lookup')
|
||||
print("\n=== ENOENT Lookup Benchmark ({} runs) ===".format(runs))
|
||||
|
||||
try:
|
||||
self.mount()
|
||||
time.sleep(1)
|
||||
|
||||
# Non-existent file path
|
||||
missing_path = os.path.join(self.mount_dir, 'nonexistent', 'cover.jpg')
|
||||
|
||||
for i in range(runs):
|
||||
start = time.time()
|
||||
try:
|
||||
os.stat(missing_path)
|
||||
except OSError:
|
||||
pass # Expected
|
||||
elapsed = time.time() - start
|
||||
result.add_timing(elapsed)
|
||||
|
||||
if result.timings:
|
||||
print(" Mean: {:.3f}ms, Min: {:.3f}ms, Max: {:.3f}ms".format(
|
||||
result.mean * 1000, result.min_time * 1000, result.max_time * 1000))
|
||||
|
||||
self.unmount()
|
||||
except Exception as e:
|
||||
result.error = str(e)
|
||||
print(" ERROR: {}".format(e))
|
||||
|
||||
self.results.append(result)
|
||||
return result
|
||||
|
||||
def save_results(self, filename='benchmark_results.json'):
|
||||
"""Save results to JSON file."""
|
||||
output_path = os.path.join(self.output_dir, filename)
|
||||
data = {
|
||||
'timestamp': datetime.datetime.now().isoformat(),
|
||||
'results': [r.to_dict() for r in self.results]
|
||||
}
|
||||
with open(output_path, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
print("\nResults saved to: {}".format(output_path))
|
||||
return output_path
|
||||
|
||||
|
||||
def main():
|
||||
print("=" * 60)
|
||||
print("beetfs Benchmark Suite")
|
||||
print("=" * 60)
|
||||
|
||||
output_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'results')
|
||||
if not os.path.exists(output_dir):
|
||||
os.makedirs(output_dir)
|
||||
|
||||
bench = BeetFSBenchmark(output_dir)
|
||||
|
||||
try:
|
||||
# Setup with 10 tracks, 5MB each
|
||||
num_tracks = bench.setup(num_tracks=10, track_size_mb=5)
|
||||
print("Setup complete: {} tracks".format(num_tracks))
|
||||
|
||||
# Run benchmarks
|
||||
bench.bench_mount_time(runs=3)
|
||||
bench.bench_readdir(runs=10)
|
||||
bench.bench_stat_latency(runs=20)
|
||||
bench.bench_enoent_lookup(runs=20)
|
||||
bench.bench_file_open(runs=5)
|
||||
bench.bench_read_throughput()
|
||||
bench.bench_memory_usage()
|
||||
|
||||
# Save results
|
||||
bench.save_results()
|
||||
|
||||
finally:
|
||||
bench.teardown()
|
||||
|
||||
# Print summary
|
||||
print("\n" + "=" * 60)
|
||||
print("SUMMARY")
|
||||
print("=" * 60)
|
||||
for r in bench.results:
|
||||
status = "OK" if not r.error else "FAIL"
|
||||
mean_str = "{:.3f}ms".format(r.mean * 1000) if r.mean else "N/A"
|
||||
print("{:20} {:6} Mean: {:>12} Error: {}".format(
|
||||
r.name, status, mean_str, r.error or "None"))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
# beetfs Benchmark Results
|
||||
|
||||
**Date**: 2026-05-12
|
||||
**Status**: ❌ ALL BENCHMARKS BLOCKED BY BUGS
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Benchmarks cannot complete due to critical bugs in beetfs. The implementation is non-functional for any library with content.
|
||||
|
||||
## Results
|
||||
|
||||
| Benchmark | Status | Mean | Error |
|
||||
|-----------|--------|------|-------|
|
||||
| mount_time | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| readdir | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| stat_latency | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| enoent_lookup | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| file_open | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| read_throughput | ❌ FAIL | N/A | Directory tree building bug |
|
||||
| memory_usage | ❌ FAIL | N/A | Directory tree building bug |
|
||||
|
||||
## Blocking Bugs
|
||||
|
||||
### Bug #1: Nested Methods (Lines 758-1144)
|
||||
|
||||
All FUSE operations (`readdir`, `open`, `read`, `write`, etc.) are indented inside the `access()` method, making them local functions instead of class methods.
|
||||
|
||||
**Impact**: Even if mount succeeds, all file operations return `ENOSYS (Function not implemented)`.
|
||||
|
||||
**Fix Required**: Dedent lines 758-1144 by 8 spaces.
|
||||
|
||||
### Bug #2: Directory Tree Building (Lines 403-414)
|
||||
|
||||
`FSNode.adddir()` calls `getnode()` which assumes parent directories already exist. When building the tree for a new library, parent directories haven't been created yet.
|
||||
|
||||
**Error**:
|
||||
```
|
||||
KeyError: u'Bench Artist'
|
||||
File "beetFs.py", line 403, in getnode
|
||||
return self.getnode(elements, root=root.dirs[topdir])
|
||||
```
|
||||
|
||||
**Impact**: Mount crashes when library contains any tracks.
|
||||
|
||||
**Fix Required**: `adddir()` must create parent directories recursively before adding child.
|
||||
|
||||
### Bug #3: Empty Library Only
|
||||
|
||||
The only working configuration is mounting with an empty beets library:
|
||||
- `test_mount_empty_library`: ✅ PASS
|
||||
- Any library with tracks: ❌ CRASH
|
||||
|
||||
## Test Environment
|
||||
|
||||
- **Python**: 2.7.15
|
||||
- **OS**: Linux (NixOS)
|
||||
- **Test data**: 10 synthetic FLAC files (5 MB each)
|
||||
- **Beets**: 1.4.9
|
||||
|
||||
## Benchmark Configuration
|
||||
|
||||
```python
|
||||
num_tracks = 10
|
||||
track_size_mb = 5
|
||||
mount_runs = 3
|
||||
stat_runs = 20
|
||||
readdir_runs = 10
|
||||
```
|
||||
|
||||
## Raw Results
|
||||
|
||||
See `benchmarks/results/benchmark_results.json` for full JSON output.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Fix Bug #2** (directory tree building) - allows mount with content
|
||||
2. **Fix Bug #1** (nested methods) - allows FUSE operations to work
|
||||
3. **Re-run benchmarks** - get actual performance numbers
|
||||
|
||||
## Conclusion
|
||||
|
||||
**beetfs is currently non-functional** for real-world use. Both bugs must be fixed before performance can be measured. The test infrastructure and benchmark suite are ready; only the implementation needs repair.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: E2E Test Results (For Reference)
|
||||
|
||||
From the e2e test suite (74 tests):
|
||||
|
||||
| Category | Passed | Failed | Errors |
|
||||
|----------|--------|--------|--------|
|
||||
| Smoke tests | 4 | 3 | 0 |
|
||||
| Nested bug detection | 3 (confirmed bug) | 10 | 0 |
|
||||
| Readdir | 0 | 10 | 0 |
|
||||
| Stat | 0 | 8 | 0 |
|
||||
| Read | 0 | 11 | 0 |
|
||||
| Write | 0 | 7 | 0 |
|
||||
| Error handling | 0 | 7 | 3 |
|
||||
| **Total** | **12** | **56** | **3** |
|
||||
|
||||
The 12 passing tests are infrastructure tests and tests that verify the bugs exist.
|
||||
Reference in New Issue
Block a user