fujin/metadata-agregator
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial implementation of metadata aggregator

Browse Source 
- gRPC service with MusicBrainz provider
- PostgreSQL schema with migrations
- Service layer with database-first caching
- Repository pattern for data access
- YAML configuration support
- Research documentation for 17 music metadata projects
This commit is contained in:
Alexander
2026-04-28 16:27:14 +02:00
commit a1f6701bac
163 changed files with 95884 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/minim/analysis/DEPLOYMENT.md
+703
View File
@@ -0,0 +1,703 @@
# minim: Deployment
## Deployment Model
minim is a **Python library**, not a deployable service. There is no server, no daemon, no container to deploy. Users install the library and import it into their own Python code.
**Installation Target:** Developer workstations, scripts, Jupyter notebooks, personal automation tools.
**Not Applicable:** Production web servers, cloud deployments, containerized services, serverless functions.
## Installation Methods
### From Source (Current)
**Clone Repository:**
```bash
git clone https://github.com/bbye98/minim.git
cd minim
```
**Install in Development Mode:**
```bash
python -m pip install -e .
```
**Install in Production Mode:**
```bash
python -m pip install .
```
**Editable Install (`-e`):**
- Changes to source code immediately reflected without reinstalling
- Useful for development and testing
- Creates symlink to source directory
**Production Install:**
- Copies files to site-packages
- Requires reinstall after code changes
- Cleaner for end users
### Via Conda
**Environment File:**
```yaml
# environment.yml
name: minim
channels:
- conda-forge
- defaults
dependencies:
- python>=3.9
- cryptography
- mutagen
- requests
- pip
- pip:
- -e .
```
**Create Environment:**
```bash
conda env create -f environment.yml
conda activate minim
```
**Update Environment:**
```bash
conda env update -f environment.yml
```
### Via PyPI (Planned for v2)
**Not Yet Available.** minim is not published to PyPI as of v1.1.0.
**Planned for v2:**
```bash
pip install minim
```
**Package Metadata (setup.py):**
```python
from setuptools import setup, find_packages
setup(
name="minim",
version="1.1.0",
author="Benjamin Ye",
author_email="bbye98@gmail.com",
description="Comprehensive music metadata library",
long_description=open("README.md").read(),
long_description_content_type="text/markdown",
url="https://github.com/bbye98/minim",
packages=find_packages(),
classifiers=[
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
],
python_requires=">=3.9",
install_requires=[
"cryptography",
"mutagen",
"requests",
],
extras_require={
"full": [
"ffmpeg-python",
"flask",
"levenshtein",
"numpy",
"pillow",
"playwright",
],
},
)
```
## Dependencies
### Required (Core)
**cryptography:**
- Purpose: TIDAL manifest decryption, secure token handling
- Version: Not pinned (latest compatible)
- Install: `pip install cryptography`
**mutagen:**
- Purpose: Audio file metadata reading/writing
- Version: Not pinned
- Install: `pip install mutagen`
**requests:**
- Purpose: HTTP client for all API calls
- Version: Not pinned
- Install: `pip install requests`
### Optional (Features)
**ffmpeg:**
- Purpose: Audio format conversion
- Type: System binary (not Python package)
- Install: `apt install ffmpeg` (Ubuntu), `brew install ffmpeg` (macOS), download from ffmpeg.org (Windows)
- Detection: `shutil.which("ffmpeg")`
**flask:**
- Purpose: OAuth callback server (alternative to http.server)
- Install: `pip install flask`
**levenshtein:**
- Purpose: Fuzzy string matching for search results
- Install: `pip install levenshtein`
**numpy:**
- Purpose: Audio analysis features
- Install: `pip install numpy`
**pillow:**
- Purpose: Image processing for album artwork
- Install: `pip install pillow`
**playwright:**
- Purpose: Browser automation for OAuth flows
- Install: `pip install playwright && playwright install chromium`
### Dependency Management
**No Lock File:** minim does not use `requirements.txt` or `Pipfile.lock` for version pinning.
**Version Constraints:** None specified. Uses latest compatible versions.
**Risk:** Dependency updates may introduce breaking changes.
**Recommendation for Production:**
```bash
# Generate lock file
pip freeze > requirements.txt
# Install from lock file
pip install -r requirements.txt
```
## System Requirements
### Python Version
**Minimum:** Python 3.9
**Tested:** Python 3.9, 3.10, 3.11
**Recommended:** Python 3.11 (latest stable)
**Version-Specific Features:**
- Type hints (PEP 585): `list[str]` instead of `List[str]` (requires 3.9+)
- Union operator: `str | None` instead of `Optional[str]` (requires 3.10+, not used in v1)
### Operating Systems
**Supported:**
- Linux (Ubuntu 20.04+, Debian 11+, Fedora 35+, Arch)
- macOS (10.15 Catalina+)
- Windows (10, 11)
**Tested in CI:** Ubuntu 22.04 (GitHub Actions)
**Platform-Specific Considerations:**
**Linux:**
- FFmpeg available via package manager (`apt`, `dnf`, `pacman`)
- Config file at `~/.minim.cfg` or `/home/username/minim.cfg`
**macOS:**
- FFmpeg via Homebrew (`brew install ffmpeg`)
- Config file at `~/minim.cfg` or `/Users/username/minim.cfg`
**Windows:**
- FFmpeg requires manual download and PATH configuration
- Config file at `C:\Users\username\minim.cfg`
- Path handling uses `os.path.expanduser("~")` (cross-platform)
### External Dependencies
**FFmpeg (Optional):**
- Required for audio format conversion
- Not required for metadata reading/writing or API access
- Version: 4.0+ recommended
**Browser (Optional):**
- Required for OAuth flows using Playwright
- Chromium installed via `playwright install chromium`
- Not required for http.server or Flask callback methods
## Configuration
### Environment Variables
minim checks environment variables for credentials:
**Discogs:**
- `DISCOGS_CONSUMER_KEY`
- `DISCOGS_CONSUMER_SECRET`
- `DISCOGS_ACCESS_TOKEN`
- `DISCOGS_ACCESS_TOKEN_SECRET`
- `DISCOGS_PERSONAL_ACCESS_TOKEN`
**Qobuz:**
- `QOBUZ_APP_ID`
- `QOBUZ_APP_SECRET`
- `QOBUZ_EMAIL`
- `QOBUZ_PASSWORD`
**Spotify:**
- `SPOTIFY_CLIENT_ID`
- `SPOTIFY_CLIENT_SECRET`
- `SPOTIFY_REDIRECT_URI`
**TIDAL:**
- `TIDAL_CLIENT_ID`
- `TIDAL_CLIENT_SECRET`
- `TIDAL_REDIRECT_URI`
**Precedence:** Environment variables > config file > constructor parameters
**Use Case:** CI/CD pipelines, containerized environments, shared systems
**Example:**
```bash
export SPOTIFY_CLIENT_ID="abc123"
export SPOTIFY_CLIENT_SECRET="def456"
python script.py # Automatically uses environment variables
```
### Config File
**Location:** `~/minim.cfg`
**Format:** INI-style (ConfigParser)
**Auto-Creation:** Created automatically when tokens are saved via `set_access_token()`
**Manual Creation:**
```ini
[spotify]
client_id = abc123
client_secret = def456
access_token = BQC...
refresh_token = AQD...
expires_at = 1672531200
```
**Permissions:** Default (0644 on Unix). Recommendation: `chmod 600 ~/minim.cfg` for security.
## CI/CD
### GitHub Actions
**Workflow File:** `.github/workflows/ci.yml`
```yaml
name: CI
on:
push:
branches: [main, dev]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.9'
- name: Install FFmpeg
run: sudo apt-get update && sudo apt-get install -y ffmpeg
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e .
pip install pytest ruff
- name: Lint with ruff
run: ruff check .
- name: Run tests
env:
SPOTIFY_CLIENT_ID: ${{ secrets.SPOTIFY_CLIENT_ID }}
SPOTIFY_CLIENT_SECRET: ${{ secrets.SPOTIFY_CLIENT_SECRET }}
TIDAL_CLIENT_ID: ${{ secrets.TIDAL_CLIENT_ID }}
TIDAL_CLIENT_SECRET: ${{ secrets.TIDAL_CLIENT_SECRET }}
run: pytest tests/
```
**Secrets Management:**
- API credentials stored in GitHub Secrets
- Accessed via `${{ secrets.SECRET_NAME }}`
- Not exposed in logs
**Test Execution:**
- Real API calls (not mocked)
- Requires valid credentials
- May fail if rate limits exceeded or services change APIs
**Linting:**
- `ruff`: Fast Python linter (replaces flake8, pylint)
- Configuration in `pyproject.toml` or `ruff.toml`
### Coverage
**Tool:** `coverage.py`
**Configuration:** `.coveragerc`
```ini
[run]
source = minim
omit =
*/tests/*
*/__init__.py
*/site-packages/*
[report]
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if __name__ == .__main__.:
```
**Execution:**
```bash
coverage run -m pytest tests/
coverage report
coverage html # Generate HTML report
```
**Current Coverage:** Not documented in repository. Likely 60-80% based on test file count.
## Documentation
### ReadTheDocs
**URL:** https://minim.readthedocs.io
**Build System:** Sphinx
**Configuration:** `docs/conf.py`
```python
project = 'minim'
author = 'Benjamin Ye'
release = '1.1.0'
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.napoleon',
'sphinx.ext.viewcode',
]
html_theme = 'sphinx_rtd_theme'
```
**Auto-Deploy:**
- Triggered on push to `main` branch
- Builds from `docs/` directory
- Parses docstrings from source code
**Docstring Format:** Google-style
```python
def search(self, query: str, types: list[str] = ["track"]) -> dict:
"""
Search Spotify catalog.
Args:
query: Search query string
types: Result types (track, album, artist, playlist)
Returns:
Dict with type-specific results arrays
Raises:
RuntimeError: If API request fails
Example:
>>> api = WebAPI(client_id="...", client_secret="...")
>>> results = api.search("Radiohead", types=["artist"])
>>> print(results["artists"]["items"][0]["name"])
Radiohead
"""
```
### Local Documentation Build
```bash
cd docs
pip install sphinx sphinx_rtd_theme
make html
open _build/html/index.html
```
## Versioning
**Scheme:** Semantic Versioning (SemVer)
**Format:** `MAJOR.MINOR.PATCH`
**Current Version:** 1.1.0
**Version History:**
- 1.0.0: Initial release
- 1.1.0: Bug fixes, minor feature additions
**Version Location:** `minim/__init__.py`
```python
__version__ = "1.1.0"
```
**Git Tags:**
```bash
git tag v1.1.0
git push origin v1.1.0
```
## Release Process
**Current (Manual):**
1. Update version in `minim/__init__.py`
2. Update `CHANGELOG.md` (if exists)
3. Commit changes: `git commit -m "Bump version to 1.1.0"`
4. Create tag: `git tag v1.1.0`
5. Push: `git push origin main --tags`
6. GitHub automatically triggers ReadTheDocs build
**Planned for v2 (Automated):**
1. Create release branch: `git checkout -b release/1.2.0`
2. Update version and changelog
3. Open pull request
4. Merge to main
5. GitHub Actions workflow:
- Run tests
- Build package: `python -m build`
- Publish to PyPI: `twine upload dist/*`
- Create GitHub release with changelog
- Trigger ReadTheDocs build
## Distribution Channels
### Current
**GitHub Releases:**
- Source code archives (`.tar.gz`, `.zip`)
- No pre-built binaries
- Download: https://github.com/bbye98/minim/releases
**ReadTheDocs:**
- Documentation only
- No package distribution
### Planned (v2)
**PyPI:**
- `pip install minim`
- Versioned releases
- Automatic dependency resolution
**Conda-Forge:**
- `conda install -c conda-forge minim`
- Cross-platform binaries
- Dependency management via conda
## Containerization
**Not Applicable:** minim is a library, not a service.
**Hypothetical Use Case:** Containerized script using minim
**Dockerfile:**
```dockerfile
FROM python:3.11-slim
# Install FFmpeg
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Install minim
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -e .
# Run script
CMD ["python", "script.py"]
```
**Docker Compose:**
```yaml
version: '3.8'
services:
minim-script:
build: .
environment:
- SPOTIFY_CLIENT_ID=${SPOTIFY_CLIENT_ID}
- SPOTIFY_CLIENT_SECRET=${SPOTIFY_CLIENT_SECRET}
volumes:
- ./audio:/app/audio
- ./config:/root/.minim.cfg
```
## Monitoring and Logging
**Not Applicable:** minim is a library. Monitoring and logging are the responsibility of the calling application.
**Library Behavior:**
- No built-in logging (uses `warnings` module for non-critical issues)
- Errors raised as exceptions (caller handles logging)
- No metrics, no telemetry, no health checks
**Caller Responsibility:**
```python
import logging
from minim import spotify
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Use minim with logging
try:
api = spotify.WebAPI(client_id="...", client_secret="...")
api.set_flow("client_credentials")
api.set_access_token()
results = api.search("Radiohead", types=["artist"])
logger.info(f"Found {len(results['artists']['items'])} artists")
except RuntimeError as e:
logger.error(f"API error: {e}")
except Exception as e:
logger.exception(f"Unexpected error: {e}")
```
## Security Considerations
### Credential Storage
**Risk:** Plain text tokens in `~/minim.cfg`
**Mitigation (Not Implemented):**
- Encrypt config file
- Use OS keychain (Keyring library)
- Use environment variables only
- Set restrictive file permissions (`chmod 600`)
### Dependency Vulnerabilities
**Risk:** Outdated dependencies with known CVEs
**Mitigation:**
```bash
# Scan for vulnerabilities
pip install safety
safety check
# Update dependencies
pip install --upgrade cryptography mutagen requests
```
### API Key Exposure
**Risk:** Hardcoded credentials in scripts
**Mitigation:**
- Use environment variables
- Use config file outside version control
- Add `minim.cfg` to `.gitignore`
### Private API Usage
**Risk:** Terms of service violations (Qobuz, TIDAL, Spotify lyrics)
**Mitigation:**
- Use only public APIs in production
- Document risks in README
- Obtain official API access if possible
## Scalability
**Not Applicable:** minim is a library for personal use, not a scalable service.
**Limitations:**
- Synchronous, blocking operations
- No connection pooling
- No rate limiting
- No caching
- Single-threaded
**For High-Volume Use:**
- Implement async version using `aiohttp`
- Add connection pooling
- Implement rate limiting and backoff
- Cache API responses (Redis, Memcached)
- Use task queue (Celery, RQ) for background processing
## Backup and Recovery
**Config File Backup:**
```bash
# Backup
cp ~/minim.cfg ~/minim.cfg.backup
# Restore
cp ~/minim.cfg.backup ~/minim.cfg
```
**Recommendation:** Exclude from cloud backup (contains sensitive tokens) or encrypt backups.
## Maintenance
**Current Status:** v1 in maintenance mode
**Maintenance Activities:**
- Bug fixes for critical issues
- Security updates for dependencies
- No new features
**Active Development:** v2 rewrite on `dev` branch
**Support Channels:**
- GitHub Issues: https://github.com/bbye98/minim/issues
- GitHub Discussions: https://github.com/bbye98/minim/discussions
## Summary
minim deployment is straightforward:
1. **Install from source:** `git clone` + `pip install -e .`
2. **Configure credentials:** Environment variables or `~/minim.cfg`
3. **Import and use:** `from minim import spotify`
No server deployment, no containers, no orchestration. The library runs in the caller's Python process.
For production use cases requiring scalability, security, and robustness, consider:
- Wrapping minim in a web service (Flask, FastAPI)
- Implementing async operations
- Adding rate limiting and caching
- Using secure credential storage
- Monitoring and logging
The v2 rewrite addresses many deployment concerns (PyPI publication, async support, secure storage) while maintaining the simple library architecture.