Nixos/README.org

#+TITLE: Installing NixOS on a Proxmox VM using nixos-anywhere
#+AUTHOR: Alexander Derevianko
#+DATE: <2025-07-26 Sat>
#+OPTIONS: toc:t num:nil

*Abstract*
This guide documents methods for installing NixOS on a Proxmox virtual machine and a bare-metal machine (Fujin). It covers remote deployment, bare-metal installation with Disko, and building Proxmox image templates. It also covers post-installation steps for secrets management with =sops-nix= and lists available custom modules.

* TL;DR: Fujin Quick Install (Bare Metal)
1. *Boot Live Environment:* Boot into a NixOS installer or another Linux environment (like Izanami).
2. *Clone Repo:* Clone this repository.
   #+begin_src sh
   git clone https://github.com/LichHunter/nixos
   #+end_src
3. *Install with Disko:* Run the Disko installer script for the minimal configuration.
   #+begin_src sh
   nix run github:nix-community/disko#disko-install -- --flake .#fujin-minimal --disk main /dev/nvme0n1
   #+end_src
4. *Reboot & Setup:* Reboot into the new system. Mount your backup drive, restore your SSH keys, and clone the repository again.
5. *Setup Root SSH for Remote Builds:* Generate SSH key for root user to enable remote builds.
   #+begin_src sh
   sudo ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519
   #+end_src
6. *Update Izanagi Configuration:* Add the new root public key to izanagi's authorized keys for remote builds.
   #+begin_src sh
   sudo cat /root/.ssh/id_ed25519.pub
   # Copy this key and add it to machines/izanagi/default.nix in the openssh.authorizedKeys.keys section
   #+end_src
7. *Rebuild to Main Config:* Use the custom script to switch to the full configuration.
   #+begin_src sh
   ./bin/rebuild.sh boot
   #+end_src

* TL;DR: Proxmox Quick Install (Remote)
1. *Prepare VM:* Boot the target Proxmox VM from a NixOS ISO and set a root password:
   #+begin_src sh
   passwd
   #+end_src
2. *Deploy NixOS:* From your workstation, run =nixos-anywhere=.
   #+begin_src sh
   nix run github:nix-community/nixos-anywhere -- --flake .#your-machine-name --target-host root@<vm-ip-address>
   #+end_src
3. *Manage Secrets:* Get the host's AGE key, add it to =.sops.yaml=, and re-encrypt.
   #+begin_src sh
   sops updatekeys secrets/secrets.yaml
   #+end_src

* Table of Contents                                                            :TOC:
- [[#tldr-fujin-quick-install-bare-metal][TL;DR: Fujin Quick Install (Bare Metal)]]
- [[#tldr-proxmox-quick-install-remote][TL;DR: Proxmox Quick Install (Remote)]]
- [[#fujin-installation-bare-metal][Fujin Installation (Bare Metal)]]
  - [[#boot-drive-installation][Boot Drive Installation]]
  - [[#minimal-system-setup][Minimal System Setup]]
- [[#proxmox-installation-methods][Proxmox Installation Methods]]
  - [[#method-1-remote-installation-with-nixos-anywhere][Method 1: Remote Installation with nixos-anywhere]]
  - [[#method-2-deployment-via-proxmox-image-template][Method 2: Deployment via Proxmox Image Template]]
- [[#post-installation-secrets-management][Post-Installation: Secrets Management]]
  - [[#step-1-generating-the-host-age-key][Step 1: Generating the Host AGE Key]]
  - [[#step-2-updating-sops-and-re-encrypting-secrets][Step 2: Updating SOPS and Re-encrypting Secrets]]
- [[#available-utility-scripts][Available Utility Scripts]]
  - [[#rebuildsh---enhanced-nixos-rebuild-wrapper][rebuild.sh - Enhanced NixOS Rebuild Wrapper]]
  - [[#backupsh---automated-backup-script][backup.sh - Automated Backup Script]]
  - [[#hash-utilsh---file-hash-verification][hash-util.sh - File Hash Verification]]
- [[#optional-nixos-modules][Optional NixOS Modules]]
  - [[#reverse-proxies][Reverse Proxies]]
  - [[#file-servers][File Servers]]
  - [[#search-engines][Search Engines]]
- [[#notes-and-configuration-details][Notes and Configuration Details]]
  - [[#remote-build-configuration][Remote Build Configuration]]
  - [[#disko-configuration-for-proxmox-mbr-boot][Disko Configuration for Proxmox (MBR Boot)]]
  - [[#generating-hardware-configuration][Generating Hardware Configuration]]
- [[#todos][TODOs]]
- [[#inspiration][Inspiration]]

* Fujin Installation (Bare Metal)
** Boot Drive Installation
1. Boot into izanami or another suitable Linux live environment.
2. Clone the repository:
   #+begin_src sh
   git clone https://github.com/LichHunter/nixos
   #+end_src
3. Use Disko to install the minimal configuration for Fujin.
   #+begin_src sh
   nix run github:nix-community/disko#disko-install -- --flake .#fujin-minimal --disk main /dev/nvme0n1
   #+end_src
4. Reboot the machine.

** Minimal System Setup
1. Mount your backup drive:
   #+begin_src sh
   mkdir /tmp/drive; sudo mount /dev/sda1 /tmp/drive
   #+end_src
2. Copy the latest backup from the drive to your home folder.
3. Unarchive the backup to restore essential files, including your SSH keys.
4. Clone your NixOS repository using your SSH key:
   #+begin_src sh
   git clone git@github.com:LichHunter/nixos
   #+end_src
5. *(Optional)* Setup SSH for remote builds. Generate SSH key for root user to enable remote builds on izanagi:
   #+begin_src sh
   sudo ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519
   #+end_src
6. *(Optional)* Add the root public key to izanagi configuration. First, get the public key:
   #+begin_src sh
   sudo cat /root/.ssh/id_ed25519.pub
   #+end_src
   Then add this key to the =openssh.authorizedKeys.keys= section in =machines/izanagi/default.nix= and rebuild izanagi.
7. Upgrade to the main configuration using the provided rebuild script. This script handles the full =nixos-rebuild boot --flake .#fujin= command, including =sudo= and build host settings.
   #+begin_src bash
   ./bin/rebuild.sh boot
   #+end_src
8. Install Emacs:
   #+begin_src bash
   git clone --depth 1 https://github.com/doomemacs/doomemacs ~/.config/emacs
   ~/.config/emacs/bin/doom install
   #+end_src
9. Copy your custom Doom Emacs configs:
   #+begin_src bash
   cp -r ~/nixos/machines/fujin/main/doom-configs/* ~/.config/doom/
   #+end_src
10. Sync your Doom Emacs configuration:
    #+begin_src bash
    ~/.config/emacs/bin/doom sync
    #+end_src

* Proxmox Installation Methods
** Method 1: Remote Installation with nixos-anywhere
This method involves booting a minimal NixOS ISO on the target VM and then "pushing" the full configuration to it remotely.

*** Prerequisites
The minimal NixOS installation ISO does not have a default password for the =root= user. The =nixos-anywhere= command requires SSH access, which necessitates a password.

1. Boot the Proxmox VM using the minimal NixOS installation ISO.
2. Open a terminal on the VM's console.
3. Set a password for the =root= user by running the following command:
   #+begin_src sh
   passwd
   #+end_src

*** Deploying NixOS
With the root password set on the target VM, run =nixos-anywhere= from your local machine to deploy your NixOS configuration.

#+begin_src sh
nix run github:nix-community/nixos-anywhere -- \
  --flake .#susano-minimal \
  --target-host root@192.168.1.85
#+end_src
After this step, proceed to the [[#post-installation-secrets-management][Post-Installation: Secrets Management]] section.

** Method 2: Deployment via Proxmox Image Template
This method involves building a complete Proxmox backup file (=.vma.zst=) directly with Nix. This image can then be restored in Proxmox to create a new VM or a reusable template. This approach is faster for creating multiple machines.

*** Step 1: Build the Proxmox Image
Build the image using a dedicated flake output. This will produce a compressed Proxmox backup file in the =./result/= directory.
#+begin_src sh
nix build .#izanami-proxmox
#+end_src

*** Step 2: Copy Image to Proxmox Host
You must copy the image to the directory Proxmox uses for backups. First, find this location by running the following command on your Proxmox host:
#+begin_src sh
cat /etc/pve/storage.cfg
#+end_src
Look for a storage location (like =dir: local=) that includes =backup= in its =content= list. The =path= for that storage (e.g., =/var/lib/vz=) is the destination. Backups are typically stored in a =dump= subdirectory within that path.

Use =scp= to copy the generated =.vma.zst= file to the backup directory.
#+begin_src sh
scp result/vzdump-*.vma.zst root@192.168.1.53:/var/lib/vz/dump/
#+end_src

*** Step 3: Restore Image from Proxmox UI
1. Navigate to your Proxmox web UI.
2. Select your backup storage location from the left-hand menu.
3. Go to the *Backups* tab, select the newly uploaded image, and click the *Restore* button.
4. *Important:* In the restore dialog, ensure the *Unique* checkbox is enabled. This generates a new MAC address and other unique identifiers for the restored VM.

*** Step 4: Test and Convert to Template
1. *(Recommended)* Before creating a template, test the restored VM. Create a full clone of it, start the clone, and verify you can access it as expected (e.g., via SSH with the pre-configured user).
   #+begin_src sh
   ssh izanami@some_ip
   #+end_src
2. Once confirmed, you can convert the original restored VM into a template for easy reuse. Right-click the VM and select *Convert to template*.

* Post-Installation: Secrets Management
*(This section is primarily for Method 1, or for when a new host key needs to be added after using Method 2)*

** Step 1: Generating the Host AGE Key
After the installation is complete, you will need the host's AGE key to manage secrets with tools like =sops-nix=.

1. SSH into the newly installed NixOS machine.
   #+begin_src sh
   ssh root@192.168.1.85
   #+end_src

2. Run the following command to convert the host's public SSH key to an AGE key.
   #+begin_src sh
   nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
   #+end_src
3. Copy the output AGE key for the next step.

** Step 2: Updating SOPS and Re-encrypting Secrets
The new AGE key must be added to your =.sops.yaml= file.

1. Open the =.sops.yaml= file in the root of your Nix flake.
2. Replace the old key for the host with the new key you generated.
   #+begin_src yaml
   keys:
     - &primary age19wvqtn4ju6k4vs8fxr34unl6xx4cv04jw0lx9ps20xlde927zfssgl4qke
     - &susano age1vkfq9gpqfpyq3s9e79e6vw8kv9485tzna4fm3dy6p0u9uz9feu8qr9sgcf # <--- REPLACE THIS WITH THE NEW KEY
   creation_rules:
     - path_regex: secrets/secrets.yaml$
       key_groups:
         - age:
             - *primary
             - *susano
   #+end_src

3. After saving, run the =updatekeys= command to re-encrypt the secrets file with the new set of keys.
   #+begin_src sh
   sops updatekeys secrets/secrets.yaml
   #+end_src

* Available Utility Scripts
This repository includes several utility scripts in the =bin/= directory to simplify common tasks.

** rebuild.sh - Enhanced NixOS Rebuild Wrapper
A powerful wrapper around =nixos-rebuild= that provides intelligent defaults and simplified syntax for various deployment scenarios.

*** Features
- Smart defaults for build hosts and machine configurations
- Automatic sudo handling for local vs remote operations
- Support for remote builds and deployments
- Dry-run capabilities for testing changes
- Comprehensive help and error messages

*** Basic Usage
#+begin_src sh
# Local build and switch (uses local machine)
./bin/rebuild.sh switch --local-build

# Remote build, local deployment (default behavior)
./bin/rebuild.sh switch

# Build only, no deployment
./bin/rebuild.sh build
#+end_src

*** Remote Operations
#+begin_src sh
# Deploy to remote machine (auto-detects machine name from hostname)
./bin/rebuild.sh switch --target-host user@remote-host

# Build on izanagi, deploy to susano
./bin/rebuild.sh switch --machine susano --target-host susano@susano

# Build on specific host, deploy to another
./bin/rebuild.sh switch --build-host builder@build-host --target-host user@target-host
#+end_src

*** Advanced Examples
#+begin_src sh
# Test configuration without making it permanent
./bin/rebuild.sh test --target-host susano@susano

# Dry run to see what would be built/changed
./bin/rebuild.sh switch --dry-run --verbose

# Build specific machine with custom build host
./bin/rebuild.sh build --machine fujin --build-host powerful@workstation

# Fast remote deployment (skip building Nix)
./bin/rebuild.sh switch --target-host remote --fast

# Verbose output with detailed tracing
./bin/rebuild.sh build --verbose --show-trace
#+end_src

*** Command Reference
**** Commands
- =build= - Build the configuration without deploying
- =test= - Build and activate temporarily (reverts on reboot)
- =switch= - Build and activate permanently
- =boot= - Build and set as boot default (requires reboot)

**** Options
- =--local-build= - Force building on local machine
- =--build-host HOST= - Specify remote build host (default: =izanagi@izanagi=)
- =--target-host HOST= - Deploy to remote target host
- =--machine NAME= - Machine configuration name (auto-detected if not specified)
- =--use-remote-sudo= - Use sudo on remote target (auto-enabled for switch/boot)
- =--fast= - Skip building Nix (useful for remote builds)
- =--dry-run= - Show what would happen without making changes
- =--verbose= - Enable verbose output
- =--show-trace= - Show detailed error traces

** backup.sh - Automated Backup Script
Creates timestamped tar archives of important directories and files.

*** Usage
#+begin_src sh
# Create backup with current timestamp
./bin/backup.sh
#+end_src

*** What it backs up
- =~/.gnupg= - GPG keys and configuration
- =~/.ssh= - SSH keys and configuration
- =~/Documents= - Personal documents
- =~/Pictures= - Photos and images
- =~/org= - Org-mode files
- =~/nixos-dotfiles= - NixOS configuration repository
- =~/nixos= - Alternative NixOS configuration location
- =~/hetzner-nixos= - Hetzner-specific configurations
- =~/.authinfo.gpg= - Encrypted authentication information

*** Backup location
Backups are stored in =~/Backup/= with filenames like =backup-YYYYMMDDHHMIN.tar=.

** hash-util.sh - File Hash Verification
Verifies file integrity using SHA256 checksums.

*** Usage
#+begin_src sh
# Verify a file against its expected hash
./bin/hash-util.sh --path /path/to/file --hash <expected-sha256-hash>
#+end_src

*** Examples
#+begin_src sh
# Verify downloaded ISO
./bin/hash-util.sh --path nixos.iso --hash a1b2c3d4e5f6...

# Check configuration file integrity
./bin/hash-util.sh --path configuration.nix --hash $(sha256sum configuration.nix | cut -d' ' -f1)
#+end_src

* Optional NixOS Modules
** Reverse Proxies
The following modules can be enabled to provide a reverse proxy.

*Note:* Previously, all reverse proxy modules were considered non-functional. Recent troubleshooting has provided a fix for Traefik, but the other modules may still have issues.

*** NGINX
The initial switch to this configuration may be slow as it waits for ACME to issue SSL certificates.
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    nginx.enable = true;
  };
};
#+end_src

*** Traefik
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    traefik.enable = true;
  };
};
#+end_src

**** Troubleshooting Traefik ACME with DuckDNS
- *Context:* Issues getting an ACME certificate from DuckDNS with Traefik.
- *Roadblock:* The process was failing, but eventually succeeded.
- *Possible Solutions:*
  - Setting =disablePropagationCheck = true;= for the DNS challenge.
  - Extending the =delay= for the DNS challenge.
- *Notes:* It's unclear which specific option resolved the issue. The first time Traefik tries to get a certificate it might fail, and a restart of the service is needed. After some time, the Let's Encrypt certificate will be received.

*** Caddy
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    caddy.enable = true;
  };
};
#+end_src

** File Servers
*** copyparty
Provides a web-based file manager. For more information, see the [[https://github.com/9001/copyparty][official documentation]].
#+begin_src nix
dov = {
  file-server.copyparty.enable = true;
};
#+end_src

*Dependency:* For =copyparty= to function correctly in this configuration, it requires a Samba share to be mounted to the path =/MEDIA=. Therefore, the =samba= module must also be enabled:
#+begin_src nix
dov = {
  samba.enable = true;
};
#+end_src

** Search Engines
*** searxng
A privacy-respecting metasearch engine. For more information, see the [[https://wiki.nixos.org/wiki/SearXNG][NixOS Wiki page]].
#+begin_src nix
dov = {
  searxng.enable = true;
};
#+end_src

* Notes and Configuration Details
** Remote Build Configuration
To leverage remote builds (e.g., building fujin configurations on izanagi), you need to set up SSH keys for the root user:

1. On the machine that will initiate builds (e.g., fujin), generate SSH keys for root:
   #+begin_src sh
   sudo ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519
   #+end_src

2. Add the root public key to the build host's (e.g., izanagi) authorized keys in the NixOS configuration:
   #+begin_src nix
   users.users.${username} = {
     openssh.authorizedKeys.keys = [
       "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBcGhVpjmWEw1GEw0y/ysJPa2v3+u/Rt/iES/Se2huH2 alexander0derevianko@gmail.com"
       "ssh-ed25519 AAAA... root@fujin"  # Add the new root key here
     ];
   };
   #+end_src

3. The =./bin/rebuild.sh= script is configured to use remote builds by default. To use local builds instead, use the =--local-build= flag:
   #+begin_src sh
   ./bin/rebuild.sh switch --local-build
   #+end_src

** Disko Configuration for Proxmox (MBR Boot)
A critical requirement for ensuring a NixOS VM can boot correctly in Proxmox is the disk partition scheme. Proxmox expects a Master Boot Record (MBR) compatible setup.

When using =disko= for declarative disk management, you must configure it to create a GPT partition table that includes a special 1M BIOS boot partition (type =EF02=). This partition is specifically used by GRUB for MBR compatibility.

Here is an example snippet for the =disko= configuration:

#+begin_src nix
{
  disko.devices = {
    disk = {
      main = {
        device = "/dev/sda";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            # ... your other partitions like root, swap, etc.
          };
        };
      };
    };
  };
}
#+end_src

For a complete example, you can refer to the official =disko= repository: [[https://github.com/nix-community/disko/blob/master/example/gpt-bios-compat.nix][gpt-bios-compat.nix]].

** Generating Hardware Configuration
The =nixos-anywhere= tool can automatically generate a hardware configuration file from the target machine. This is useful for capturing machine-specific settings.

#+begin_src sh
nix run github:nix-community/nixos-anywhere -- \
  --flake .#your-flake-output \
  --target-host root@192.168.1.85 \
  --generate-hardware-config ./hardware-configuration.nix
#+end_src

* TODOs
- [X] Update izanagi to include git by default.
- [X] Add NetworkManager to the fujin-minimal configuration.
- [ ] Investigate and fix remaining issues with reverse proxy modules (NGINX, Caddy).
- [ ] Troubleshoot and fix an issue that occurs when reloading the NixOS configuration remotely, which breaks the SSH pipe and requires entering the root password three times.
  Seems like this can be fixed if we allow remote user, for example 'fujin', to execute switch without sudo, but claude gave strange code to implement this. Will need to look into it some time later
- [ ] Investigate and resolve the issue where updating a user's password declaratively using a secret managed by =sops= failed after the initial installation.
- [X] Refactor the =disko= configuration to make the disk device name (e.g., =/dev/sda=) a variable. This will avoid hardcoding the value and make the configuration more portable.
  You can provide drive name when installing with disko and then update disko-config. Example of install command src_bash[:exports code]{nix run github:nix-community/disko#disko-install -- --flake .#fujin-minimal --disk main /dev/nvme0n1}
- [X] Create a custom ISO image to streamline the installation process, potentially pre-configuring items like the root user to avoid manual console steps.
  Now you can build izanami iso to for it. Use src_bash[:exports code]{nix build .#izanami-iso}
- [ ] Develop an automated installation script to handle the post-install process, such as fetching the AGE key and updating sops, based on [[https://unmovedcentre.com/posts/remote-install-nixos-config/#update-sops-file][this guide]].
- [X] Document remote build setup process and add automation for SSH key exchange between machines.
  There is now rebuild.sh script that can be used for easy remote build
- [ ] Add prometheus for services on Proxmox
- [ ] Add matrix service in Proxmox

* Inspiration
The configuration and structure of this setup were inspired by the following repository:
- [[https://github.com/notthebee/nix-config][notthebee/nix-config]]