Nixos/README.org

#+TITLE: Installing NixOS on a Proxmox VM using nixos-anywhere
#+AUTHOR: Alexander Derevianko
#+DATE: <2025-07-26 Sat>
#+OPTIONS: toc:t num:nil

*Abstract*
This guide documents methods for installing NixOS on a Proxmox virtual machine. It covers a remote deployment using =nixos-anywhere= as well as a more advanced method of building a Proxmox image template directly with Nix. It also covers post-installation steps for secrets management with =sops-nix= and lists available custom modules.

* TL;DR: Quick Install Guide (Remote Install)
1. *Prepare VM:* Boot the target Proxmox VM from a NixOS ISO and set a root password:
   #+begin_src sh
   passwd
   #+end_src

2. *Deploy NixOS:* From your workstation, run =nixos-anywhere=, pointing to your flake and the VM's IP address.
   #+begin_src sh
   nix run github:nix-community/nixos-anywhere -- \
     --flake .#your-machine-name \
     --target-host root@<vm-ip-address>
   #+end_src

3. *Get Host Key:* After installation, SSH into the new VM and get its host AGE key.
   #+begin_src sh
   ssh root@<vm-ip-address>
   nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
   #+end_src

4. *Update Secrets:* On your workstation, add the new AGE key to =.sops.yaml= and re-encrypt secrets.
   #+begin_src sh
   sops updatekeys secrets/secrets.yaml
   #+end_src

* Table of Contents                                                            :TOC:
- [[#tldr-quick-install-guide-remote-install][TL;DR: Quick Install Guide (Remote Install)]]
- [[#installation-methods][Installation Methods]]
  - [[#method-1-remote-installation-with-nixos-anywhere][Method 1: Remote Installation with nixos-anywhere]]
  - [[#method-2-deployment-via-proxmox-image-template][Method 2: Deployment via Proxmox Image Template]]
- [[#post-installation-secrets-management][Post-Installation: Secrets Management]]
  - [[#step-1-generating-the-host-age-key][Step 1: Generating the Host AGE Key]]
  - [[#step-2-updating-sops-and-re-encrypting-secrets][Step 2: Updating SOPS and Re-encrypting Secrets]]
- [[#optional-nixos-modules][Optional NixOS Modules]]
  - [[#reverse-proxies][Reverse Proxies]]
  - [[#file-servers][File Servers]]
  - [[#search-engines][Search Engines]]
- [[#notes-and-configuration-details][Notes and Configuration Details]]
  - [[#disko-configuration-for-proxmox-mbr-boot][Disko Configuration for Proxmox (MBR Boot)]]
  - [[#generating-hardware-configuration][Generating Hardware Configuration]]
- [[#todos][TODOs]]
- [[#inspiration][Inspiration]]
- [[#fujin-install][Fujin install]]
  - [[#boot-drive][Boot Drive]]
  - [[#minimal][Minimal]]

* Installation Methods
** Method 1: Remote Installation with nixos-anywhere
This method involves booting a minimal NixOS ISO on the target VM and then "pushing" the full configuration to it remotely.

*** Prerequisites
The minimal NixOS installation ISO does not have a default password for the =root= user. The =nixos-anywhere= command requires SSH access, which necessitates a password.

1. Boot the Proxmox VM using the minimal NixOS installation ISO.
2. Open a terminal on the VM's console.
3. Set a password for the =root= user by running the following command:
   #+begin_src sh
   passwd
   #+end_src

*** Deploying NixOS
With the root password set on the target VM, run =nixos-anywhere= from your local machine to deploy your NixOS configuration.

#+begin_src sh
nix run github:nix-community/nixos-anywhere -- \
  --flake .#susano-minimal \
  --target-host root@192.168.1.85
#+end_src
After this step, proceed to the [[#post-installation-secrets-management][Post-Installation: Secrets Management]] section.

** Method 2: Deployment via Proxmox Image Template
This method involves building a complete Proxmox backup file (=.vma.zst=) directly with Nix. This image can then be restored in Proxmox to create a new VM or a reusable template. This approach is faster for creating multiple machines.

*** Step 1: Build the Proxmox Image
Build the image using a dedicated flake output. This will produce a compressed Proxmox backup file in the =./result/= directory.
#+begin_src sh
nix build .#izanami-proxmox
#+end_src

*** Step 2: Copy Image to Proxmox Host
You must copy the image to the directory Proxmox uses for backups. First, find this location by running the following command on your Proxmox host:
#+begin_src sh
cat /etc/pve/storage.cfg
#+end_src
Look for a storage location (like =dir: local=) that includes =backup= in its =content= list. The =path= for that storage (e.g., =/var/lib/vz=) is the destination. Backups are typically stored in a =dump= subdirectory within that path.

Use =scp= to copy the generated =.vma.zst= file to the backup directory.
#+begin_src sh
scp result/vzdump-*.vma.zst root@192.168.1.53:/var/lib/vz/dump/
#+end_src

*** Step 3: Restore Image from Proxmox UI
1. Navigate to your Proxmox web UI.
2. Select your backup storage location from the left-hand menu.
3. Go to the *Backups* tab, select the newly uploaded image, and click the *Restore* button.
4. *Important:* In the restore dialog, ensure the *Unique* checkbox is enabled. This generates a new MAC address and other unique identifiers for the restored VM.

*** Step 4: Test and Convert to Template
1. *(Recommended)* Before creating a template, test the restored VM. Create a full clone of it, start the clone, and verify you can access it as expected (e.g., via SSH with the pre-configured user).
   #+begin_src sh
   ssh izanami@some_ip
   #+end_src
2. Once confirmed, you can convert the original restored VM into a template for easy reuse. Right-click the VM and select *Convert to template*.

* Post-Installation: Secrets Management
*(This section is primarily for Method 1, or for when a new host key needs to be added after using Method 2)*

** Step 1: Generating the Host AGE Key
After the installation is complete, you will need the host's AGE key to manage secrets with tools like =sops-nix=.

1. SSH into the newly installed NixOS machine.
   #+begin_src sh
   ssh root@192.168.1.85
   #+end_src

2. Run the following command to convert the host's public SSH key to an AGE key.
   #+begin_src sh
   nix-shell -p ssh-to-age --run 'cat /etc/ssh/ssh_host_ed25519_key.pub | ssh-to-age'
   #+end_src
3. Copy the output AGE key for the next step.

** Step 2: Updating SOPS and Re-encrypting Secrets
The new AGE key must be added to your =.sops.yaml= file.

1. Open the =.sops.yaml= file in the root of your Nix flake.
2. Replace the old key for the host with the new key you generated.
   #+begin_src yaml
   keys:
     - &primary age19wvqtn4ju6k4vs8fxr34unl6xx4cv04jw0lx9ps20xlde927zfssgl4qke
     - &susano age1vkfq9gpqfpyq3s9e79e6vw8kv9485tzna4fm3dy6p0u9uz9feu8qr9sgcf # <--- REPLACE THIS WITH THE NEW KEY
   creation_rules:
     - path_regex: secrets/secrets.yaml$
       key_groups:
         - age:
             - *primary
             - *susano
   #+end_src

3. After saving, run the =updatekeys= command to re-encrypt the secrets file with the new set of keys.
   #+begin_src sh
   sops updatekeys secrets/secrets.yaml
   #+end_src

* Optional NixOS Modules
** Reverse Proxies
The following modules can be enabled to provide a reverse proxy.

*Note:* Previously, all reverse proxy modules were considered non-functional. Recent troubleshooting has provided a fix for Traefik, but the other modules may still have issues.

*** NGINX
The initial switch to this configuration may be slow as it waits for ACME to issue SSL certificates.
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    nginx.enable = true;
  };
};
#+end_src

*** Traefik
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    traefik.enable = true;
  };
};
#+end_src

**** Troubleshooting Traefik ACME with DuckDNS
- *Context:* Issues getting an ACME certificate from DuckDNS with Traefik.
- *Roadblock:* The process was failing, but eventually succeeded.
- *Possible Solutions:*
  - Setting =disablePropagationCheck = true;= for the DNS challenge.
  - Extending the =delay= for the DNS challenge.
- *Notes:* It's unclear which specific option resolved the issue. The first time Traefik tries to get a certificate it might fail, and a restart of the service is needed. After some time, the Let's Encrypt certificate will be received.

*** Caddy
#+begin_src nix
dov = {
  # Reverse Proxy
  reverse-proxy = {
    caddy.enable = true;
  };
};
#+end_src

** File Servers
*** copyparty
Provides a web-based file manager. For more information, see the [[https://github.com/9001/copyparty][official documentation]].
#+begin_src nix
dov = {
  file-server.copyparty.enable = true;
};
#+end_src

*Dependency:* For =copyparty= to function correctly in this configuration, it requires a Samba share to be mounted to the path =/MEDIA=. Therefore, the =samba= module must also be enabled:
#+begin_src nix
dov = {
  samba.enable = true;
};
#+end_src

** Search Engines
*** searxng
A privacy-respecting metasearch engine. For more information, see the [[https://wiki.nixos.org/wiki/SearXNG][NixOS Wiki page]].
#+begin_src nix
dov = {
  searxng.enable = true;
};
#+end_src

* Notes and Configuration Details
** Disko Configuration for Proxmox (MBR Boot)
A critical requirement for ensuring a NixOS VM can boot correctly in Proxmox is the disk partition scheme. Proxmox expects a Master Boot Record (MBR) compatible setup.

When using =disko= for declarative disk management, you must configure it to create a GPT partition table that includes a special 1M BIOS boot partition (type =EF02=). This partition is specifically used by GRUB for MBR compatibility.

Here is an example snippet for the =disko= configuration:

#+begin_src nix
{
  disko.devices = {
    disk = {
      main = {
        device = "/dev/sda";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            boot = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            # ... your other partitions like root, swap, etc.
          };
        };
      };
    };
  };
}
#+end_src

For a complete example, you can refer to the official =disko= repository: [[https://github.com/nix-community/disko/blob/master/example/gpt-bios-compat.nix][gpt-bios-compat.nix]].

** Generating Hardware Configuration
The =nixos-anywhere= tool can automatically generate a hardware configuration file from the target machine. This is useful for capturing machine-specific settings.

#+begin_src sh
nix run github:nix-community/nixos-anywhere -- \
  --flake .#your-flake-output \
  --target-host root@192.168.1.85 \
  --generate-hardware-config ./hardware-configuration.nix
#+end_src

* TODOs
- [ ] Investigate and fix remaining issues with reverse proxy modules (NGINX, Caddy).
- [ ] Troubleshoot and fix an issue that occurs when reloading the NixOS configuration remotely, which breaks the SSH pipe and requires entering the root password three times.
- [ ] Investigate and resolve the issue where updating a user's password declaratively using a secret managed by =sops= failed after the initial installation.
- [ ] Refactor the =disko= configuration to make the disk device name (e.g., =/dev/sda=) a variable. This will avoid hardcoding the value and make the configuration more portable.
- [ ] Create a custom ISO image to streamline the installation process, potentially pre-configuring items like the root user to avoid manual console steps.
- [ ] Develop an automated installation script to handle the post-install process, such as fetching the AGE key and updating sops, based on [[https://unmovedcentre.com/posts/remote-install-nixos-config/#update-sops-file][this guide]].

* Inspiration
The configuration and structure of this setup were inspired by the following repository:
- [[https://github.com/notthebee/nix-config][notthebee/nix-config]]

* Fujin install
** Boot Drive
1. Boot into izanami
2. git clone repo 'git clone https://github.com/LichHunter/susano-nixos'
3. Disko install fujin-minimal 'nix run github:nix-community/disko#disko-install -- --flake .#fujin-minimal --disk main /dev/nvme0n1'
4. Reboot
** Minimal
1. Mount backup drive 'mkdir /tmp/drive; sudo mount /dev/sda1 /tmp/drive'
2. Copy latest backup from drive to home folder
3. Unarchive it (you should now have ssh keys)
4. Git clone nixos repo 'git clone git@github.com:LichHunter/susano-nixos'
5. Upgrade to man configuration
  #+begin_src bash
sudo nixos-rebuild boot --flake .#fujin
  #+end_src
6. Install emacs
  #+begin_src bash
git clone --depth 1 https://github.com/doomemacs/doomemacs ~/.config/emacs
~/.config/emacs/bin/doom install
  #+end_src
7. Copy configs for emacs
  #+begin_src bash
cp ~/susano-nixos/machines/fujin/main/doom-configs ~/.config/doom
  #+end_src
8. Update doom-emacs
  #+begin_src bash
~/.config/emacs/bin/doom sync
  #+end_src