In January I started setting up a home server/NAS based on FreeBSD on a HP Microserver. Read about my setup in part 1 and part 2.

While I generally like the idea (BSD license, complete base system in one repo) and community behind FreeBSD, I have the feeling that the project is missing some manpower. VIMAGE is still experimental and in combination with PF it will crash every night (because of a Cron job). There seems to be a bug that IPSec tunnels bypass the firewall. There is no AMD support in bhyve yet (it’s scheduled for October 2014 with the 10.1 release), so I cannot run any virtual machines on my home server.

So my concerns about manpower and the fact that I cannot run any virtual machines yet lead me back to Debian Linux.

The Plan

Because with Debian I can use KVM and run multiple virtual machines, I’ll set up a minimalistic, fully encrypted base system with Debian. All services the NAS will supply will run in virtual machines that run Ubuntu, Debian or FreeBSD.

The Setup

Before we finally talk about the setup, I’d like to give attribution to the blog posts that I based this guide on:

Hardware

I have four disks in my HP MicroServer:

  • Disk 1: Operating System - 3.5’ 250GB 7200RPM HDD
  • Disk 2: Operating System - 2.5’ 200GB 7200RPM HDD
  • Disk 3: Data - 3.5’ 4TB NAS HDD
  • Disk 4: Data - 3.5’ 4TB NAS HDD

The first two disks will hold the base operating system and maybe the virtual machine operating system images. The data disks will be for data only.

The storage system layers will look like this:

| Filesystem (eg. ext4) |
| LVM                   |
| LUKS Crypto           |
| Linux Software RAID 1 |
| Physical Hard Disk    |

Above the physical block layer, we’ll put a Linux software RAID. The first RAID 1 will span disks 1 and 2 and a second RAID 1 will span the data disks (disks 3 and 4).

And on top of the software RAID will be the encryption layer. Why not the other way round? Because otherwise we would have two crypto devices instead of one, and the CPU would have to encrypt/decrypt any write/read operation twice.

This thread on the dm-crypt list discusses the two options.

Software

We are going to use grml, a Debian-based rescue/admin live distribution, to install the system.

So after downloading grml and booting the live CD, let’s start with becoming root:

sudo su -

Initialize the disks with random data

We will start writing random data to the two operating system disks.

badblocks -c 10240 -w -t random -s -v /dev/sda
badblocks -c 10240 -w -t random -s -v /dev/sdb

This may take a very long time, depending on how big your disks are.

Partitioning the OS disks

root@grml ~ # parted /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt                                                      
(parted) mkpart primary 2048s 4095s
(parted) set 1 bios_grub on                                               
(parted) name 1 "BIOS Boot Partition"                                     
(parted) mkpart primary 4096s 100%                                        
(parted) set 2 raid on
(parted) name 2 "SW-RAID / Linux"                                         
(parted) quit                                                             
Information: You may need to update /etc/fstab.

Copy the the partition table from the first disk to the second:

root@grml ~ # sgdisk -R=/dev/sdb /dev/sda
The operation has completed successfully.

Set new UUIDs on /dev/sdb:

root@grml ~ # sgdisk -G /dev/sdb
The operation has completed successfully.

RAID Mirror Setup

root@grml ~ # mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 /dev/sda2 /dev/sdb2
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 10474496K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

LUKS Crypto Setup

We use aes-xts as XTS works especially well for encrypting filesystems.
The keysize of 512 is actually 256, because XTS splits the key in half.
Because we use sha512 instead of sha1, we need to increase the time for the hash iterations.
Also, we have to use /dev/random instead of /dev/urandom, as urandom does not stop giving data if entropy gets low.

cryptsetup -v --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/md0

Now let’s open the crypto device:

cryptsetup luksOpen /dev/md0 cryptomd0

LVM Setup

Let’s create a physical volume and a volume group:

root@grml ~ # pvcreate /dev/mapper/cryptomd0
  Physical volume "/dev/mapper/cryptomd0" successfully created
root@grml ~ # vgcreate system /dev/mapper/cryptomd0
  Volume group "system" successfully created

Now the logical volumes. Be sure to ajust the sizes of the volumes to fit your system:

root@grml ~ # lvcreate -n swap -L1G system
  Logical volume "swap" created
root@grml ~ # lvcreate -n root -L6G system
  Logical volume "root" created

Create the file systems

root@grml ~ # mkfs.ext4 /dev/system/root
mke2fs 1.42.9 (4-Feb-2014)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
393216 inodes, 1572864 blocks
78643 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1610612736
48 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
  32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
root@grml ~ # mkswap -f /dev/system/swap
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=a44ea90d-72b4-4d2c-864f-70e9d2218651

Preparing for installation

root@grml ~ # mkdir /mnt/root
root@grml ~ # mount /dev/system/root /mnt/root

Installation

grml-debootstrap --target /mnt/root --password YOUR_PASSWORD --hostname YOUR_HOSTNAME --release jessie

Finishing the installation

Let’s get into our new installation:

root@grml ~ # grml-chroot /mnt/root /bin/bash
Writing /etc/debian_chroot ...
(YOUR_HOSTNAME)root@grml:/#
(rna)root@grml:~# apt-get install console-setup

Edit /etc/fstab to look like:

/dev/system/root  / auto    defaults,errors=remount-ro  0 1
/dev/system/swap  none  swap    sw                      0 0
proc      /proc proc    defaults                        0 0

Edit /etc/crypttab to look like:

cryptomd0 /dev/md0 none luks
(YOUR_HOSTNAME)root@grml:~# echo GRUB_CRYPTODISK_ENABLE=y >> /etc/default/grub
(YOUR_HOSTNAME)root@grml:~# echo 'GRUB_PRELOAD_MODULES="lvm cryptodisk mdraid1x"' >> /etc/default/grub
(YOUR_HOSTNAME)root@grml:/# grub-install /dev/sda
Installation finished. No error reported.
(YOUR_HOSTNAME)root@grml:/# grub-install /dev/sdb
Installation finished. No error reported.
(YOUR_HOSTNAME)root@grml:/# update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-3.14-1-amd64
df: Warning: cannot read table of mounted file systems
(YOUR_HOSTNAME)root@grml:~# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.14-1-amd64
Found initrd image: /boot/initrd.img-3.14-1-amd64
done

Prepare for reboot

(YOUR_HOSTNAME)root@grml:~# exit
exit
grml-chroot /mnt/root /bin/bash  9.21s user 2.43s system 0% cpu 21:20.00 total
root@grml ~ # umount /mnt/root

Reboot

root@grml ~ # reboot