Changing the backup approach

Here is my current backup strategy. Basically all devices backup to a NAS (which also hosts shared files). And periodically, backup the whole NAS to an offline PC:

The "backup of backup" setup has a few downsides:

• The work of digging out and connecting the old PC to the network periodically. Additional hardware / software maintenance.
• The software RAID 5 needs multiple HDDs, usually retired hardware previously used in the NAS. High risk of failure.

An alternative would be using external USB HDDs for the "backup of backup":

First attempt:

• Use external HDDs via USB/eSATA for the "backup of backup". Rotate between two external HDDs to minimize the chance of single point of failure
• Use veracrypt to encrypt the whole external HDD device
• Within the encrypted volume, use BTRFS and mount with compression. Can also use the snapshot function of BTRFS if necessary
• Sadly the NAS is running a very old version of Synology software and it doesn't support veracrypt or BTRFS. Will need to do this on a desktop PC.

While BTRFS do compression and snapshots, will need to pay attention on the disk usage. It is inefficient and difficult to re-balance if the free space is low (the main reason why I gave up on using BTRFS on boot partition even when it is the default for OpenSUSE). Should probably do a re-balance after every rsync.

Preliminary results:

• Only less than 1/10th of the files are actually compressed. Probably because the NAS manly stores photos, movies, and backup archives... in which already in compressed format.

~$ sudo compsize /mnt/backup
Processed 117244 files, 1223867 regular extents (1223867 refs), 17472 inline.
Type       Perc     Disk Usage   Uncompressed Referenced   
TOTAL       95%      1.5T         1.6T         1.6T        
none       100%      1.5T         1.5T         1.5T        
zstd        22%       22G         103G         103G

• Encounter RAID read errors and veracrypt issues. May need to think about doing checksum after backup

ata7.00: exception Emask 0x0 SAct 0xff800c SErr 0x0 action 0x0

ata7.00: irq_stat 0x40000008

ata7.00: failed command: READ FPDMA QUEUED

ata7.00: cmd 60/08:78:88:92:ed/00:00:73:00:00/40 tag 15 ncq dma 4096 in

                                    res 51/40:01:8f:92:ed/00:00:73:00:00/40 Emask 0x409 (media error) <F>

ata7.00: status: { DRDY ERR }

ata7.00: error: { UNC }

ata7.00: ATA Identify Device Log not supported

ata7.00: ATA Identify Device Log not supported

ata7.00: configured for UDMA/133

sd 6:0:0:0: [sde] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s

sd 6:0:0:0: [sde] tag#15 Sense Key : Medium Error [current] 

sd 6:0:0:0: [sde] tag#15 Add. Sense: Unrecovered read error - auto reallocate failed

sd 6:0:0:0: [sde] tag#15 CDB: Read(10) 28 00 73 ed 92 88 00 00 08 00

blk_update_request: I/O error, dev sde, sector 1944949391 op 0x0:(READ) flags 0x4000 phys_seg 1 prio class 0

ata7: EH complete

md/raid:md0: read error corrected (8 sectors at 1944947336 on sde1)

To be continued...

Some useful commands, assuming sdd is the external HDD:

# create veracrypt volume

sudo veracrypt --volume-type=normal -c /dev/sdd

# mount veracrypt volume
sudo veracrypt --filesystem=none --slot=1 /dev/sdd

# create BTRFS
sudo mkfs.btrfs -L backup /dev/mapper/veracrypt1

# mount BTRFS with compression
sudo mount -o compress=zstd:15 /dev/mapper/veracrypt1 /mnt/backup

# check BTRFS compression ratio
sudo compsize /mnt/backup

References:

https://wiki.archlinux.org/title/VeraCrypt

https://man.archlinux.org/man/btrfs.5#COMPRESSION