Worth
knowing if you use VirtualBox with Debian9 guests? :
The
rubyv3 guest is a fresh install of Debian9 from a CD including
graphics and intended as a workstation.
The rubyv4 guest is a
fresh install of Debian9 from a network install CD with no optional
software and intended to be built as a server.
The rubyv5 guest
is a fresh install of Ubuntu Server with no optional software.
All
are Virtualbox virtual machines.
The underlying hardware OS is
the same Debian9 with a 6TB array of five hard disks.
File Sizes: rubyv3(debian
w/s) rubyv4(debian srv) rubyv5(ubuntu srv)
System
use inside VM 3,071,064,000
1,189,448,000 1,554,552,000
reported
by 'ls -l'
.vdi size
4,942,987,264 109,515,374,592
4,632,608,768
.ova size
1,569,787,904 110,322,772,480
4,482,602,496
Why the difference in the .vdi and .ova
file sizes for rubyv4, as compared to the files for rubyv3 and
ruby5? All the file sizes for rubyv3 are reasonable, but
...
... requiring a 110GB .ova backup file for a 1.2GB install
for rubyv4 is not!
The reason for this appears to lie in
the Debian install process and its interaction with Virtualbox.
rubyv3 was installed onto an ext4 partition; rubyv4 was installed
onto an encrypted LVM partition where the dynamic
110GB .vdi file was zeroed(with random numbers?) to its full size.
Ubuntu server, using an encrypted LVM, dynamic
100GB partition layout,
does not do the same(no zeroing of free space) but the compression
algorithm achieves next to nothing in producing the .ova file.
The
desire to completely
'zero' a hardware disk
before installing a new operating system is sensible but doing the
same to a dynamic
virtual disk that
is encrypted is
questionable.
Compacting Virtualbox .vdi files:
>
dd if=/dev/zero of=zerofillfile status=progress bs=1M
until the
disk fills, then
>
rm zerofillfile
and all free space is zeroed. Do not use
'if=/dev/urandom' in the 'dd' command!!
>
VBoxManage modifyhd <path/file.vdi>
--compact
https://unix.stackexchange.
cat
/etc/crypttab issue_discards = 1
sda3_crypt
UUID=[...] none luks,discard
cat /etc/lvm/lvm.conf
#
[...]
devices {
# [
... ]
issue_discards = 1
# [ ... ]
}
# [...]
To test the setup I
just did 'sudo fstrim -v /' which resulted in /: [...] bytes were
trimmed
Doing this again resulted in /: 0 bytes were trimmed
which seems to make sense and indicated that TRIM seems to
work.However then I did this test:
dd if=/dev/urandom
of=tempfile count=100 bs=512k oflag=direct
Note
'urandom'
https://www.virtualbox.org/
'The
--compact option, can be used to compact disk images, i.e. remove
blocks that only contains zeroes[and not the output of
'urandom'!!].'
'compacting is currently only available for
VDI images. A similar effect can be achieved by zeroing[not
urandom!!] out free blocks and then cloning the disk to any other
dynamically allocated format.'
In my tests I failed to
achieve a .vdi compaction, including by cloning.
Further
tests :
File Sizes:
rubyv4(debian srv)
8GB virtualdisk
System use inside VM
876,928,000
reported by 'ls -l'
.vdi size
8,498,708,480
.ova size
8,508,642,304
resize to 20GB using
VBoxManage modifyhd ruby4.vdi --resize 20000
Before
starting
8,498,708,480 reported as 19.53GB in VB
After run and
shutdown 8,498,708,480
File
Sizes: rubyv6(debian
srv)
8GB virtualdisk
Debian install zeroing of free disk space halted
by escaping the zero procedure
System use inside VM
867,948,000
reported by 'ls -l'
.vdi size
1,809,842,176
.ova size
1,636,825,000
resize to 20GB using VBoxManage
modifyhd ruby6.vdi --resize 20000
Before starting
1,809,842,176 reported as 19.53GB in VB
After run and
shutdown 1,809,842,176 which is what it should
be
Conclusions :
1:After installing Debian9 to
a large dynamic Virtualbox virtual disk, the .vdi file goes to full
size as the disk is zeroed (with random numbers?) and will
not compress into the .ova file or clone.
2: Best to escape the
Debian install zeroing of free space or use Ubuntu server.
3: If
free space zeroing is required, use the dd command in the running
server, but it will take the .vdi file up to full size.
4:
Dynamic file expansion will negate the zeroing, unless the zeroing is
repeated.
5: Virtualbox .vdi
file compaction still
seems problematic.
Recomendation :
Add a clear
warning in the Debian install process about zeroing free space in
large Virtualbox virtual disks. The resulting
.vdi
file bloat is, for
current
purposes, irreversible.
####