Bug#876352: marked as done (Package: installation-report for Debian9)

To: Holger Wansing <hwansing@mailbox.org>

Subject: Bug#876352: marked as done (Package: installation-report for Debian9)

From: "Debian Bug Tracking System" <owner@bugs.debian.org>

Date: Fri, 21 Aug 2020 03:05:03 +0000

Message-id: <[🔎] handler.876352.D876352.15979787172929.ackdone@bugs.debian.org>

References: <[🔎] 20200821045802.07acb7c9ded347064600db2d@mailbox.org> <CAOxdkMh3-=fC3S_j=wmz=iizBgxFCapj_WMQQeCsr6cmvoVBoA@mail.gmail.com>

Your message dated Fri, 21 Aug 2020 04:58:02 +0200 with message-id <[🔎] 20200821045802.07acb7c9ded347064600db2d@mailbox.org> and subject line Re: Mass-closing old installation-report bugs --- round 5 has caused the Debian Bug report #876352, regarding Package: installation-report for Debian9 to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) -- 876352: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876352 Debian Bug Tracking System Contact owner@bugs.debian.org with problems

--- Begin Message ---

To: submit@bugs.debian.org
Subject: Package: installation-report for Debian9
From: Givea DogaBone <paul.dodgshun@gmail.com>
Date: Thu, 21 Sep 2017 10:14:12 +0100
Message-id: <CAOxdkMh3-=fC3S_j=wmz=iizBgxFCapj_WMQQeCsr6cmvoVBoA@mail.gmail.com>

Package: installation-report

Worth knowing if you use VirtualBox with Debian9 guests? :

The rubyv3 guest is a fresh install of Debian9 from a CD including graphics and intended as a workstation.
The rubyv4 guest is a fresh install of Debian9 from a network install CD with no optional software and intended to be built as a server.
The rubyv5 guest is a fresh install of Ubuntu Server with no optional software.
All are Virtualbox virtual machines.
The underlying hardware OS is the same Debian9 with a 6TB array of five hard disks.

File Sizes:        rubyv3(debian w/s) rubyv4(debian srv) rubyv5(ubuntu srv)
System use inside VM   3,071,064,000       1,189,448,000      1,554,552,000

reported by 'ls -l'
.vdi size              4,942,987,264     109,515,374,592      4,632,608,768
.ova size              1,569,787,904     110,322,772,480      4,482,602,496

Why the difference in the .vdi and .ova file sizes for rubyv4, as compared to the files for rubyv3 and ruby5? All the file sizes for rubyv3 are reasonable, but ...
... requiring a 110GB .ova backup file for a 1.2GB install for rubyv4 is not!

The reason for this appears to lie in the Debian install process and its interaction with Virtualbox. rubyv3 was installed onto an ext4 partition; rubyv4 was installed onto an encrypted LVM partition where the dynamic 110GB .vdi file was zeroed(with random numbers?) to its full size. Ubuntu server, using an encrypted LVM, dynamic 100GB partition layout, does not do the same(no zeroing of free space) but the compression algorithm achieves next to nothing in producing the .ova file.

The desire to completely 'zero' a hardware disk before installing a new operating system is sensible but doing the same to a dynamic virtual disk that is encrypted is questionable.

Compacting Virtualbox .vdi files:
> dd if=/dev/zero of=zerofillfile status=progress bs=1M
until the disk fills, then
> rm zerofillfile
and all free space is zeroed. Do not use 'if=/dev/urandom' in the 'dd' command!!
> VBoxManage modifyhd <path/file.vdi> --compact

https://unix.stackexchange.
cat /etc/crypttab   issue_discards = 1
sda3_crypt UUID=[...] none luks,discard
cat /etc/lvm/lvm.conf
# [...]
devices {
      # [ ... ]
      issue_discards = 1
      # [ ... ]
   }
# [...]
To test the setup I just did 'sudo fstrim -v /' which resulted in /: [...] bytes were trimmed
Doing this again resulted in /: 0 bytes were trimmed which seems to make sense and indicated that TRIM seems to work.However then I did this test:
dd if=/dev/urandom of=tempfile count=100 bs=512k oflag=direct
Note 'urandom'

https://www.virtualbox.org/
'The --compact option, can be used to compact disk images, i.e. remove blocks that only contains zeroes[and not the output of 'urandom'!!].'

'compacting is currently only available for VDI images. A similar effect can be achieved by zeroing[not urandom!!] out free blocks and then cloning the disk to any other dynamically allocated format.'

In my tests I failed to achieve a .vdi compaction, including by cloning.

Further tests :
File Sizes:         rubyv4(debian srv)
                       8GB virtualdisk
System use inside VM     876,928,000
reported by 'ls -l'
.vdi size               8,498,708,480
.ova size               8,508,642,304
   resize to 20GB using VBoxManage modifyhd ruby4.vdi --resize 20000
Before starting         8,498,708,480 reported as 19.53GB in VB
After run and shutdown 8,498,708,480

File Sizes:         rubyv6(debian srv)
                       8GB virtualdisk
Debian install zeroing of free disk space halted by escaping the zero procedure
System use inside VM     867,948,000
reported by 'ls -l'
.vdi size               1,809,842,176
.ova size               1,636,825,000
   resize to 20GB using VBoxManage modifyhd ruby6.vdi --resize 20000
Before starting         1,809,842,176 reported as 19.53GB in VB
After run and shutdown 1,809,842,176   which is what it should be

Conclusions :
1:After installing Debian9 to a large dynamic Virtualbox virtual disk, the .vdi file goes to full size as the disk is zeroed (with random numbers?) and will not compress into the .ova file or clone.
2: Best to escape the Debian install zeroing of free space or use Ubuntu server.
3: If free space zeroing is required, use the dd command in the running server, but it will take the .vdi file up to full size.
4: Dynamic file expansion will negate the zeroing, unless the zeroing is repeated.
5: Virtualbox .vdi file compaction still seems problematic.

Recomendation :
Add a clear warning in the Debian install process about zeroing free space in large Virtualbox virtual disks. The resulting .vdi file bloat is, for current purposes, irreversible.

####

--- End Message ---

Reply to: