Re: Bug#1023870: dpkg: Problems in buildds due to slow compression
Hi,
On Sun, 13 Nov 2022 at 02:00, Guillem Jover <guillem@debian.org> wrote:
>
> Err sorry, the patch was computing the used memory and not the truly
> available one! The updated patch should do better. :)
Given Aurélien's tests this is a bit redundant, but since I had the
building going I let it finish to add more data points...
I couldn't replicate the conditions without stopping buildds, but
tried in a similar machine to those of the buildds but with 8GB of RAM
and 8GB of swap (on nbd) plus some zram. I filled /dev/shm with files
from urandom, 7.5GB of them.
The machine kept sending those files to swap as it needed RAM for
building the kernel, so when it reached the compression phase there
was enough memory to use all threads (4), minimum ~1.3GB in the loop
deciding how many threads to use, for example:
dpkg-deb: building package 'ext4-modules-6.0.0-2-riscv64-di' in
'debian/.debhelper/scratch-space/build-ext4-modules-6.0.0-2-riscv64-di/ext4-modules-6.0.0-2-riscv64-di_6.0.6-2_riscv64.deb'.
- initial mt_options.threads=4
-- loop: mt_options.threads=4, mt_memusage=692517284, mt_memlimit=1357987840
- initial mt_options.threads=4
-- loop: mt_options.threads=4, mt_memusage=692517284, mt_memlimit=1355665408
This is with the default: "vm.swappiness = 60". With systems having
this value close to zero not sure if the results would be so
positive... but I think that at the point of compressing the resulting
packages, there had to be quite a lot of free memory from the previous
processes of the compiler and available now to use for compression.
So I am not sure if the patch would cover absolutely all corner cases,
but it think that it would help to solve the known problems in buildds
and it's much better than the current calculation of MemAvailable.
Cheers and thanks.
--
Manuel A. Fernandez Montecelo <manuel.montezelo@gmail.com>
Reply to: