Bug#1022199: apt: certificate verification fails after adding custom root certificates through ca-certificates

To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: Bug#1022199: apt: certificate verification fails after adding custom root certificates through ca-certificates
From: Marc Riudalbas Clemente <marc.riudalbas.clemente@aiticon.com>
Date: Fri, 21 Oct 2022 22:05:45 +0200
Message-id: <[🔎] 1967d7f0-8a94-d7ee-687f-5e745e10f890@aiticon.com>
Reply-to: Marc Riudalbas Clemente <marc.riudalbas.clemente@aiticon.com>, 1022199@bugs.debian.org

Package: apt
Version: 2.5.3+b1
Severity: important
X-Debbugs-Cc: marc.riudalbas.clemente@aiticon.com

Dear Maintainer,

*** What led up to the situation?

We need to install CA certificates to our servers so we can access to
services which are running with certificates singed by our CA. If we
dont, we can not use them. Example:

$ docker login dockerregistry.aiticon.net
Username: ait-docker
Password:
Error response from daemon: Get "https://dockerregistry.aiticon.net/v2/": x509: certificate signed by unknown authority

After installing custom root certificates through ca-certificates, apt
complains about other certificates (when pulling packages through https
sources.lists):

This is what we have done so far:

* We installed our CA and its intermediate in a fresh Vanilla-Bullseye
this way:

* Create our own directory to store CA and intermediate:
$ sudo mkdir -p /usr/share/ca-certificates/aiticon.net

* saved the certificates files under the created folder:
$ cat <<_EOF_ |sudo tee /usr/share/ca-certificates/aiticon.net/Aiticon_Trust_PKI_intermediate_east.crt
-----BEGIN CERTIFICATE-----
XXXXX
-----END CERTIFICATE-----

* Did the same for the following files
/usr/share/ca-certificates/aiticon.net/Aiticon_Trust_PKI_root.crt
/usr/share/ca-certificates/aiticon.net/Aiticon_Trust_PKI_intermediate_west.crt
/usr/share/ca-certificates/aiticon.net/Aiticon_Trust_PKI_intermediate_east.crt

* The certificates are there:
$ ls -lh
total 12K
-rw-r--r-- 1 root root 2,1K 18. Okt 13:46 Aiticon_Trust_PKI_intermediate_east.crt
-rw-r--r-- 1 root root 2,1K 18. Okt 13:45 Aiticon_Trust_PKI_intermediate_west.crt
-rw-r--r-- 1 root root 2,1K 18. Okt 13:45 Aiticon_Trust_PKI_root.crt

* Added the following lines in /etc/ca-certificates.conf
$ cat <<_EOF_ |sudo tee --append /etc/ca-certificates.conf
aiticon.net/Aiticon_Trust_PKI_root.crt
aiticon.net/Aiticon_Trust_PKI_intermediate_west.crt
aiticon.net/Aiticon_Trust_PKI_intermediate_east.crt
_EOF_

* Updated the CA Certificates:
$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
3 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

* After doing this, we can successfully use our services with the
installed signed certificates:

* Restart the docker service so it notices the are new CAs in the system
$ sudo systemctl restart docker

* Try docker login again
$ docker login dockerregistry.aiticon.net
Username: ait-docker
Password:
WARNING! Your password will be stored unencrypted in /home/aiticon/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

* And now, when executing a simple apt-get update, the certificate
verification for other https lists fail. THIS HAPPENS WITH ALL sources.list,
not only with docker (this is only an example):
$ sudo apt update
Err:3 https://download.docker.com/linux/debian bullseye InRelease
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 13.225.78.127 443]

*** What exactly did you do (or not do) that was effective (or
innefective)?

* Deleting again the CAs from /etc/ca-certificates.conf and running
update-ca-certificates again leads to the situation from the
beggining:

- apt works propperly, even with https source lists
- we cant use our services, because the CA is not known in the system

*** What outcome did you expect instead?

* We expected to be able to use all our services running with our CA, because we installed the CA in our system.
* We expected that the command apt update updates the package lists correctly.

-- Package-specific info:

-- apt-config dump --

APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Sandbox "";
APT::Sandbox::User "_apt";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-.*";
APT::VersionedKernelPackages:: "kfreebsd-.*";
APT::VersionedKernelPackages:: "gnumach-.*";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
APT::Never-MarkAuto-Sections "";
APT::Never-MarkAuto-Sections:: "metapackages";
APT::Never-MarkAuto-Sections:: "contrib/metapackages";
APT::Never-MarkAuto-Sections:: "non-free/metapackages";
APT::Never-MarkAuto-Sections:: "restricted/metapackages";
APT::Never-MarkAuto-Sections:: "universe/metapackages";
APT::Never-MarkAuto-Sections:: "multiverse/metapackages";
APT::Move-Autobit-Sections "";
APT::Move-Autobit-Sections:: "oldlibs";
APT::Move-Autobit-Sections:: "contrib/oldlibs";
APT::Move-Autobit-Sections:: "non-free/oldlibs";
APT::Move-Autobit-Sections:: "restricted/oldlibs";
APT::Move-Autobit-Sections:: "universe/oldlibs";
APT::Move-Autobit-Sections:: "multiverse/oldlibs";
APT::Architectures "";
APT::Architectures:: "amd64";
APT::Compressor "";
APT::Compressor::. "";
APT::Compressor::.::Name ".";
APT::Compressor::.::Extension "";
APT::Compressor::.::Binary "";
APT::Compressor::.::Cost "0";
APT::Compressor::zstd "";
APT::Compressor::zstd::Name "zstd";
APT::Compressor::zstd::Extension ".zst";
APT::Compressor::zstd::Binary "false";
APT::Compressor::zstd::Cost "60";
APT::Compressor::lz4 "";
APT::Compressor::lz4::Name "lz4";
APT::Compressor::lz4::Extension ".lz4";
APT::Compressor::lz4::Binary "false";
APT::Compressor::lz4::Cost "50";
APT::Compressor::gzip "";
APT::Compressor::gzip::Name "gzip";
APT::Compressor::gzip::Extension ".gz";
APT::Compressor::gzip::Binary "gzip";
APT::Compressor::gzip::Cost "100";
APT::Compressor::gzip::CompressArg "";
APT::Compressor::gzip::CompressArg:: "-6n";
APT::Compressor::gzip::UncompressArg "";
APT::Compressor::gzip::UncompressArg:: "-d";
APT::Compressor::xz "";
APT::Compressor::xz::Name "xz";
APT::Compressor::xz::Extension ".xz";
APT::Compressor::xz::Binary "xz";
APT::Compressor::xz::Cost "200";
APT::Compressor::xz::CompressArg "";
APT::Compressor::xz::CompressArg:: "-6";
APT::Compressor::xz::UncompressArg "";
APT::Compressor::xz::UncompressArg:: "-d";
APT::Compressor::bzip2 "";
APT::Compressor::bzip2::Name "bzip2";
APT::Compressor::bzip2::Extension ".bz2";
APT::Compressor::bzip2::Binary "bzip2";
APT::Compressor::bzip2::Cost "300";
APT::Compressor::bzip2::CompressArg "";
APT::Compressor::bzip2::CompressArg:: "-6";
APT::Compressor::bzip2::UncompressArg "";
APT::Compressor::bzip2::UncompressArg:: "-d";
APT::Compressor::lzma "";
APT::Compressor::lzma::Name "lzma";
APT::Compressor::lzma::Extension ".lzma";
APT::Compressor::lzma::Binary "xz";
APT::Compressor::lzma::Cost "400";
APT::Compressor::lzma::CompressArg "";
APT::Compressor::lzma::CompressArg:: "--format=lzma";
APT::Compressor::lzma::CompressArg:: "-6";
APT::Compressor::lzma::UncompressArg "";
APT::Compressor::lzma::UncompressArg:: "--format=lzma";
APT::Compressor::lzma::UncompressArg:: "-d";
Dir "/";
Dir::State "var/lib/apt";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::extended_states "extended_states";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::netrc "auth.conf";
Dir::Etc::netrcparts "auth.conf.d";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Etc::preferencesparts "preferences.d";
Dir::Etc::trusted "trusted.gpg";
Dir::Etc::trustedparts "trusted.gpg.d";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::solvers "";
Dir::Bin::solvers:: "/usr/lib/apt/solvers";
Dir::Bin::planners "";
Dir::Bin::planners:: "/usr/lib/apt/planners";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Bin::gzip "/bin/gzip";
Dir::Bin::bzip2 "/bin/bzip2";
Dir::Bin::xz "/usr/bin/xz";
Dir::Bin::lz4 "/usr/bin/lz4";
Dir::Bin::zstd "/usr/bin/zstd";
Dir::Bin::lzma "/usr/bin/xz";
Dir::Media "";
Dir::Media::MountPath "/media/cdrom";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
Dir::Log::History "history.log";
Dir::Log::Planner "eipp.log.xz";
Dir::Ignore-Files-Silently "";
Dir::Ignore-Files-Silently:: "~$";
Dir::Ignore-Files-Silently:: "\.disabled$";
Dir::Ignore-Files-Silently:: "\.bak$";
Dir::Ignore-Files-Silently:: "\.dpkg-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.ucf-[a-z]+$";
Dir::Ignore-Files-Silently:: "\.save$";
Dir::Ignore-Files-Silently:: "\.orig$";
Dir::Ignore-Files-Silently:: "\.distUpgrade$";
Acquire "";
Acquire::AllowInsecureRepositories "0";
Acquire::AllowWeakRepositories "0";
Acquire::AllowDowngradeToInsecureRepositories "0";
Acquire::cdrom "";
Acquire::cdrom::mount "/media/cdrom";
Acquire::IndexTargets "";
Acquire::IndexTargets::deb "";
Acquire::IndexTargets::deb::Packages "";
Acquire::IndexTargets::deb::Packages::MetaKey "$(COMPONENT)/binary-$(ARCHITECTURE)/Packages";
Acquire::IndexTargets::deb::Packages::flatMetaKey "Packages";
Acquire::IndexTargets::deb::Packages::ShortDescription "Packages";
Acquire::IndexTargets::deb::Packages::Description "$(RELEASE)/$(COMPONENT) $(ARCHITECTURE) Packages";
Acquire::IndexTargets::deb::Packages::flatDescription "$(RELEASE) Packages";
Acquire::IndexTargets::deb::Packages::Optional "0";
Acquire::IndexTargets::deb::Translations "";
Acquire::IndexTargets::deb::Translations::MetaKey "$(COMPONENT)/i18n/Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatMetaKey "$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::ShortDescription "Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::Description "$(RELEASE)/$(COMPONENT) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb::Translations::flatDescription "$(RELEASE) Translation-$(LANGUAGE)";
Acquire::IndexTargets::deb-src "";
Acquire::IndexTargets::deb-src::Sources "";
Acquire::IndexTargets::deb-src::Sources::MetaKey "$(COMPONENT)/source/Sources";
Acquire::IndexTargets::deb-src::Sources::flatMetaKey "Sources";
Acquire::IndexTargets::deb-src::Sources::ShortDescription "Sources";
Acquire::IndexTargets::deb-src::Sources::Description "$(RELEASE)/$(COMPONENT) Sources";
Acquire::IndexTargets::deb-src::Sources::flatDescription "$(RELEASE) Sources";
Acquire::IndexTargets::deb-src::Sources::Optional "0";
Acquire::Changelogs "";
Acquire::Changelogs::URI "";
Acquire::Changelogs::URI::Origin "";
Acquire::Changelogs::URI::Origin::Debian "https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog";
Acquire::Changelogs::URI::Origin::Ubuntu "https://changelogs.ubuntu.com/changelogs/pool/@CHANGEPATH@/changelog";
Acquire::Changelogs::AlwaysOnline "";
Acquire::Changelogs::AlwaysOnline::Origin "";
Acquire::Changelogs::AlwaysOnline::Origin::Ubuntu "1";
Acquire::Languages "";
Acquire::Languages:: "en";
Acquire::Languages:: "none";
Acquire::CompressionTypes "";
Acquire::CompressionTypes::xz "xz";
Acquire::CompressionTypes::bz2 "bzip2";
Acquire::CompressionTypes::lzma "lzma";
Acquire::CompressionTypes::gz "gzip";
Acquire::CompressionTypes::lz4 "lz4";
Acquire::CompressionTypes::zst "zstd";
DPkg "";
DPkg::Path "/usr/sbin:/usr/bin:/sbin:/bin";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
Binary "apt-config";
Binary::apt "";
Binary::apt::APT "";
Binary::apt::APT::Color "1";
Binary::apt::APT::Cache "";
Binary::apt::APT::Cache::Show "";
Binary::apt::APT::Cache::Show::Version "2";
Binary::apt::APT::Cache::AllVersions "0";
Binary::apt::APT::Cache::ShowVirtuals "1";
Binary::apt::APT::Cache::Search "";
Binary::apt::APT::Cache::Search::Version "2";
Binary::apt::APT::Cache::ShowDependencyType "1";
Binary::apt::APT::Cache::ShowVersion "1";
Binary::apt::APT::Get "";
Binary::apt::APT::Get::Upgrade-Allow-New "1";
Binary::apt::APT::Get::Update "";
Binary::apt::APT::Get::Update::InteractiveReleaseInfoChanges "1";
Binary::apt::APT::Cmd "";
Binary::apt::APT::Cmd::Show-Update-Stats "1";
Binary::apt::APT::Cmd::Pattern-Only "1";
Binary::apt::APT::Keep-Downloaded-Packages "0";
Binary::apt::DPkg "";
Binary::apt::DPkg::Progress-Fancy "1";
Binary::apt::DPkg::Lock "";
Binary::apt::DPkg::Lock::Timeout "-1";
CommandLine "";
CommandLine::AsString "apt-config dump";

-- (no /etc/apt/preferences present) --

-- (no /etc/apt/preferences.d/* present) --

-- /etc/apt/sources.list --

# deb cdrom:[Debian GNU/Linux 11.4.0 _Bullseye_ - Official amd64 NETINST 20220709-10:31]/ bullseye main

#deb cdrom:[Debian GNU/Linux 11.4.0 _Bullseye_ - Official amd64 NETINST 20220709-10:31]/ bullseye main

deb http://deb.debian.org/debian bullseye main
deb-src http://deb.debian.org/debian bullseye main

deb http://security.debian.org/debian-security bullseye-security main
deb-src http://security.debian.org/debian-security bullseye-security main

# This system was installed using small removable media
# (e.g. netinst, live or single CD). The matching "deb cdrom"
# entries were disabled at the end of the installation process.
# For information about how to configure apt package sources,
# see the sources.list(5) manual.

-- /etc/apt/sources.list.d/docker.list --

deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian   bullseye stable

-- System Information:
Debian Release: 11.4
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-17-amd64 (SMP w/4 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages apt depends on:
ii adduser                 3.118
ii debian-archive-keyring 2021.1.1
ii gpgv                    2.2.27-2+deb11u2
ii libapt-pkg6.0           2.5.3+b1
ii libc6                   2.35-3
ii libgcc-s1               10.2.1-6
ii libgnutls30             3.7.8-2
ii libseccomp2             2.5.1-1+deb11u1
ii libstdc++6              12.2.0-3
ii libsystemd0             247.3-7

Versions of packages apt recommends:
ii ca-certificates 20210119

Versions of packages apt suggests:
pn apt-doc         <none>
ii aptitude        0.8.13-3
pn dpkg-dev        <none>
ii gnupg           2.2.27-2+deb11u2
ii gnupg2          2.2.27-2+deb11u2
pn powermgmt-base <none>

-- no debconf information

-- 
aiticon GmbH
Stephanstraße 1
60313 Frankfurt am Main

t. +49 69 795 83 83-0
f. +49 69 795 83 83-28

Geschäftsführer: Matthias Herlitzius
Amtsgericht Frankfurt am Main · HRB 79310
USt.-ID-Nr.: DE 218319776

Reply to:

Follow-Ups:
- Processed: Re: Bug#1022199: apt: certificate verification fails after adding custom root certificates through ca-certificates
  - From: "Debian Bug Tracking System" <owner@bugs.debian.org>
- Bug#1022199: apt: certificate verification fails after adding custom root certificates through ca-certificates
  - From: Julian Andres Klode <jak@debian.org>

Prev by Date: Bug#1020285: marked as done (apt-get, version apt 2.5.2 (i386))
Next by Date: apt 2.4.8 (amd64) 503 Service Unavailable [IP: 91.189.91.49 80]
Previous by thread: Bug#1020285: marked as done (apt-get, version apt 2.5.2 (i386))
Next by thread: Processed: Re: Bug#1022199: apt: certificate verification fails after adding custom root certificates through ca-certificates
Index(es):
- Date
- Thread