On Wed, Nov 03, 2021 at 08:53:15PM +0100, Paul Gevers wrote: > On 03-11-2021 16:45, Julian Andres Klode wrote: > > There is some software "parsing" sources.list on its own, most of that > > is better served by `apt-get indextargets` (and for downloading stuff > > based on the urls, `apt-helper download-file`, such that it respects > > proxies and supports all transports users may use in sources.list) > > Like autopkgtest. When I was working on it to support Debian's migration > testing, I looked at python-apt and because that didn't support it, > stopped thinking. With indextargets and download-file I guess we could > work on it again. When were those introduced? Ubuntu needs it on old > releases so before autopkgtest can change it, we'd need support for a while. `apt-get indextargets` is from 2015 and a part of the acquire-additional- files feature used mainly by apt-file and appstream to have apt download files it isn't using itself, so those tools don't have to implement it. The job of indextargets is it mostly to give access to metadata (and crucially filenames) for those previously configured and hopefully now downloaded files. apt-file e.g. asks for the Contents files in this way to avoid exposing file naming logic and location to other tools. So, for the filenames of all (downloaded) Packages files: apt-get indextargets --format '$(FILENAME)' 'Identifier: Packages' (the default output is deb822 stanzas you could grep with more powerful tools than the simple inbuilt line-based filter) Note that you either have to implement opening compressed files yourself or use `/usr/lib/apt/apt-helper cat-file`. That was historically the most common reason to fiddle with sources.list parsing, hence Julian referring to it, but this seems not what autpkgtest is aiming for. On a casual look (well, grep) I see only: * lib/adt_testbed.py apt-pocket codepath seems to want to construct new sources.list entries based on existing ones. That should be possible with some indextargets busywork in general, but I am not completely sure what is going on here and the gymnastics should be similar to… * setup-commands/setup-testbed tries to find the mirror (and release) used for target distribution based on your current system. I am a bit surprised that works actually… Anyway, the later could perhaps be implement with: apt-get indextargets --format '$(ORIGIN)|$(REPO_URI)' | sort -u | \ grep -i -e "^$(. /etc/os-release; echo "$ID")" | cut -d'|' -f 2 (That is one line only for posterity – as you see, I am trying to fix the too general search by checking against Origin as defined by the Release file of a repository, but that would need work still to eliminate same-origin-but-different-repo cases) Parsing of the sources files is not really indextargets job through, so it might not always work for that task: It e.g. doesn't work if the data files are not on disk which might or might not be okay for you (there is 'guess' mode, but that of course has no metadata extracted from the Release files – the Origin I was using above). The apt family doesn't really have a publicly exposed way of reasoning about sources.list (or .sources) files and I am not quite sure it really can as subtil differences between repositories make it hard to give them all a common interface which makes sense. (I will probably be proven wrong by Julian though.) Like, for example, if stable is in the sources, make sure there is also updates and security there and/or add them. What for Debian are three distinct repositories might for others very well be components. Assuming you even know which line refers to Debian: I was using Origin above for this task as we can't really guess based on the URI. And even then… that logic above finds the tor+mirror+file source I am using, that won't work for autopkgtest, but I am special and this just a default fallback, so I might be thinking way too much about it… Anyway, if you have specific needs/questions feel free to ask on deity@ or #debian-apt. I am sure we will work something out even if in this case it might very well be new code nobody really uses for years (as is common in apt land – backward compat be damned 😉). Best regards David Kalnischkies
Attachment:
signature.asc
Description: PGP signature