On Wed, 29 Jun 2011 01:42:10 +0100 Wookey <wookey@wookware.org> wrote: > +++ Neil Williams [2011-06-28 08:17 +0100]: > > On Tue, 28 Jun 2011 01:03:22 +0200 > > > Most of them have not been of particularly good quality (many > > haven't deserved to be part of Debian and I'm working on removing > > those, once alternatives become available). Most have relied on > > hacks upon hacks upon hacks. dpkg-cross is a classic example > > - it stinks and causes immense amounts of aggravation but it is what we > > have and until MultiArch delivers results there will be nothing better. > > dpkg-cross is a perfectly sensible idea, which now has a 10-yr > pedigree. SOme of it's better bits have been absorded into dpkg some > time ago. It has limitations, and thus so does the cross-compilation > system it implements, but it's silly to say 'it stinks'. I plan to > improve it to allow packages to supply autoconf cache data, rather > than have it all contained in dpkg-cross (which makes no sense), for > example. The idea of putting foreign files in a known location by converting natively built packages is the good part of the idea. The hack of doing this by renaming the *binary* package thereby creating a package which exists nowhere else is a rotten idea. Yes, it is/was the only way of doing it (probably) but that does not mean that it is defensible or anything other than a means to an end. It is time to move away. dpkg-cross is fine for small builds but, as Emdebian Crush proved, renaming the binary entirely trashes any dependency calculations more complex than those of busybox or the kernel. apt-cross tried to solve this cleverly, xapt tries to solve this stupidly. The only real solution is not to change the package name in the first place because that simply removes the -cross package from the entire dependency chain. Rebuilding that chain is hard - whether or not you try to compensate for packages which should not be of any interest to dpkg-cross. I know there are frequent complaints that xapt insists on preparing debhelper-armel-cross and other nonsense - those problems *only* exist because the behaviour of dpkg-cross in renaming the package is a truly horrible hack with disastrous consequences. We both agree that MultiArch is the way to fix these problems and that dpkg-cross gets retired as a consequence (slowly). MultiArch will bring huge benefits in resolving cross dependencies, principally because it does not involve renaming the package. It is a complex proposal (mainly as a result of *not* renaming the package and therefore requiring that the same package can be installed for multiple, incompatible, architectures) and there will be breakage but the objective of MultiArch and the proposed support for cross-building is sound. dpkg-cross, by comparison, is a ready for the knackers yard. > > Multistrap is just something which I hacked together because it was > > useful and which is now suffering from the results of feature creep. > > In what way? It's a very useful tool, and in some ways has a better > design than debootstrap. It is becoming quite popular as a result. I agree that working from the original goals, multistrap is better than debootstrap. It copes with multiple repositories, it handles --foreign without bloat and is architecture-agnostic as a result. In those important aspects, multistrap has fulfilled the original objectives very well. The problems are: 0: The hooks and the inherited machine:variant support - the two mechanisms don't sit well together. One needs to replace the other. I'm unconvinced that hooks are actually a good idea, or even particularly useful compared to the setupscript and configscript and the hooks themselves seem particularly unfriendly. 1: The duality (common to all bootstrap tools) of one tool which both creates both root filesystems to boot and chroots for building other packages is confusing and causes documentation duplication. 2: The manpage is too long, it does need to be split and I'm thinking that multistrap needs a man.5 and possibly a man.8 but the wiki and the website will remain important resources for levels of detail and examples which simply won't work in a manpage. 3: The unpacking algorithm will probably not work once a lot of MultiArch packages start coming through - there is outline support but it is almost completely theoretical and untested. 4: apt and dpkg are fast moving and multistrap needs to adapt frequently and quickly. Too much complexity in the script hinders the maintenance of the core functionality. Multistrap needs to return to KISS and go back to being a small tool which does a known set of tasks very well and leaves the rest to everything else. It stemmed from the idea of trying to let multistrap create the entire rootfs in a single call. For simple systems that can still work. The goal of doing that for every possible system is not achievable and multistrap needs to step back and be confined to what it does best. Again, MultiArch will come to a partial rescue here because it will make it easier to setup cross-building pbuilder chroots and that entire side of multistrap may actually disappear. Equally, getting cross-building toolchains into Debian will ease the problem as well. > > What is core to Emdebian are three things: > > > > 0: Getting Multi-Arch to a point where Emdebian Crush can be restarted > > as a cross-built binary-incompatible spin off Debian without coreutils > > and without perl. > > Actually IMHO 0 is getting Multiarch apt and dpkg to a point where they > correctly solve the 'determining and installing cross-dependencies' > problem, which is by far the biggest thing making cross-building > on Debian difficult and unreliable. > > That might be the same thing you said in different terms. I hope to > make some progress on that with Michael Vogt this week. I think we are looking for the same thing using different words, yes. You've described the fundamental detail upon which the larger issue of restarting Crush development depends. The apt and dpkg support is a precursor but not the only blocker - a lot of old mechanisms need to be rewritten and that can't start until we know how apt and dpkg are actually going to operate. > > Creating an -$arch-cross package is a stupid idea > > (due to the required dependency mangling) but it is the only way to get > > things done until MultiArch delivers. > > As stated above this is hyperbole at best. I disagree. If -$arch-cross wasn't such a bad hack you wouldn't have the same problems with xapt needing to be so damnably unintelligent. Renaming the package containing the foreign architecture files is the one fundamental blocker which prohibits writing a sane tool to handle dependencies across architectures. There is a very good reason why MultiArch does not do it that way and it is the principle reason why MultiArch is so complex. Allowing the same package to exist in the dpkg and apt data in multiple forms is key to resolving all of the problems with dependencies. dpkg-cross avoided the issue and is fundamentally broken as a direct result. MultiArch contends with the problem head on and is complex and will cause widespread breakage as a direct result. It hardly matters what files are contained in the various forms of the one package which are installed at any one time - that's down to Policy and path resolution for whatever tools need to use those files - the core of the problem has always been the data sources used by dpkg and apt. Solve that problem, as MultiArch must, and dependency resolution across arbitrary architecture barriers just ceases to be an issue. The only thing that matters is that the package name does not change. Nearly everything else about MultiArch flows from that requirement. Everything that is wrong with dpkg-cross results from ignoring that requirement. I know you already know all this, I'm just explaining why I still assert that whilst it is completely obvious that dpkg-cross had to be implemented the way it was, this does not excuse the fact that the method it uses is fundamentally the wrong approach. It is a horrible hack whose time has come, nearly. > > Perfection is the enemy of good. What we have is light years from > > perfection but it can be useful, it can work and people do find it > > useful. The cracks show up continually because the underlying methods > > are all hacks. Papering over the cracks with docs is NOT the answer, we > > have to fix the foundations and that means getting the hard work done > > of changing the infrastructure. It means MultiArch, it means Grip > > processing within Debian and it means throwing out our hacks. What we > > have cannot be perfected because it is built on a house of cards. We > > can only fix the foundations by having the time to work on the hard > > technical challenges which gave rise to the original hacks. We have to > > stop fixing the cop-outs and fix the real problems. > > Most of this I do agree with. Implementing cross-dependecy resolution > using multiarch status and build-depency qualifiers, rather than > fiddling with xdeb forever is currently the best use of my time, I > believe. Then working with package maintainers and upstreams to fix > cross-building problems (some of which are throny), and implemnting a > distributed cached-value provision mechansim should make a good > proportion of Debian cross-buildable in a reasonably rigorous fashion. > > I think continuous integration testing of that is also very important, > and I'm hoping to get that done in the next couple of months too. Yes, I'm aware of that and I'm glad it is being done. > > > Some references: > > > http://bugs.debian.org/630258 > > > http://bugs.debian.org/630406 > > > http://bugs.debian.org/631241 > > > http://bugs.debian.org/630314 > > > > None of which relate to Emdebian Grip, only to multistrap. Separate > > issue. I am looking at exactly which bits of multistrap need to be > > removed and how the docs will be reorganised but none of the bugs you > > quote above did anything to help that process except delay it. > > Neil, Yann provided some useful fixes in suggestions in all those > bugs, and you were generally dismissive. I'm not surprised he's > discouraged. There were earlier issues but things have improved in this thread, so I'm not going to drag those up again. It wasn't much fun at this end either. > > I do not want to be distracted into making large scale changes to > > multistrap just when it would be most beneficial to Emdebian to be > > working on the technical challenges of migrating the infrastructure of > > the actual package processing. > > Neil - you worry about the emdebian grip integration stuff, and I'll > worry about these minor doc and config fixes in multistrap. How about > that? Multistrap may well need work to handle MultiArch (particularly in the complex unpacking routines which populate /var/lib/dpkg/info) and that will complicate things. I need to keep one eye on multistrap but at a deeper level. It does (or soon will) require potentially disruptive changes and a lot of testing. This isn't a time to go tweaking minor doc details because the whole documentation will need a rewrite (and then a full re-translation). You've got enough to do getting MultiArch supporting cross building, moving tasks from my free time to your free time doesn't actually solve much... Let's leave the minor stuff to one side for now. I'm quite happy to deal with it in-the-round and do the full retranslation of all the affected packages. Just not before DebConf. (Possibly @ DebConf though.) -- Neil Williams ============= http://www.linux.codehelp.co.uk/
Attachment:
pgpvDe6d4jKdr.pgp
Description: PGP signature