Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=1.2.3 2015-12-19T22:04:34Z 2015-12-19T22:04:34Z David Kalnischkies david@kalnischkies.de 2015-12-15T16:20:26Z urn:sha1:3abb6a6a1e485b3bc899b64b0a1b7dc2db25a9c2 This doesn't allow all tests to run cleanly, but it at least allows to write tests which could run successfully in such environments. Git-Dch: Ignore 2015-11-04T17:42:27Z David Kalnischkies david@kalnischkies.de 2015-10-28T13:38:49Z urn:sha1:1dd20368486820efb6ef4476ad739e967174bec4 Based on a discussion with Niels Thykier who asked for Contents-all this implements apt trying for all architecture dependent files to get a file for the architecture all, which is treated internally now as an official architecture which is always around (like native). This way arch:all data can be shared instead of duplicated for each architecture requiring the user to download the same information again and again. There is one problem however: In Debian there is already a binary-all/ Packages file, but the binary-any files still include arch:all packages, so that downloading this file now would be a waste of time, bandwidth and diskspace. We therefore need a way to decide if it makes sense to download the all file for Packages in Debian or not. The obvious answer would be a special flag in the Release file indicating this, which would need to default to 'no' and every reasonable repository would override it to 'yes' in a few years time, but the flag would be there "forever". Looking closer at a Release file we see the field "Architectures", which doesn't include 'all' at the moment. With the idea outlined above that 'all' is a "proper" architecture now, we interpret this field as being authoritative in declaring which architectures are supported by this repository. If it says 'all', apt will try to get all, if not it will be skipped. This gives us another interesting feature: If I configure a source to download armel and mips, but it declares it supports only armel apt will now print a notice saying as much. Previously this was a very cryptic failure. If on the other hand the repository supports mips, too, but for some reason doesn't ship mips packages at the moment, this 'missing' file is silently ignored (= that is the same as the repository including an empty file). The Architectures field isn't mandatory through, so if it isn't there, we assume that every architecture is supported by this repository, which skips the arch:all if not listed in the release file. 2015-09-15T08:16:09Z David Kalnischkies david@kalnischkies.de 2015-09-14T22:33:12Z urn:sha1:6c0765c096ffb4df14169236c865bbb2b10974ae This allows running tests in parallel. Git-Dch: Ignore 2015-09-14T13:22:18Z David Kalnischkies david@kalnischkies.de 2015-09-10T17:00:51Z urn:sha1:95278287f4e1eeaf5d96749d6fc9bfc53fb400d0 Our error reporting is historically grown into some kind of mess. A while ago I implemented stacking for the global error which is used in this commit now to wrap calls to functions which do not report (all) errors via return, so that only failures in those calls cause a failure to propergate down the chain rather than failing if anything (potentially totally unrelated) has failed at some point in the past. This way we can avoid stopping the entire acquire process just because a single source produced an error for example. It also means that after the acquire process the cache is generated – even if the acquire process had failures – as we still have the old good data around we can and should generate a cache for (again). There are probably more instances of this hiding, but all these looked like the easiest to work with and fix with reasonable (aka net-positive) effects. 2015-08-10T15:27:59Z David Kalnischkies david@kalnischkies.de 2015-08-09T17:01:49Z urn:sha1:0efb29eb36184bbe6de7b1013d1898796d94b171 I never understood why there is an extra newline in those messages, so now is as good time as any to drop them. Lets see if someone complains with a good reason to keep it… 2015-06-15T21:35:55Z David Kalnischkies david@kalnischkies.de 2015-06-15T14:41:43Z urn:sha1:1eb1836f4b5397497bd34f0cf516e6e4e73117bf Again, consistency is the main sellingpoint here, but this way it is now also easier to explain that some files move through different stages and lines are printed for them hence multiple times: That is a bit hard to believe if the number is changing all the time, but now that it keeps consistent. 2015-06-09T10:57:36Z David Kalnischkies david@kalnischkies.de 2015-06-08T13:22:01Z urn:sha1:8d041b4f4f353079268039dcbfd8b5e575196b66 If we have a file on disk and the hashes are the same in the new Release file and the old one we have on disk we know that if we ask the server for the file, we will at best get an IMS hit – at worse the server doesn't support this and sends us the (unchanged) file and we have to run all our checks on it again for nothing. So, we can save ourselves (and the servers) some unneeded requests if we figure this out on our own. 2015-06-09T10:57:35Z David Kalnischkies david@kalnischkies.de 2015-06-06T10:28:00Z urn:sha1:448c38bdcd72b52f11ec5f326f822cf57653f81c Having every item having its own code to verify the file(s) it handles is an errorprune process and easy to break, especially if items move through various stages (download, uncompress, patching, …). With a giant rework we centralize (most of) the verification to have a better enforcement rate and (hopefully) less chance for bugs, but it breaks the ABI bigtime in exchange – and as we break it anyway, it is broken even harder. It shouldn't effect most frontends as they don't deal with the acquire system at all or implement their own items, but some do and will need to be patched (might be an opportunity to use apt on-board material). The theory is simple: Items implement methods to decide if hashes need to be checked (in this stage) and to return the expected hashes for this item (in this stage). The verification itself is done in worker message passing which has the benefit that a hashsum error is now a proper error for the acquire system rather than a Done() which is later revised to a Failed(). 2015-06-07T07:42:53Z David Kalnischkies david@kalnischkies.de 2015-05-19T08:40:55Z urn:sha1:58702f8563a443a7c6e66253b259c2488b877290 If we e.g. fail on hash verification for Packages.xz its highly unlikely that it will be any better with Packages.gz, so we just waste download bandwidth and time. It also causes us always to fallback to the uncompressed Packages file for which the error will finally be reported, which in turn confuses users as the file usually doesn't exist on the mirrors, so a bug in apt is suspected for even trying it… 2015-05-18T20:15:06Z David Kalnischkies david@kalnischkies.de 2015-05-18T20:15:06Z urn:sha1:6bf93605fdb8e858d3f0a79a124c1d39f760094d Valid-Until protects us from long-living downgrade attacks, but not all repositories have it and an attacker could still use older but still valid files to downgrade us. While this makes it sounds like a security improvement now, its a bit theoretical at best as an attacker with capabilities to pull this off could just as well always keep us days (but in the valid period) behind and always knows which state we have, as we tell him with the If-Modified-Since header. This is also why this is 'silently' ignored and treated as an IMSHit rather than screamed at the user as this can at best be an annoyance for attackers. An error here would 'regularily' be encountered by users by out-of-sync mirrors serving a single run (e.g. load balancer) or in two consecutive runs on the other hand, so it would just help teaching people ignore it. That said, most of the code churn is caused by enforcing this additional requirement. Crisscross from InRelease to Release.gpg is e.g. very unlikely in practice, but if we would ignore it an attacker could sidestep it this way.