Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=master 2017-07-26T17:07:56Z 2017-07-26T17:07:56Z David Kalnischkies david@kalnischkies.de 2017-07-26T16:35:42Z urn:sha1:f2f8e89f08cdf01c83a0b8ab053c65329d85ca90 Failing on too much data is good, but we can do better by checking for exact filesizes as we know with hashsums how large a file should be, so if we get a file which has a size we do not expect we can drop it directly, regardless of if the file is larger or smaller than what we expect which should catch most cases which would end up as hashsum errors later now a lot sooner. 2016-08-16T16:49:37Z David Kalnischkies david@kalnischkies.de 2016-08-11T16:24:35Z urn:sha1:d94b1d80d8326334d17f6a43061368e783b8e0aa If the server told us in a previous request that it isn't supporting Ranges with bytes via an Accept-Ranges header missing bytes, we don't try to formulate requests using Ranges. 2016-04-25T13:35:52Z David Kalnischkies david@kalnischkies.de 2016-04-07T15:48:17Z urn:sha1:742f67eaede80d2f9b3631d8697ebd63b8f95427 We have this situation in cases were parts of the transaction are refused (e.g. in a hashsum mismatch) and rerun the update (e.g. in the hope that we get a mirror which is synced this time). Previously we would ask the server with an if-range and in the best case recieve a 416 in response (less featureful server might end up giving us the entire file again or we get the wrong file this time giving us a hashsum mismatch…), which is a waste of time if we know already by checking the hashsums that we got the complete and correct file. 2015-12-19T22:04:34Z David Kalnischkies david@kalnischkies.de 2015-12-15T16:20:26Z urn:sha1:3abb6a6a1e485b3bc899b64b0a1b7dc2db25a9c2 This doesn't allow all tests to run cleanly, but it at least allows to write tests which could run successfully in such environments. Git-Dch: Ignore 2015-11-04T17:42:27Z David Kalnischkies david@kalnischkies.de 2015-10-28T13:38:49Z urn:sha1:1dd20368486820efb6ef4476ad739e967174bec4 Based on a discussion with Niels Thykier who asked for Contents-all this implements apt trying for all architecture dependent files to get a file for the architecture all, which is treated internally now as an official architecture which is always around (like native). This way arch:all data can be shared instead of duplicated for each architecture requiring the user to download the same information again and again. There is one problem however: In Debian there is already a binary-all/ Packages file, but the binary-any files still include arch:all packages, so that downloading this file now would be a waste of time, bandwidth and diskspace. We therefore need a way to decide if it makes sense to download the all file for Packages in Debian or not. The obvious answer would be a special flag in the Release file indicating this, which would need to default to 'no' and every reasonable repository would override it to 'yes' in a few years time, but the flag would be there "forever". Looking closer at a Release file we see the field "Architectures", which doesn't include 'all' at the moment. With the idea outlined above that 'all' is a "proper" architecture now, we interpret this field as being authoritative in declaring which architectures are supported by this repository. If it says 'all', apt will try to get all, if not it will be skipped. This gives us another interesting feature: If I configure a source to download armel and mips, but it declares it supports only armel apt will now print a notice saying as much. Previously this was a very cryptic failure. If on the other hand the repository supports mips, too, but for some reason doesn't ship mips packages at the moment, this 'missing' file is silently ignored (= that is the same as the repository including an empty file). The Architectures field isn't mandatory through, so if it isn't there, we assume that every architecture is supported by this repository, which skips the arch:all if not listed in the release file. 2015-06-09T10:57:36Z David Kalnischkies david@kalnischkies.de 2015-06-08T13:22:01Z urn:sha1:8d041b4f4f353079268039dcbfd8b5e575196b66 If we have a file on disk and the hashes are the same in the new Release file and the old one we have on disk we know that if we ask the server for the file, we will at best get an IMS hit – at worse the server doesn't support this and sends us the (unchanged) file and we have to run all our checks on it again for nothing. So, we can save ourselves (and the servers) some unneeded requests if we figure this out on our own. 2015-06-07T07:42:53Z David Kalnischkies david@kalnischkies.de 2015-05-19T08:40:55Z urn:sha1:58702f8563a443a7c6e66253b259c2488b877290 If we e.g. fail on hash verification for Packages.xz its highly unlikely that it will be any better with Packages.gz, so we just waste download bandwidth and time. It also causes us always to fallback to the uncompressed Packages file for which the error will finally be reported, which in turn confuses users as the file usually doesn't exist on the mirrors, so a bug in apt is suspected for even trying it… 2015-05-13T14:09:12Z David Kalnischkies david@kalnischkies.de 2015-05-13T14:09:12Z urn:sha1:8eafc759544298211cd0bfaa3919afc0fadd47d1 Not all servers we are talking to support If-Modified-Since and some are not even sending Last-Modified for us, so in an effort to detect such hits we run a hashsum check on the 'old' compared to the 'new' file, we got the hashes for the 'new' already for "free" from the methods anyway and hence just need to calculated the old ones. This allows us to detect hits even with unsupported servers, which in turn means we benefit from all the new hit behavior also here. 2015-05-11T22:30:16Z David Kalnischkies david@kalnischkies.de 2015-05-11T22:30:16Z urn:sha1:dcbb364fc69e1108b3fea3adb12a7ba83d9af467 If we have the expected hashes we can check with them if the file we have in partial we got a 416 for is the expected file. We detected this with same-size before, but not every server sends a good Content-Range header with a 416 response. 2015-04-18T23:13:10Z David Kalnischkies david@kalnischkies.de 2015-04-12T15:08:46Z urn:sha1:ba6b79bd0090077724fa1272ea4d3a31706fcd5a If we get a IMSHit for the Transaction-Manager (= the InRelease file or as its still supported fallback Release + Release.gpg combo) we can assume that every file we would queue based on this manager, but already have locally is current and hence would get an IMSHit, too. We therefore save us and the server the trouble and skip the queuing in this case. Beside speeding up repetative executions of 'apt-get update' this way we also avoid hitting hashsum errors if the indexes are in fact already updated, but the Release file isn't yet as it is the case on well behaving mirrors as Release files is updated last. The implementation is a bit harder than the theory makes it sound as we still have to keep reverifying the Release files (e.g. to detect now expired once to avoid an attacker being able to silently stale us) and have to handle cases in which the Release file hits, but some indexes aren't present (e.g. user added a new foreign architecture).