Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=1.4_beta2 2016-11-24T23:15:13Z 2016-11-24T23:15:13Z David Kalnischkies david@kalnischkies.de 2016-11-13T01:29:46Z urn:sha1:5832913a49d4f7c75527264a935cc0ce00627f1d In ad9416611ab83f7799f2dcb4bf7f3ef30e9fe6f8 we fall back to asking the original mirror (e.g. a redirector) if we do not get the expected result. This works for the indexes, but patches are a different beast and much simpler. Adding this fallback code here seems like overkill as they are usually right along their Index file, so actually forward the relevant settings to the patch items which fixes pdiff support combined with a redirector and partial mirrors as in such a situation the pdiff patches would be 404 and the complete index would be downloaded. 2016-09-02T15:16:36Z Julian Andres Klode jak@debian.org 2016-06-15T21:13:43Z urn:sha1:2a440328ea19e9646a93f847dd9eff21e03ad16d Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff. 2016-08-17T05:55:46Z David Kalnischkies david@kalnischkies.de 2016-08-16T05:47:44Z urn:sha1:77e274f5ad23d79294f28ecc9868fc6f534214a4 In af81ab9030229b4ce6cbe28f0f0831d4896fda01 by-hash got implemented as a special compression type for our usual index files like Packages. Missing in this scheme was the special .diff/Index index file containing the info about individual patches for this index file. Deriving from the index file class directly we inherent the compression handling infrastructure and in this way also by-hash nearly for free. Closes: #824926 2016-08-10T21:19:44Z David Kalnischkies david@kalnischkies.de 2016-08-02T20:44:50Z urn:sha1:57401c48fadc0c78733a67294f9cc20a57e527c9 Having the detection handled in specific (http) workers means that a redirection loop over different hostnames isn't detected. Its also not a good idea have this implement in each method independently even if it would work 2016-06-22T12:05:01Z David Kalnischkies david@kalnischkies.de 2016-06-20T18:50:43Z urn:sha1:d03b947b0ce4f87d7d5cc48d4d274ab3bd0b289a Weak had no dedicated option before and Insecure and Downgrade were both global options, which given the effect they all have on security is rather bad. Setting them for individual repositories only isn't great but at least slightly better and also more consistent with other settings for repositories. 2016-05-07T12:52:08Z David Kalnischkies david@kalnischkies.de 2016-05-07T12:52:08Z urn:sha1:a8f565d3f69e6dba59195469959106da3eb8f33f We don't have to initialize the Release files with a set of IndexTargets to acquire, but instead wait for the Release file to be acquired and only then ask which IndexTargets to get. Git-Dch: Ignore 2016-05-07T12:44:53Z David Kalnischkies david@kalnischkies.de 2016-05-07T12:44:53Z urn:sha1:1eba782fc3c55528a4da14d79e114874b9299453 Progress reporting used an "upper bound" on files we might get, expect that this wasn't correct in case pdiff entered the picture. So instead of calculating a value which is perhaps incorrect, we just accept that we can't tell how many files we are going to download and just keep at 0% until we know. Additionally, if we have pdiffs we wait until we got these (sub)index files, too. That could all be done better by downloading all Release files first and planing with them in hand accordingly, but one step at a time. 2016-05-07T12:08:35Z David Kalnischkies david@kalnischkies.de 2016-05-07T12:08:35Z urn:sha1:b7ec7a8020e6d6a8f47177a6a03e3c9f21e5219b The code naturally evolved from a TransactionManager optional to a required setup which resulted in various places doing unneeded checks suggesting a more complicated setup than is actually needed. Git-Dch: Ignore 2016-04-25T13:35:52Z David Kalnischkies david@kalnischkies.de 2016-04-15T22:07:28Z urn:sha1:9b8034a9fd40b4d05075fda719e61f6eb4c45678 Redirection services like httpredir.debian.org tend to use a set of mirrors from which they pick a mirror at "random" for each requested file, which is usually benefitial for the download of debs, but for the index files this can quickly cause problems (aka hashsum mismatches) if the two (or more) mirrors involved are only slightly out-of-sync. This commit "resolves" this issue by using the mirror we ended up using to get the (signed) Release file directly to get the index files belonging to this Release file instead of asking the redirection service which eliminates the risk of hitting out-of-sync mirrors. As an obvious downside the redirection service can't serve partial mirrors anymore for indexes and the download of indexes indexed in the same Release file can't be done in parallel (from different mirrors). This does not effect the download of non-index files like deb-files as out-of-sync mirrors aren't a huge problem there, so the parallel download outweights a potentially 404 error (also because this causes no errenous downloads while hashsum mismatches download the entire file before finding out that it was pointless). The rational for this is that indexes are relative to the Release file. If we would be talking about a HTML page including images, such a behaviour is obvious and intended – not doing it means in the best case a bunch of "useless" requests which will all be answered with a redirect. 2016-04-25T13:35:52Z David Kalnischkies david@kalnischkies.de 2016-03-12T19:29:04Z urn:sha1:0340069cc4709a18ba117090763d9f263de999a9 Users tend to report these errors with just this error message… not very actionable and hard to figure out if this is a temporary or 'permanent' mirror-sync issue or even the occasional apt bug. Showing the involved hashsums and modification times should help in triaging these kind of bugs – and eventually we will have less of them via by-hash. The subheaders aren't marked for translation for now as they are technical glibberish and probably easier to deal with if not translated. After all, our iconic "Hash Sum mismatch" is translated at least. These additions were proposed in #817240 by Peter Palfrader.