Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=1.2.6 2016-03-06T11:57:38Z 2016-03-06T11:57:38Z David Kalnischkies david@kalnischkies.de 2016-03-06T11:03:34Z urn:sha1:dfcf7f356b790338f0a3e9df3c5d6f159814fe53 If a single pdiff fails, we have to fail the entire patching endeavour and fall back to getting the complete file instead. That is easy in serverside merged pdiffs as we get them one by one. For clientside we get them all at once through, which means that a failure in one has to stop the entire pipeline, which works as expected (as proven by the bugreporters as they don't even notice it happening). The problem is just that the first failing pdiff will do the cleanup, so another pdiff which happens to be successfully acquired after we processed the failure doesn't find the file it is supposed to use as a basename anymore, so the patch is renamed to what should be the unique extension and moved into the current working directory. Processing is then stopped as the patch realizes that it isn't the last one which completed downloading. On the plus side this means this is neither us using a bad temporary location nor a security problem. It "just" overrides unconditionally files in your current working directory (if you happen to have them named like a pdiff patch – a bit unlikely perhaps) and so drops files there which are never used again. I guess this was introduced in 4e3c5633b1e74b4f58b95f339cfbbf4cbf21ab3e for real as I made the need for the existence of the base file rather explicit, but the potential lingers in the code for far longer. Closes: #816837 2016-01-08T16:51:23Z David Kalnischkies david@kalnischkies.de 2016-01-08T16:51:23Z urn:sha1:ef3c549e00b2a0487ddee0aeb70e3a29f76c2fbb The code already deals with compressed leftovers, but forgot the uncompressed files. The opertunity is picked to reorder this code and add debug messages about the actions taken as well as produce such a leftover file in the associated testcase. 2016-01-08T14:40:01Z David Kalnischkies david@kalnischkies.de 2016-01-08T14:30:05Z urn:sha1:4e6219da0dd1e68fad7db972f7ddd76598645228 With the addition of the $HASH-Download field in the .diff/Index we got the size of the compressed patches for 'free', so if that information is available we can use it for a more fitting calculation of the size requirements of the patches vs. the complete file. Note that this predicts a too small size in the transition case in which the information isn't available for all patches, but figuring this out would be a lot of code for practically nothing as only one update can ever be in such a transition phase. 2016-01-08T14:40:01Z David Kalnischkies david@kalnischkies.de 2016-01-07T19:32:09Z urn:sha1:0179cfa83cf0042235eda41db7f35c420781c63e Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access. 2016-01-08T14:40:01Z David Kalnischkies david@kalnischkies.de 2016-01-05T23:05:24Z urn:sha1:4e3c5633b1e74b4f58b95f339cfbbf4cbf21ab3e There is no reason to enforce that the file we start the bootstrap with is compressed with a compressor which is available online. This allows us to change the on-disk format as well as deals with repositories adding/removing support for a specific compressor. 2015-12-19T22:04:34Z David Kalnischkies david@kalnischkies.de 2015-12-15T16:20:26Z urn:sha1:3abb6a6a1e485b3bc899b64b0a1b7dc2db25a9c2 This doesn't allow all tests to run cleanly, but it at least allows to write tests which could run successfully in such environments. Git-Dch: Ignore 2015-09-14T13:22:19Z David Kalnischkies david@kalnischkies.de 2015-09-14T00:26:13Z urn:sha1:63c7141275c8c5c0f6e60f5242785e50cabaf2a0 Not all tests work yet, most notable the cdrom tests, but those require changes in libapt itself to have a proper fix and what we have fixed so far is good enough progress for now. Git-Dch: Ignore 2015-09-14T13:22:19Z David Kalnischkies david@kalnischkies.de 2015-09-13T15:25:23Z urn:sha1:c5ede4cac6e6496ce19ccea3313ac6b49ba5e8d6 This is mostly a small speedup for the testcases, but it is also handy to document which tests actually deal with a specific hash compared to those which 'just' need some hash which can be important while adding new hashes. Git-Dch: Ignore 2015-09-14T13:22:19Z David Kalnischkies david@kalnischkies.de 2015-09-12T08:35:49Z urn:sha1:7414af7fa88164209eec9c585b8d175c1618ecbc And of course, testing obscure things ends up showing obscure 'bugs' or better shortcomings/inconsitencies, so lets fix them with the tests. Git-Dch: Ignore 2015-08-28T17:26:44Z David Kalnischkies david@kalnischkies.de 2015-08-28T17:26:44Z urn:sha1:d7a51997c30b2098bb60b3397095ec58ec825303 Some additional files like 'Contents' are very big and should therefore kept compressed on the disk, which apt-file did in the past. It also implemented pdiff patching of these files by un- and recompressing these files on-the-fly, with this commit we can do the same – but we can do this in both pdiff patching styles (client and server merging) and secured by hashes. Hashes are in so far slightly complicated as we can't compare the hashes of the compressed files as we might compress them differently than the server would (different compressor versions, options, …), so we must compare the hashes of the uncompressed content. While this commit has changes in public headers, the classes it changes are marked as hidden, so nobody can use them directly, which means the ABI break is internal only.