Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=1.2.4 2016-01-27T15:39:52Z 2016-01-27T15:39:52Z David Kalnischkies david@kalnischkies.de 2016-01-27T14:28:17Z urn:sha1:6fc2e03084c7e027c2b9a63c1fe99ff743aae3b6 The Date field in the Release file is useful to avoid allowing an attacker to 'downgrade' a user to earlier Release files (and hence to older states of the archieve with open security bugs). It is also needed to allow a user to define min/max values for the validation of a Release file (with or without the Release file providing a Valid-Until field). APT wasn't formally requiring this field before through and (agrueable not binding and still incomplete) online documentation declares it optional (until now), so we downgrade the error to a warning for now to give repository creators a bit more time to adapt – the bigger ones should have a Date field for years already, so the effected group should be small in any case. It should be noted that earlier apt versions had this as an error already, but only showed it if a Valid-Until field was present (or the user tried to used the configuration items for min/max valid-until). Closes: 809329 2016-01-08T14:40:01Z David Kalnischkies david@kalnischkies.de 2016-01-07T19:32:09Z urn:sha1:0179cfa83cf0042235eda41db7f35c420781c63e Downloading and storing are two different operations were different compression types can be preferred. For downloading we provide the choice via Acquire::CompressionTypes::Order as there is a choice to be made between download size and speed – and limited by whats available in the repository. Storage on the other hand has all compressions currently supported by apt available and to reduce runtime of tools accessing these files the compression type should be a low-cost format in terms of decompression. apt traditionally stores its indexes uncompressed on disk, but has options to keep them compressed. Now that apt downloads additional files we also deal with files which simply can't be stored uncompressed as they are just too big (like Contents for apt-file). Traditionally they are downloaded in a low-cost format (gz) as repositories do not provide other formats, but there might be even lower-cost formats and for download we could introduce higher-cost in the repositories. Downloading an entire index potentially requires recompression to another format, so an update takes potentially longer – but big files are usually updated via pdiffs which has to de- and re-compress anyhow and does it on the fly anyhow, so there is no extra time needed and in general it seems to be benefitial to invest the time in update to save time later on file access. 2015-12-27T16:04:33Z David Kalnischkies david@kalnischkies.de 2015-12-27T16:04:33Z urn:sha1:a628ca5256b4a2f3ae300697b17adf150b6e17b0 Debian has a Packages file for arch:all already, but the arch:any files contain arch:all packages as well, so downloading it would be a total waste of resources. Getting this solved is on the list of things to do, but it is also the hardest part – for index targets like Contents the situation is much easier and less server/client implementations are involved so we might not want to stall them. A repository can now declare via: No-Support-for-Architecture-all: Packages that even if an arch:all Packages exists, it shouldn't be downloaded, so that support for Contents files can be added now. See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation of downloading arch:all index targets, which this is limiting. The field uses the name of the target from the apt configuration for simplicity and is negative by design as this field is intended to be supported/needed only for a "short" time (one or two Debian releases). While this commit theoretically supports any target, its expected to only see "Packages" as a value in reality. 2015-12-14T01:26:23Z David Kalnischkies david@kalnischkies.de 2015-12-14T01:18:25Z urn:sha1:bd4a8f51649ee37291c6e07310104a94f4f5fbed If we can't work with the hashes we parsed from the Release file we display now an error message if the Release file includes only weak hashes instead of downloading the indexes and failing to verify them with "Hash Sum mismatch" even through the hashes didn't mismatch (they were just weak). If for some (unlikely) reason we have got weak hashes only for individual targets we will show a warning to this effect (again, befor downloading and failing the index itself). Closes: 806459 2015-11-21T17:04:29Z Justin B Rye justin.byam.rye@gmail.com 2015-11-21T16:50:06Z urn:sha1:d04e44ac8177fc5b70ae0189bb5e437c2502f910 Reference mail: https://lists.debian.org/debian-l10n-english/2015/11/msg00006.html 2015-11-04T17:42:27Z David Kalnischkies david@kalnischkies.de 2015-10-28T13:38:49Z urn:sha1:1dd20368486820efb6ef4476ad739e967174bec4 Based on a discussion with Niels Thykier who asked for Contents-all this implements apt trying for all architecture dependent files to get a file for the architecture all, which is treated internally now as an official architecture which is always around (like native). This way arch:all data can be shared instead of duplicated for each architecture requiring the user to download the same information again and again. There is one problem however: In Debian there is already a binary-all/ Packages file, but the binary-any files still include arch:all packages, so that downloading this file now would be a waste of time, bandwidth and diskspace. We therefore need a way to decide if it makes sense to download the all file for Packages in Debian or not. The obvious answer would be a special flag in the Release file indicating this, which would need to default to 'no' and every reasonable repository would override it to 'yes' in a few years time, but the flag would be there "forever". Looking closer at a Release file we see the field "Architectures", which doesn't include 'all' at the moment. With the idea outlined above that 'all' is a "proper" architecture now, we interpret this field as being authoritative in declaring which architectures are supported by this repository. If it says 'all', apt will try to get all, if not it will be skipped. This gives us another interesting feature: If I configure a source to download armel and mips, but it declares it supports only armel apt will now print a notice saying as much. Previously this was a very cryptic failure. If on the other hand the repository supports mips, too, but for some reason doesn't ship mips packages at the moment, this 'missing' file is silently ignored (= that is the same as the repository including an empty file). The Architectures field isn't mandatory through, so if it isn't there, we assume that every architecture is supported by this repository, which skips the arch:all if not listed in the release file. 2015-09-14T13:22:19Z David Kalnischkies david@kalnischkies.de 2015-09-14T11:18:29Z urn:sha1:24e8f24e1e94ec3816b0bfc7a05d1c4e3f73248e This changes the semantics of the option (which is renamed too) to be a yes/no value with the special additional value "force" as this allows by-hash to be disabled even if the repository indicates it would be supported and is more in line with our other yes/no options like pdiff which disable themselves if no support can be detected. The feature wasn't documented so far and hasn't reached a (un)stable release yet, so changing it without trying too hard to keep compatibility seems okay. 2015-09-14T13:22:18Z David Kalnischkies david@kalnischkies.de 2015-09-10T17:00:51Z urn:sha1:95278287f4e1eeaf5d96749d6fc9bfc53fb400d0 Our error reporting is historically grown into some kind of mess. A while ago I implemented stacking for the global error which is used in this commit now to wrap calls to functions which do not report (all) errors via return, so that only failures in those calls cause a failure to propergate down the chain rather than failing if anything (potentially totally unrelated) has failed at some point in the past. This way we can avoid stopping the entire acquire process just because a single source produced an error for example. It also means that after the acquire process the cache is generated – even if the acquire process had failures – as we still have the old good data around we can and should generate a cache for (again). There are probably more instances of this hiding, but all these looked like the easiest to work with and fix with reasonable (aka net-positive) effects. 2015-08-30T20:50:55Z David Kalnischkies david@kalnischkies.de 2015-08-30T20:34:28Z urn:sha1:3090ae6972fd0e15767a96708c248f3ab87502f2 Multiple targets downloading the same file is bad™ as it leads us to all sorts of problems like the acquire system breaking or simply a problem of which settings to use for them. Beside that this is most likely a mistake and silently ignoring it doesn't help the user realizing his mistake… On the other hand, we have 'duplicates' which are 'created' by how we create indextargets, so we have to prevent those from being created to but do not emit a warning for them as this is an implementation detail. And then, there is the absolute and most likely user mistake: Having the same target(s) activated in multiple entries. 2015-08-30T10:14:06Z David Kalnischkies david@kalnischkies.de 2015-08-30T10:14:06Z urn:sha1:c4d1ab98921cddc8bd01f1e23ec1f4f9e7d2a90a