Debians commandline package manager https://git.kalnischkies.de/apt/atom?h=1.3_rc4 2016-09-02T15:16:36Z 2016-09-02T15:16:36Z Julian Andres Klode jak@debian.org 2016-06-15T21:13:43Z urn:sha1:2a440328ea19e9646a93f847dd9eff21e03ad16d Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff. 2016-09-02T12:44:08Z Julian Andres Klode jak@debian.org 2016-09-02T12:44:08Z urn:sha1:544e1528b18025fad8318e6fb825ad296976cf24 This accidentally used ICONV_DIRECTORIES, which does not even exist. Weird. 2016-09-01T14:13:14Z David Kalnischkies david@kalnischkies.de 2016-08-31T08:11:07Z urn:sha1:644478e8db56f305601c3628a74e53de048b28c8 memcpy is marked as nonnull for its input, but ignores the input anyhow if the declared length is zero. Our SHA2 implementations do this as well, it was "just" MD5 and SHA1 missing, so we add the length check here as well as along the callstack as it is really pointless to do all these method calls for "nothing". Reported-By: gcc -fsanitize=undefined 2016-08-31T15:40:15Z Julian Andres Klode jak@debian.org 2016-08-31T15:18:07Z urn:sha1:cf7503d8a09ebce695423fdeb2402c456c18f3d8 If the inner Base256ToNum() returned false, it did not set Num to a new value, causing it to be uninitialized, and thus might have caused the function to exit despite a good result. Also document why the Res = Num, if (Res != Num) magic is done. Reported-By: valgrind 2016-08-31T15:39:06Z Julian Andres Klode jak@debian.org 2016-08-31T15:01:04Z urn:sha1:923c592ceb6014b31ec751b97b3ed659fa3e88ae Adding 1 to the value of d->End - current makes restLength one byte too long: If we pass memchr(current, ..., restLength) has thus undefined behavior. Also, reading the value of current has undefined behavior if current >= d->End, not only for current > d->End: Consider a string of length 1, that is d->End = d->Current + 1. We can only read at d->Current + 0, but d->Current + 1 is beyond the end of the string. This probably caused several inexplicable build failures on hurd-i386 in the past, and just now caused a build failure on Ubuntu's amd64 builder. Reported-By: valgrind 2016-08-31T10:12:56Z Julian Andres Klode jak@debian.org 2016-08-31T09:36:44Z urn:sha1:ce6cd75dc367b92f65e4fb539dd166d0f3361f8c If a Binary field contains one or more spaces before a comma, the code produced a segmentation fault, as it accidentally set a pointer to 0 instead of the value of the pointer. If the comma is at the beginning of the field, the code would create a binStartNext that points one element before the start of the string, which is undefined behavior. We also need to check that we do not exit the string during the replacement of spaces before commas: A string of the form " ," would normally exit the boundary of the Buffer: binStartNext = offset 1 ',' binEnd = offset 0 ' ' isspace_ascii(*binEnd) = true => --binEnd => binEnd = - 1 We get rid of the problem by only allowing spaces to be eliminated if they are not the first character of the buffer: binStartNext = offset 1 ',' binEnd = offset 0 ' ' binEnd > buffer = false, isspace_ascii(*binEnd) = true => exit loop => binEnd remains 0 2016-08-29T13:04:51Z Julian Andres Klode jak@debian.org 2016-08-29T13:04:51Z urn:sha1:06372c6a4f2bb8812f68c56788e96dc8fa69b3de Apparently we had no default defined for this. Reported-By: David Kalnischkies 2016-08-29T12:58:13Z Julian Andres Klode jak@debian.org 2016-08-29T12:58:13Z urn:sha1:865b46c18e38cab493141e9888eea74ed0d7da21 This accidentally had two apt in it. This fixes a regression from commit 8757a0f. Gbp-Dch: ignore 2016-08-29T07:22:25Z David Kalnischkies david@kalnischkies.de 2016-08-28T20:56:17Z urn:sha1:e950b7e2f89b5e48192cd469c963a44fff9f1450 An absolute filename for a *.deb file starts with a /. A package with the name of the file is inserted in the cache which is provided by the "real" package for internal reasons. The pinning code detects a regex based wildcard by having the regex start with /. That is no problem as a / can not be included in a package name… expect that our virtual filename package can and does. We fix this two ways actually: First, a regex is only being considered a regex if it also ends with / (we don't support flags). That stops our problem with the virtual filename packages already, but to be sure we also do not enter the loop if matcher and package name are equal. It has to be noted that the creation of pins for virtual packages like the here effected filename packages is pointless as only versions can be pinned, but checking that a package is really purely virtual is too costly compared to just creating an unused pin. Closes: 835818 2016-08-29T07:22:25Z David Kalnischkies david@kalnischkies.de 2016-08-28T10:58:20Z urn:sha1:4ff5e237d5685be187a75c563b86e80ea3e7cc01 Without randomizing the order in which we download the index files we leak needlessly information to the mirrors of which architecture is native or foreign on this system. More importantly, we leak the order in which description translations will be used which in most cases will e.g. have the native tongue first. Note that the leak effect in practice is limited as apt detects if a file it wants to download is already available in the latest version from a previous download and does not query the server in such cases. Combined with the fact that Translation files are usually updated infrequently and not all at the same time, so a mirror can never be sure if it got asked about all files the user wants.