summaryrefslogtreecommitdiff
path: root/apt-pkg
Commit message (Collapse)AuthorAgeFilesLines
* get pdiff files from the same mirror as the indexDavid Kalnischkies2016-11-253-36/+52
| | | | | | | | | | | In ad9416611ab83f7799f2dcb4bf7f3ef30e9fe6f8 we fall back to asking the original mirror (e.g. a redirector) if we do not get the expected result. This works for the indexes, but patches are a different beast and much simpler. Adding this fallback code here seems like overkill as they are usually right along their Index file, so actually forward the relevant settings to the patch items which fixes pdiff support combined with a redirector and partial mirrors as in such a situation the pdiff patches would be 404 and the complete index would be downloaded.
* report apt-key errors via status-fd messagesDavid Kalnischkies2016-11-241-10/+53
| | | | | | | | | | | | | | | | | | | | We report warnings from apt-key this way already since 29c590951f812d9e9c4f17706e34f2c3315fb1f6, so reporting errors seems like a good addition. Most of those errors aren't really from apt-key through, but from the code setting up and actually calling it which used to just print to stderr which might or might not intermix them with (other) progress lines in update calls. Having them as proper error messages in the system means that the errors are actually collected later on for the list instead of ending up with our relatively generic but in those cases bogus hint regarding "is gpgv installed?". The effective difference is minimal as the errors apply mostly to systems which have far worse problems than a not as nice looking error message, which makes this pretty hard to test – but at least now the hint that your system is broken can be read in proper order (= there aren't many valid cases in which the permissions of /tmp are messed up…). LP: #1522988
* skip unconfigure for unconfigured to-be removed pkgsDavid Kalnischkies2016-11-241-0/+12
|
* do not configure unconfigured to be removed packagesDavid Kalnischkies2016-11-241-1/+2
| | | | | | | | | | | | | We try to configure all packages at the end which need to be configured, but that also applies to packages which weren't completely installed (e.g. maintainerscript failed) we end up removing in this interaction instead. APT doesn't perform this explicit configure in the end as it is using "dpkg --configure --pending", but it does confuse the progress report and potentially also hook scripts. Regression-Of: 9ffbac99e52c91182ed8ff8678a994626b194e69
* don't perform implicit crossgrades involving M-A:sameDavid Kalnischkies2016-11-241-3/+22
| | | | | | dpkg stumbles over these (#844300) and we haven't dropped 'easier' removes to be implicit and to be scheduled by dpkg by default so far so we shouldn't push the decision in such cases to dpkg either.
* improve arch-unqualified dpkg-progress parsingDavid Kalnischkies2016-11-242-62/+155
| | | | | | | | | | | Our old idea was to look for the first package which would be "touched" and take this as the package dpkg is talking about, but that is incorrect in complicated situations like a package upgraded to/from multiple M-A:same siblings installed. As we us the progress report to decide what is still needed we have to be reasonabily right about the package dpkg is talking about, so we jump to quite a few loops to get it.
* correct cross & disappear progress detectionDavid Kalnischkies2016-11-231-35/+58
| | | | | | | | | | | | Given that we use the progress information to skip over actions dpkg has already done like not purging a package which was already removed and had no config files or not acting on disappeared packages and such it is important that apt and dpkg agree on which states the package has to pass through. To ensure that we keep tabs on this in the future a warning is added at the end if apt hasn't seen all the action it was supposed to see. I can't wait for the first bugreporters to wonder about this…
* react to trig-pend only if we have nothing else to doDavid Kalnischkies2016-11-231-12/+12
| | | | | | | | | If a package is triggered dpkg frequently issues two messages about it causing us to make a note about it both times which messes up our planned dpkg actions view. Adding these actions if we have nothing else planned fixes this and should still be correct as those planned actions will deal with the triggering just fine and we avoid strange problems like a package triggered before its removed…
* Do not use MD5SumValue for Description_md5()Julian Andres Klode2016-11-226-33/+42
| | | | | | | | | | | Our profile says we spend about 5% of the time transforming the hex digits into the binary format used by HashsumValue, all for comparing them against the other strings. That makes no sense at all. According to callgrind, this reduces the overall instruction count from 5,3 billion to 5 billion in my example, which roughly matches the 5%.
* debListParser: Micro-optimize AvailableDescriptionLanguages()Julian Andres Klode2016-11-221-8/+7
| | | | | | | | | | | | | Generating a string for each version we see is somewhat inefficient. The problem here is that the Description tag names are longer than 15 byte, and thus require an allocation on the heap, which we should avoid. It seems reasonable that 20 characters works for all languages codes used for archive descriptions, but if not, there's a warning, so we'll catch that. This should improve performance by about 2%.
* Compare size before data when ordering cache bucket entriesJulian Andres Klode2016-11-223-3/+14
| | | | | | | This has the effect of significantly reducing actual string comparisons, and should improve the performance of FindGrp a bit, although it's hardly measureable (callgrind says it uses 10% instructions less now).
* Optimize VersionHash() to not need temporary copy of inputJulian Andres Klode2016-11-223-14/+11
| | | | | | | Stop copying stuff, and just parse the bytes one by-one to the newly created AddCRC16Byte. This improves the instruction count for an update run from 720,850,121 to 455,801,749 according to callgrind.
* Introduce tolower_ascii_unsafe() and use it for hashingJulian Andres Klode2016-11-223-4/+9
| | | | | | | This one has some obvious collisions for non-alphabetical characters, like some control characters also hashing to numbers, but we don't really have those, and these are hash functions which are not collision free to begin with.
* Bump the cache major version for non-backportable changesJulian Andres Klode2016-11-221-2/+2
| | | | | | | We already have two stable series with major version 10, and the next commits will introduce non-backportable performance changes that affect the cache algorithms, so we need to bump the major version now to prevent future problems.
* debListParser: Convert to use pkgTagSection::Key-based lookupJulian Andres Klode2016-11-222-42/+42
| | | | | | This basically gets rid of 40-50% of the hash table lookups, making things a bit faster that way, and the profiles look far cleaner.
* TagSection: Introduce functions for looking up by key idsJulian Andres Klode2016-11-224-11/+190
| | | | | Introduce a new enum class and add functions that can do a lookup with that enum class. This uses triehash.
* TagSection: Extract Find() methods taking Pos instead of KeyJulian Andres Klode2016-11-222-20/+65
| | | | | This allows us to add a perfect hash function to the tag file without having to reimplement the methods a second time.
* TagSection: Split AlphaIndexes into AlphaIndexes and BetaIndexesJulian Andres Klode2016-11-222-11/+14
| | | | | | | | | Move the use of the AlphaHash to a new second hash table in preparation for the arrival of the new perfect hash function. With the new perfect hash function hashing most of the keys for us, having 128 slots for a fallback hash function seems enough and prevents us from wasting space.
* show distribution mismatch for changed codenamesDavid Kalnischkies2016-11-111-0/+8
| | | | | | | | | | We have the last Release file around for other checks, so its trivial to look if the new Release file contains a new codename (e.g. the user has "testing" in the sources and it flipped from stretch to buster). Such a change can be okay and expected, but also be a hint of problems, so a warning if we see it happen seems okay. We can only print it once anyhow and frontends and co are likely to ignore/hide it.
* don't warn if untransformed distribution matchesDavid Kalnischkies2016-11-112-45/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A suite or codename entry in the Release file is checked against the distribution field in the sources.list entry that lead to the download of that Release file. This distribution entry can contain slashes in the distribution field: deb http://security.debian.org/debian wheezy/updates main However, the Release file may only contain "wheezy" in the Codename field and not "wheezy/updates". So a transformation needs to take place that removes the last / and everything that comes after (e.g. "/updates"). This fails, however, for valid cases like a reprepro snapshot where the given Codename contains slashes but is perfectly fine and doesn't need to be transformed. Since that transformation is essentially just a workaround for special cases like the security repository, it should be checked if the literal Codename without any transformations happened is valid and only if isn't the dist should be checked against the transformated one. This way special cases like security.debian.org are handled and reprepro snapshots work too. The initial patch was taken as insperationto move whole transformation to CheckDist() which makes this method more accepting & easier to use (but according to codesearch.d.n we are the only users anyhow). Thanks: Lukas Anzinger for initial patch Closes: 644610
* add hidden config to set packages as Essential/ImportantDavid Kalnischkies2016-11-112-4/+16
| | | | | | | | | | | | | | | | You can pretty much achieve the same with a local dummy package if you want to, but libapt has an inbuilt setting for essential: "apt" which can be overridden with this option as well – it could be helpful in quick tests and what not so adding this alternative shouldn't really hurt much. We aren't going to document them much through as care must be taken in regards to the binary caches as they aren't invalidated by config options alone, so the effects of old settings could still be in them, similar to the other already existing pkgCacheGen option(s). Closes: 767891 Thanks: Anthony Towns for initial patch
* add TMP/TEMP/TEMPDIR to the TMPDIR DropPrivileges danceDavid Kalnischkies2016-11-111-9/+20
| | | | | | | apt tools do not really support these other variables, but tools apt calls might, so lets play save and clean those up as needed. Reported-By: Paul Wise (pabs) on IRC
* support 'apt build-dep .' (aka: without /)David Kalnischkies2016-11-101-2/+2
| | | | Reported-By: Christoph Berg (Myon) on IRC
* rename Checksum-FileSize to Filesize in hashsum mismatchDavid Kalnischkies2016-11-091-14/+17
| | | | | | | | Some people do not recognize the field value with such an arcane name and/or expect it to refer to something different (e.g. #839257). We can't just rename it internally as its an avoidance strategy as such fieldname existed previously with less clear semantics, but we can spare the general public from this implementation detail.
* show the conflicting distribution warning againDavid Kalnischkies2016-11-091-2/+1
| | | | | | | Sometimes you should really act upon your todos. Especially if you have placed them directly in the code. Closes: 841874
* reset HOME, USER(NAME), TMPDIR & SHELL in DropPrivilegesDavid Kalnischkies2016-11-091-0/+20
| | | | | | | | | We can't cleanup the environment like e.g. sudo would do as you usually want the environment to "leak" into these helpers, but some variables like HOME should really not have still the value of the root user – it could confuse the helpers (USER) and HOME isn't accessible anyhow. Closes: 842877
* fix three typos in sources & manpagesChris Leick2016-11-091-1/+1
|
* add support for Build-Depends/Conflicts-ArchJohannes Schauer2016-11-094-6/+11
| | | | | | | | | | | | | | These new enum values might cause "interesting" behaviour in tools not expecting them – like an old apt would think a Build-Conflicts-Arch is some sort of Build-Depends – but that can't reasonably be avoided and effects only packages using B-D/C-A so if there is any breakage the tools can easily be adapted. The APT_PKG_RELEASE number is increased so that libapt users can detect the availability of these new enum fields via: #if APT_PKG_ABI > 500 || (APT_PKG_ABI == 500 && APT_PKG_RELEASE >= 1) Closes: #837395
* keep Release.gpg on untrusted to trusted IMS-HitDavid Kalnischkies2016-11-021-3/+9
| | | | | | | | | | | | | | | A user relying on the deprecated behaviour of apt-get to accept a source with an unknown pubkey to install a package containing the key expects that the following 'apt-get update' causes the source to be considered as trusted, but in case the source hadn't changed in the meantime this wasn't happening: The source kept being untrusted until the Release file was changed. This only effects sources not using InRelease and only apt-get, the apt binary downright refuses this course of actions, but it is a common way of adding external sources. Closes: 838779
* Do not read stderr from proxy autodetection scriptsJulian Andres Klode2016-10-043-2/+11
| | | | | | | | | | | | | This fixes a regression introduced in commit 8f858d560e3b7b475c623c4e242d1edce246025a don't leak FD in AutoProxyDetect command return parsing which accidentally made the proxy autodetection code also read the scripts output on stderr, not only on stdout when it switched the code from popen() to Popen(). Reported-By: Tim Small <tim@seoss.co.uk>
* VersionHash: Do not skip too long dependency linesJulian Andres Klode2016-09-182-3/+3
| | | | | | | | | | | | | | | If the dependency line does not contain spaces in the repository but does in the dpkg status file (because dpkg normalized the dependency list), the dpkg line might be longer than the line in the repository. If it now happens to be longer than 1024 characters, it would be skipped, causing the hashes to be out of date. Note that we have to bump the minor cache version again as this changes the format slightly, and we might get mismatches with an older src cache otherwise. Fixes Debian/apt#23
* Coverage: Do not print messages from gcovJulian Andres Klode2016-09-111-0/+4
| | | | | | | We need to ignore messages from gcov. All those messages start with profiling: and are printed using vfprintf(), so the only thing we can do is add a library overriding those functions and linking apt-pkg to it.
* edsp: try 2 to read responses even if writing failedDavid Kalnischkies2016-09-071-15/+27
| | | | | | | | | | | | | | | | Commit b60c8a89c281f2bb945d426d2215cbf8f5760738 improved the situation, but due to inconsistency mostly for planners, not for solvers. As the idea of hiding errors if we show another error is a bit scary (as the extern error might be a followup of our intern error, rather than the reason for our intern error as it is at the moment) we don't discard the errors, but if we got an extern error we show them directly removing them from the error list at the end of the run – that list will contain the extern error which hopefully gives us the best of both worlds. The problem itself is the same as before: The externals exiting before apt is done talking to them. Reported-By: Johannes 'josch' Schauer on IRC
* acquire: Use priority queues and a 3 stage pipeline designJulian Andres Klode2016-09-024-4/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff.
* CMake: apt-pkg: Use correct ICONV_INCLUDE_DIRS variableJulian Andres Klode2016-09-021-1/+1
| | | | | This accidentally used ICONV_DIRECTORIES, which does not even exist. Weird.
* try not to call memcpy with length 0 in hash calculationsDavid Kalnischkies2016-09-018-15/+21
| | | | | | | | | | memcpy is marked as nonnull for its input, but ignores the input anyhow if the declared length is zero. Our SHA2 implementations do this as well, it was "just" MD5 and SHA1 missing, so we add the length check here as well as along the callstack as it is really pointless to do all these method calls for "nothing". Reported-By: gcc -fsanitize=undefined
* Base256ToNum: Fix uninitialized valueJulian Andres Klode2016-08-311-1/+2
| | | | | | | | | | If the inner Base256ToNum() returned false, it did not set Num to a new value, causing it to be uninitialized, and thus might have caused the function to exit despite a good result. Also document why the Res = Num, if (Res != Num) magic is done. Reported-By: valgrind
* TagFile: Fix off-by-one errors in comment strippingJulian Andres Klode2016-08-311-2/+2
| | | | | | | | | | | | | | | | | | | Adding 1 to the value of d->End - current makes restLength one byte too long: If we pass memchr(current, ..., restLength) has thus undefined behavior. Also, reading the value of current has undefined behavior if current >= d->End, not only for current > d->End: Consider a string of length 1, that is d->End = d->Current + 1. We can only read at d->Current + 0, but d->Current + 1 is beyond the end of the string. This probably caused several inexplicable build failures on hurd-i386 in the past, and just now caused a build failure on Ubuntu's amd64 builder. Reported-By: valgrind
* Fix segfault and out-of-bounds read in Binary fieldsJulian Andres Klode2016-08-311-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | If a Binary field contains one or more spaces before a comma, the code produced a segmentation fault, as it accidentally set a pointer to 0 instead of the value of the pointer. If the comma is at the beginning of the field, the code would create a binStartNext that points one element before the start of the string, which is undefined behavior. We also need to check that we do not exit the string during the replacement of spaces before commas: A string of the form " ," would normally exit the boundary of the Buffer: binStartNext = offset 1 ',' binEnd = offset 0 ' ' isspace_ascii(*binEnd) = true => --binEnd => binEnd = - 1 We get rid of the problem by only allowing spaces to be eliminated if they are not the first character of the buffer: binStartNext = offset 1 ',' binEnd = offset 0 ' ' binEnd > buffer = false, isspace_ascii(*binEnd) = true => exit loop => binEnd remains 0
* init: Add Dir::Bin::planners default entryJulian Andres Klode2016-08-291-0/+1
| | | | | | Apparently we had no default defined for this. Reported-By: David Kalnischkies
* init: Fix path to external solversJulian Andres Klode2016-08-291-1/+1
| | | | | | | This accidentally had two apt in it. This fixes a regression from commit 8757a0f. Gbp-Dch: ignore
* don't loop on pinning pkgs from absolute debs by regexDavid Kalnischkies2016-08-291-5/+4
| | | | | | | | | | | | | | | | | | | | | An absolute filename for a *.deb file starts with a /. A package with the name of the file is inserted in the cache which is provided by the "real" package for internal reasons. The pinning code detects a regex based wildcard by having the regex start with /. That is no problem as a / can not be included in a package name… expect that our virtual filename package can and does. We fix this two ways actually: First, a regex is only being considered a regex if it also ends with / (we don't support flags). That stops our problem with the virtual filename packages already, but to be sure we also do not enter the loop if matcher and package name are equal. It has to be noted that the creation of pins for virtual packages like the here effected filename packages is pointless as only versions can be pinned, but checking that a package is really purely virtual is too costly compared to just creating an unused pin. Closes: 835818
* randomize acquire order for same type index filesDavid Kalnischkies2016-08-291-0/+18
| | | | | | | | | | | | | | | Without randomizing the order in which we download the index files we leak needlessly information to the mirrors of which architecture is native or foreign on this system. More importantly, we leak the order in which description translations will be used which in most cases will e.g. have the native tongue first. Note that the leak effect in practice is limited as apt detects if a file it wants to download is already available in the latest version from a previous download and does not query the server in such cases. Combined with the fact that Translation files are usually updated infrequently and not all at the same time, so a mirror can never be sure if it got asked about all files the user wants.
* Merge branch 'portability/freebsd'Julian Andres Klode2016-08-2713-30/+42
|\
| * CMake: Add missing iconv dependencyJulian Andres Klode2016-08-261-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | FreeBSD has two iconv systems: It ships an iconv.h itself, and symbols for that in the libc. But there's also the port of GNU libiconv, which unfortunately for us, Doxygen depends on. This changes things to prefer a separate libiconv library over the system one; that is, the port on FreeBSD. Gbp-Dch: ignore
| * Make root group configurable via ROOT_GROUPJulian Andres Klode2016-08-264-8/+8
| | | | | | | | | | This is needed on BSD where root's default group is wheel, not root.
| * changelog: Respect Dir setting for local changelog gettingJulian Andres Klode2016-08-261-1/+2
| | | | | | | | | | This fixes issues with chroots, but the goal here was to get the test suite working on systems without dpkg.
| * Make directory paths configurableJulian Andres Klode2016-08-264-12/+12
| | | | | | | | | | | | | | This allows other vendors to use different paths, or to build your own APT in /opt for testing. Note that this uses + 1 in some places, as the paths we receive are absolute, but we need to strip of the initial /.
| * Use C locale instead of C.UTF-8 for protocol stringsJulian Andres Klode2016-08-264-6/+6
| | | | | | | | | | | | The C.UTF-8 locale is not portable, so we need to use C, otherwise we crash on other systems. We can use std::locale::classic() for that, which might also be a bit cheaper than using locale("C").
| * CMake: Do not use -lresolv if res_init exists in libcJulian Andres Klode2016-08-261-1/+1
| | | | | | | | Gbp-Dch: ignore
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-01 20:29:31 +0000