summaryrefslogtreecommitdiff
path: root/apt-pkg/acquire.cc
Commit message (Collapse)AuthorAgeFilesLines
* Improve error handling of cycling delayed queuesJulian Andres Klode2021-09-061-5/+8
| | | | | | | | | When an item has been delayed and the queue is cycled to start it, we did not properly report an error from the cycling, and we would then fail in the assert(), causing all errors to be lost. Propagate the error instead and make the assert a warning.
* Properly convert PulseInterval to timevalJulian Andres Klode2021-07-281-6/+3
| | | | | Fix the typo, and use the helper function to convert it, so we do not end up with 5 seconds encoded as 0s and 5*10^6 microseconds.
* Convert to monotonic clockJulian Andres Klode2021-07-281-14/+18
| | | | | This yields more accurate delays and avoids issues with clock skew.
* acquire: Automatically reduce select() timeout as neededJulian Andres Klode2021-07-281-1/+35
| | | | | | | | | | | | | | If there is an item with fetchAfter at the top of a queue, reduce sleep() timeout so we can detect it and start it, by calling Cycle() on the queue in the next iteration. For some reasons we have to call select() with a 0s timeout if we just marked an item as ready. Oh well. Previous versions of this patch only called global Bump() after a timeout from select(); this was unfortunately incorrect - it meant that we never bumped a queue that did not start yet while other queues were running, potentially significantly delaying retries.
* Implement exponential delay between retriesJulian Andres Klode2021-07-281-3/+22
| | | | | | | Add a new Item field called FetchAfter, which determines the earliest time the item should be fetched at. Adjust insertion into queue to take it into account alongside priority, and only fill pipelines with items that are ready.
* Keep URIs encoded in the acquire systemDavid Kalnischkies2020-12-181-6/+40
| | | | | | | | | | | | | | | | | | | We do not deal a lot with URIs which need encoding, but then we do it is a pain that we store it decoded in the acquire system as it means we have to decode and reencode URIs eventually which is potentially giving us slightly different URIs. We see that in our own testing framework while setting up redirects as the config options are effectively double-encoded and decoded to pass them around successfully as otherwise %2f and / in an URI are treated the same. This commit adds the infrastructure for methods to opt into getting URIs send in encoded form (and returning them to us in encoded form, too) so that we eventually do not have to touch the URIs which is how it should be. This means though that we have to deal with methods who do not support this yet (aka: all at the moment) for which we decode and encode while communicating with them.
* Do not hardcode (wrong) group and mode in setup warningDavid Kalnischkies2020-06-061-3/+3
| | | | | | | | | | | | | | Partial directories are created with 0700, but the parent is 0755, while the error message would report 0700 for both… that isn't right and can be pretty confusing. Turns out that the messages aren't marked for translation, so no unfuzzing is required & we just leave it as untranslated for now. Especially as the more detailed error strings derived from errno are translated. Reported-By: Wakko Warner <wakko@animx.eu.org> Closes: #962310
* Fix small memory leak in MethodConfigDavid Kalnischkies2020-05-271-1/+1
| | | | | | | | | We are leaking a d-pointer currently weighting a boolean in size and MethodConfig is instantiated in small numbers only, so nobody will actually notice a difference, but proper cleanup is important. Reported-By: clang LeakSanitizer References: 04ab37fecaf286f724bef2e0969d2b67ab5ac1b1
* Merge pkgAcquire::RunFdsSane back into RunFdsJulian Andres Klode2020-02-261-9/+3
|
* acquire: Move queue startup after calling log's Start()Julian Andres Klode2020-01-081-4/+4
| | | | | | | | | | Startup() was checking for bad items and failing them, but we did not actually call Start() in the log, so the log might not be setup correctly. This caused a crash in python-apt when items were being failed on queue startup, as it released the GIL when Start() is being called and re-acquires it when running callbacks.
* Fix some style warnings from cppcheckDavid Kalnischkies2019-11-261-2/+0
| | | | | | | | Unused variable, std::algorithms instead of raw for-loops. There should be no observeable difference in behaviour. Reported-By: cppcheck Gbp-Dch: Ignore
* Fix segfault in pkgAcquire::Enqueue() with Acquire::Queue-Mode=accessJulian Andres Klode2019-08-151-4/+6
| | | | | | | | | | | | | | | | | In commit 79b1a8298, QueueName() was changed, amongst other things, to exit early when the queue mode was single access, as single access does not need any fancy queue name. The exit became too early though, as Config was not initialized anymore, but the caller was relying on it. Fix QueueName() to always initialize Config and in Enqueue() initialize Config with a nullptr, so if this regresses it's guaranteed to fail harder. Also add a test case - this is very simple, but the first and only test case for access queue mode. Regression-Of: 79b1a82983e737e74359bc306d9edb357c5bdd46 LP: #1839714
* Distribute host-less work based on backlog of the queuesDavid Kalnischkies2019-07-081-44/+66
| | | | | | | | | | | | | | | | | Work like applying patches via rred can be performed by many concurrent rred processes, but we can't just spawn new ones forever: We limit us to the number of CPUs which can drive them and reuse existing ones if they have nothing to do at the moment. The problem arises if we have reached the limit of queues and all of them are busy which is more likely to happen on "slow" machines with few CPUs. In this case we opted for random distribution, but that can result in many big files (e.g. Contents) being added to one queue while the others get none or only small files. Ideally we would ask the methods how much they still have to do, but they only know that for the current item, not for all items in the queue, so we use the filesize of the expected result.
* apt-pkg: URI: Add 'explicit' to single argument constructorJulian Andres Klode2019-04-301-1/+1
| | | | | This needs a fair amount of changes elsewhere in the code, hence this is separate from the previous commits.
* acq: worker: Move CurrentSize, TotalSize, ResumePoint to CurrentItemJulian Andres Klode2019-04-301-3/+3
| | | | | These status fields belong to the current item, move them there. This prepares us for eventually having multiple current items.
* Don't limit cpu-limited queues to at most 10David Kalnischkies2019-04-161-5/+7
| | | | | | | | | | | Queues for processes like rred are not created by hostname but we spawn at most CPU*2 queues to place items in. The problem is that we then proceeded to limit it to at most 10 queues (via QueueHost::Limit) again at the end of the method so that all items (after the first 10 queues are busy) are forcibly placed into a generic catch-all instance which is bad because we don't keep all CPUs we have available busy and worse we end up sheduling the most work to a single one while random distribution was intended.
* acquire: Remove deprecated pkgAcquire::Setup() functionJulian Andres Klode2019-02-261-19/+0
|
* acquire: Fold pkgAcquireStatus2 into pkgAcquireStatusJulian Andres Klode2019-02-261-9/+2
| | | | Clean up the code, make it neat, lalala
* Fix calculation of elapsed usec in downloadsJulian Andres Klode2018-09-241-2/+2
| | | | | | A recent change to use chronos inadvertently replaced the difference of new usec - old usec with new sec - old usec, which is obviously wrong.
* Use a steady clock source for progress reportingDavid Kalnischkies2018-05-281-27/+35
| | | | | | Clock changes while apt is running can result in strange reports confusing (and amusing) users. Sadly, to keep the ABI for now the code is a bit more ugly than it would need to be.
* Remove obsolete RCS keywordsGuillem Jover2018-05-071-1/+0
| | | | Prompted-by: Jakub Wilk <jwilk@debian.org>
* ensure correct file permissions for auxfilesDavid Kalnischkies2018-02-191-0/+2
| | | | | | | | | | | | | | The interesting takeaway here is perhaps that 'chmod +w' is effected by the umask – obvious in hindsight of course. The usual setup helps with hiding that applying that recursively on all directories (and files) isn't correct. Ensuring files will not be stored with the wrong permissions even if in strange umask contexts is trivial in comparison. Fixing the test also highlighted that it wasn't bulletproof as apt will automatically fix the permissions of the directories it works with, so for this test we actually need to introduce a shortcut in the code. Reported-By: Ubuntu autopkgtest CI
* allow the apt/lists/auxfiles/ directory to be missingDavid Kalnischkies2018-01-191-17/+35
| | | | | | | | | | apt 1.6~alpha6 introduced aux requests to revamp the implementation of a-t-mirror. This already included the potential of running as non-root, but the detection wasn't complete resulting in errors or could produce spurious warnings along the way if the directory didn't exist yet. References: ef9677831f62a1554a888ebc7b162517d7881116 Closes: 887624
* simplify translating "lists directory missing" msgDavid Kalnischkies2018-01-041-6/+6
| | | | | | | | The appended "partial" should not be translated, but some translations got this wrong and now that there is also "auxfiles" we can just fix that problem by hiding these untranslatables from the translators. Gbp-Dch: Ignore
* require methods to request AuxRequest capability at startupDavid Kalnischkies2018-01-031-5/+18
| | | | | | Allowing a method to request work from other methods is a powerful capability which could be misused or exploited, so to slightly limited the surface let method opt-in into this capability on startup.
* allow a method to request auxiliary filesDavid Kalnischkies2018-01-031-15/+25
| | | | | | | | | | | | | | | | | | | | If a method needs a file to operate like e.g. mirror needs to get a list of mirrors before it can redirect the the actual requests to them. That could easily be solved by moving the logic into libapt directly, but by allowing a method to request other methods to do something we can keep this logic contained in the method and allow e.g. also methods which perform binary patching or similar things. Previously they would need to implement their own acquire system inside the existing one which in all likelyhood will not support the same features and methods nor operate with similar security compared to what we have already running 'above' the requesting method. That said, to avoid methods producing conflicts with "proper" files we are downloading a new directory is introduced to keep the auxiliary files in. [The message magic number 351 is a tribute to the german Grundgesetz article 35 paragraph 1 which defines that all authorities of the state(s) help each other on request.]
* remove pointless APT_PURE from void functionsDavid Kalnischkies2017-12-141-3/+3
| | | | | | | | | | | Earlier gcc versions used to complain that you should add them althrough there isn't a lot of point to it if you think about it, but now gcc (>= 8) complains about the attribute being present. warning: ‘pure’ attribute on function returning ‘void’ [-Wattributes] Reported-By: gcc -Wattributes Gbp-Dch: Ignore
* deal with floats without old-style castDavid Kalnischkies2017-12-131-6/+7
| | | | | | | | We have no speed problem with handling floats/doubles in our progress handling, but that shouldn't prevent us from cleaning up the handling slightly to avoid unclean casting to ints. Reported-By: gcc -Wdouble-promotion -Wold-style-cast
* Replace APT_CONST with APT_PURE everywhereJulian Andres Klode2017-08-241-3/+3
| | | | | As a follow up to the last commit, let's replace APT_CONST with APT_PURE everywhere to clean stuff up.
* don't hang if multiple sources use unavailable methodDavid Kalnischkies2017-08-041-3/+6
| | | | | | | | | | | | | | | APT clients always noticed if a method isn't supported and nowadays generate a message of the form: E: The method driver …/foobar could not be found. N: Is the package apt-transport-foobar installed? This only worked if a single source was using such an unavailable method through as we were registering the failed config the first round and the second would try to send requests to the not started method, which wouldn't work and hang instead (+ hiding the error messages as they would be shown only at the end of the execution). Closes: 870675
* allow the auth.conf to be root:root ownedDavid Kalnischkies2017-07-261-15/+0
| | | | | | | | | | | Opening the file before we drop privileges in the methods allows us to avoid chowning in the acquire main process which can apply to the wrong file (imagine Binary scoped settings) and surprises users as their permission setup is overridden. There are no security benefits as the file is open, so an evil method could as before read the contents of the file, but it isn't worse than before and we avoid permission problems in this setup.
* send weak-only hashes to methodsDavid Kalnischkies2017-07-261-15/+8
| | | | | | | | | | Weak hashes like filesize can be used by methods for basic checks and early refusals even if we can't use them for hard security proposes. Normal apt operations are not affected by this as they fail if no strong hash is available, but if apt is forced to work with weak-only files or e.g. in apt-helper context it can have benefits as weak is better than no hash for the methods.
* Reformat and sort all includes with clang-formatJulian Andres Klode2017-07-121-13/+13
| | | | | | | | | | | | | This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
* allow frontends to override releaseinfo change behaviourDavid Kalnischkies2017-06-281-1/+30
| | | | | | | | Having messages being printed on the error stack and confirm them by commandline flags is an okayish first step, but some frontends will probably want to have a more interactive feeling here with a proper question the user can just press yes/no for as for some frontends a commandline flag makes no sense…
* Avoid chdir in acquire clean with unlinkatDavid Kalnischkies2017-06-261-32/+24
| | | | | | | | | | | | POSIX.1-2008 gives us a range of *at calls to deal with files including the unlinkat so we can remove a file from a directory based on a path to the file relative to the directory. (In our case here the path we have is just the filename) We avoid changing directories in this way which e.g. fails if the directory we started in no longer exists or is otherwise inaccessible. Closes: 860738
* Only merge acquire items with the same meta keyJulian Andres Klode2017-01-281-1/+17
| | | | | | | | | | | | | | | Since the introduction of by-hash, two differently named files might have the same real URL. In our case, the files icons-64x64.tar.gz and icons-128x128.tar.gz of empty tarballs. APT would try to merge them and end with weird errors because it completed the first download and enters the second stage for decompressing and verifying. After that it would queue a new item to copy the original file to the location, but that copy item would be in the wrong stage, causing it to use the hashes for the decompressed item. Closes: #838441
* remove 'old' FAILED files in the next acquire callDavid Kalnischkies2017-01-191-0/+6
| | | | | | | | | | | | | | | If apt renames a file to .FAILED it leaves its namespace and is never touched again – expect since 1.1~exp4 in which "apt clean" will remove those files. The usefulness of these files rapidly degrades if you don't keep the update log itself (together with debug output in the best case) through and on 99% of all system they will be kept around forever just to collect dust over time and eat up space. With this commit an update call will remove all FAILED files of previous runs, so that the FAILED files you have on disk are always only the ones related to the last apt run stopping apt from hoarding files. Closes: 846476
* reword "Can't drop priv" warning messageDavid Kalnischkies2016-12-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | Note: This is a warning about disabling a security feature. It is supposed to be scary as we are disabling a security feature and we can't just be silent about it! Downloads really shouldn't happen any longer as root to decrease the attack surface – but if a warning causes that much uproar, consider what an error would do… The old WARNING message: | W: Can't drop privileges for downloading as file 'foobar' couldn't be | accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) is frequently (incorrectly) considered to be an error message indicating that the download didn't happen which isn't the case, it was performed, but without all the security features enabled we could have used if run from some other place… The word "unsandboxed" is chosen as the term 'sandbox(ed)' is a common encounter in feature lists/changelogs and more people are hopefully able to make the connection to 'security' than it is the case for 'privilege dropping' which is more correct, but far less known. Closes: #813786 LP: #1522675
* get pdiff files from the same mirror as the indexDavid Kalnischkies2016-11-251-1/+1
| | | | | | | | | | | In ad9416611ab83f7799f2dcb4bf7f3ef30e9fe6f8 we fall back to asking the original mirror (e.g. a redirector) if we do not get the expected result. This works for the indexes, but patches are a different beast and much simpler. Adding this fallback code here seems like overkill as they are usually right along their Index file, so actually forward the relevant settings to the patch items which fixes pdiff support combined with a redirector and partial mirrors as in such a situation the pdiff patches would be 404 and the complete index would be downloaded.
* acquire: Use priority queues and a 3 stage pipeline designJulian Andres Klode2016-09-021-4/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Employ a priority queue instead of a normal queue to hold the items; and only add items to the running pipeline if their priority is the same or higher than the priority of items in the queue. The priorities are designed for a 3 stage pipeline system: In stage 1, all Release files and .diff/Index files are fetched. This allows us to determine what files remain to be fetched, and thus ensures a usable progress reporting. In stage 2, all Pdiff patches are fetched, so we can apply them in parallel with fetching other files in stage 3. In stage 3, all other files are fetched (complete index files such as Contents, Packages). Performance improvements, mainly from fetching the pdiff patches before complete files, so they can be applied in parallel: For the 01 Sep 2016 03:35:23 UTC -> 02 Sep 2016 09:25:37 update of Debian unstable and testing with Contents and appstream for amd64 and i386, update time reduced from 37 seconds to 24-28 seconds. Previously, apt would first download new DEP11 icon tarballs and metadata files, causing the CPU to be idle. By fetching the diffs in stage 2, we can now patch our contents and Packages files while we are downloading the DEP11 stuff.
* Merge branch 'portability/freebsd'Julian Andres Klode2016-08-271-3/+3
|\
| * Make root group configurable via ROOT_GROUPJulian Andres Klode2016-08-261-2/+2
| | | | | | | | | | This is needed on BSD where root's default group is wheel, not root.
| * Use C locale instead of C.UTF-8 for protocol stringsJulian Andres Klode2016-08-261-1/+1
| | | | | | | | | | | | The C.UTF-8 locale is not portable, so we need to use C, otherwise we crash on other systems. We can use std::locale::classic() for that, which might also be a bit cheaper than using locale("C").
* | improve code & doc for aquire weak/loop failingDavid Kalnischkies2016-08-241-6/+6
| | | | | | | | | | | | Improve-Upon: 2e2865ae53a65c00dd55a892d5b48458f3110366 Reported-By: Julian Andres Klode Gbp-Dch: Ignore
* | do fail on weakhash/loop earlier in acquireDavid Kalnischkies2016-08-241-3/+56
|/ | | | | | | | | | | | | | | | | The bugreport shows a segfault caused by the code not doing the correct magical dance to remove an item from inside a queue in all cases. We could try hard to fix this, but it is actually better and also easier to perform these checks (which cause instant failure) earlier so that they haven't entered queue(s) yet, which in return makes cleanup trivial. The result is that we actually end up failing "too early" as if we wouldn't be careful download errors would be logged before that process was even started. Not a problem for the acquire system, but likely to confuse users and programs alike if they see the download process producing errors before apt was technically allowed to do an acquire (it didn't, so no violation, but it looks like it to the untrained eye). Closes: 835195
* prevent C++ locale number formatting in text APIs (try 3)David Kalnischkies2016-08-231-2/+5
| | | | | | | | | | | This time it is the formatting of floating numbers in progress reporting with a radix charater potentially not being dot. Followup of 7303e11ff28f920a6277c159aa46f80c007350bb. Regression of b58e2c7c56b1416a343e81f9f80cb1f02c128e25 in so far as it exchanging very effected with slightly less effected code. LP: 1611010
* prevent C++ locale number formatting in text APIsDavid Kalnischkies2016-05-271-7/+3
| | | | | | | | | | | Setting the C++ locale via std::locale::global(std::locale("")); which would otherwise default to the default C locale (aka: unaffected by setlocale) effects the formatting of numeric types in IO streams, which for output for humans is perfectly sensible, but breaks our many text interfaces used and parsed by us and others without expecting the numbers to be formatted. Closes: #825396
* fix and document on the fly compressor configDavid Kalnischkies2016-05-271-1/+1
| | | | | | | libapt allows to configure compressors to be used by its system via configuration implemented in 03bef78461c6f443187b60799402624326843396, but that was never really documented and also only partly working, which also explains why the tests weren't using it…
* delay progress until Release files are downloadedDavid Kalnischkies2016-05-071-8/+5
| | | | | | | | | | | | Progress reporting used an "upper bound" on files we might get, expect that this wasn't correct in case pdiff entered the picture. So instead of calculating a value which is perhaps incorrect, we just accept that we can't tell how many files we are going to download and just keep at 0% until we know. Additionally, if we have pdiffs we wait until we got these (sub)index files, too. That could all be done better by downloading all Release files first and planing with them in hand accordingly, but one step at a time.
* make random acquire queues work less randomDavid Kalnischkies2016-04-251-8/+22
| | | | | | | | | | | | | | Queues feeding workers like rred are created in a random pattern to get a few of them to run in parallel – but if we already have an idling queue we don't need to assign it to a (potentially new) random queue as that saves us the (agruably small) overhead of starting up a new queue, avoids adding jobs to an already busy queue while others idle and as a bonus reduces the size of debug logs a bit. We also keep starting new queues now until we reach our limit before we assign work at random to them, which should give us a more effective utilisation overall compared to potentially adding work to busy queues while we haven't reached our queue limit yet.
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-04 18:17:17 +0000