summaryrefslogtreecommitdiff
path: root/apt-pkg/acquire.cc
Commit message (Collapse)AuthorAgeFilesLines
* pkgAcqMetaClearSig: Abort transaction when pkgAcquire::Run has been cancelledSebastian Krzyszkowiak2026-05-011-0/+4
| | | | | | | | | | | | Ongoing Run can be cancelled by returning false from ::Pulse(). In this case the items would still be called pkgAcquire::Item::Finished() as usual, causing pkgAcqMetaClearSig to commit the incomplete transaction anyway as at that point it's still Started and has no errors. Add a pkgAcquire::Item::Cancelled() method and use it in pkgAcqMetaClearSig to abort the transaction. Closes: #1078608
* apt: modernize to make_unique C++17Herman Semenoff2026-04-081-1/+1
| | | | | | | References: - https://stackoverflow.com/questions/79700634/pros-and-cons-of-make-unique-vs-direct-constructor-call-in-c17 - https://towardsdev.com/why-std-make-unique-beats-new-in-modern-c-7e2ba653737e - https://www.sololearn.com/en/Discuss/3334779/where-make_unique-is-better-than-unique_ptrraw-pointer
* apt-pkg/acquire: use range based for loop C++17Herman Semenoff2026-04-081-14/+14
|
* Move "Tried to start delayed item […], but failed" to debuggingJulian Andres Klode2025-10-041-4/+2
| | | | | | | | We don't quite know why a stuck queue cannot get unstuck here, and it's possible that this is causing the queue to stall, but shrug. Closes: #1111554
* Fix stuck acquire queues on partial server errorsJulian Andres Klode2025-06-241-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a server responds with two InRelease files, but one is good, and the other indicates a temporary error, we queued the other indices from the good repository for downloading and then queued the retry. The resulting queue ended up with items having fetchAfter=0 before the item with fetchAfter=<something>, causing the Run() loop to not detect a stuck queue. Change the order of the queue such that the highest retry-after items comes first; this ensures that we always see stuck queues. This of course changes the behavior of retries in that if one file fails temporarily we block the entire server. This does seem more beneficial in the common case - if one file fails, probably all of them fail, and there's no point bombarding the server with requests for indices from good repositories until all have failed. The actual root cause is more that the remaining items are Enqueued like this: 1. Enqueue jammy/InRelease (delayed) 2. Enqueue jammy-updates/main all Packages The resulting queue ended up being: jammy-updates/main all Packages jammy/InRelease (delayed) But Enqueue() only calls Cycle() when there are no items in the queue already - after all, any item that is already running will call Cycle() eventually. Or so was the case until we added the retry-after handling. It's unclear why we don't Cycle() all the time when enqueuing a new item, given that our pipeline might not be filled yet, and we could send the request to the server while waiting for data on a running item. Trying to always Cycle() however led to regressions that still need investigating. Given that, this solution certainly is the more easy to reason about one. LP: #2003851
* Include standard headers to fix clang compiler errorBiswapriyo Nath2025-06-181-0/+1
| | | | | | | | This commit fixes the following compiler errors. apt-pkg/acquire.cc:833:51: error: no template named 'function' in namespace 'std' apt-pkg/contrib/error.cc:198:59: error: no member named 'front_inserter' in namespace 'std' apt-pkg/solver3.h:44:22: error: no template named 'is_trivially_constructible_v' in namespace 'std' methods/http.cc:1029:24: error: no member named 'inserter' in namespace 'std'
* Rewrite hidden config option checker for more patternsDavid Kalnischkies2024-11-221-1/+1
| | | | | | | | | | | | | The previous version allowed only foo::*::bar as a pattern, now you can have as many starred parts as you want allowing for a finer control instead of hardcoding specific common star-values or giving up with **. It also adds a new INVALID type for an option to mark options which should not be used, but are covered by a pattern which would normally allow it. Our tests are more or less the only users of this hidden feature making this change not very risky even through it has no dedicated test itself.
* Turn unique_ptr<decltype(fclose and free)> into real deleter types (warnings ↵наб2024-11-121-1/+1
| | | | now, UB in C++20)
* Modernize standard library includesJulian Andres Klode2024-02-201-4/+4
| | | | | | This was automated with sed and git-clang-format, and then I had to fix up the top of policy.cc by hand as git-clang-format accidentally indented it by two spaces.
* Improve and test distclean implementationDavid Kalnischkies2024-01-031-13/+11
| | | | | | | | | | | | | The implementation as-is as various smaller/esoteric bugs and inconsistencies like apt-get not supporting them, the option -s being supported in code but not accepted on the command line, the regex not escaping the dot before the file extension and exposing more implementation details to public headers than we actually need. Also comes with a small test case to ensure it actually works. References: bd7c126e3fb1b94e76e0e632c657cea854586844
* Add 'dist-clean' command to remove packages and list filesGábor Németh2023-11-221-13/+41
| | | | | | | We assume all files in the 'listsdir' are candidates. Keep only files ending with Release, Release.gpg, and InRelease. Closes: #959093
* Avoid comparison of integers of different signedness for ETADavid Kalnischkies2023-01-291-1/+2
| | | | | Reported-By: gcc -Wsign-compare Gbp-Dch: Ignore
* typecast time_t and suseconds_t from std::chronoKhem Raj2022-09-161-2/+2
| | | | | | | | This fixes build on some architectures like mips progress.cc:125:31: error: non-constant-expression cannot be narrowed from type 'std::chrono::duration<long long>::rep' (aka 'long long') to '__time_t' (aka 'long') in initializer list [-Wc++11-narrowing] struct timeval NowTime = { Now_sec.count(), Now_usec.count() }; Signed-off-by: Khem Raj <raj.khem@gmail.com>
* Improve error handling of cycling delayed queuesJulian Andres Klode2021-09-061-5/+8
| | | | | | | | | When an item has been delayed and the queue is cycled to start it, we did not properly report an error from the cycling, and we would then fail in the assert(), causing all errors to be lost. Propagate the error instead and make the assert a warning.
* Properly convert PulseInterval to timevalJulian Andres Klode2021-07-281-6/+3
| | | | | Fix the typo, and use the helper function to convert it, so we do not end up with 5 seconds encoded as 0s and 5*10^6 microseconds.
* Convert to monotonic clockJulian Andres Klode2021-07-281-14/+18
| | | | | This yields more accurate delays and avoids issues with clock skew.
* acquire: Automatically reduce select() timeout as neededJulian Andres Klode2021-07-281-1/+35
| | | | | | | | | | | | | | If there is an item with fetchAfter at the top of a queue, reduce sleep() timeout so we can detect it and start it, by calling Cycle() on the queue in the next iteration. For some reasons we have to call select() with a 0s timeout if we just marked an item as ready. Oh well. Previous versions of this patch only called global Bump() after a timeout from select(); this was unfortunately incorrect - it meant that we never bumped a queue that did not start yet while other queues were running, potentially significantly delaying retries.
* Implement exponential delay between retriesJulian Andres Klode2021-07-281-3/+22
| | | | | | | Add a new Item field called FetchAfter, which determines the earliest time the item should be fetched at. Adjust insertion into queue to take it into account alongside priority, and only fill pipelines with items that are ready.
* Keep URIs encoded in the acquire systemDavid Kalnischkies2020-12-181-6/+40
| | | | | | | | | | | | | | | | | | | We do not deal a lot with URIs which need encoding, but then we do it is a pain that we store it decoded in the acquire system as it means we have to decode and reencode URIs eventually which is potentially giving us slightly different URIs. We see that in our own testing framework while setting up redirects as the config options are effectively double-encoded and decoded to pass them around successfully as otherwise %2f and / in an URI are treated the same. This commit adds the infrastructure for methods to opt into getting URIs send in encoded form (and returning them to us in encoded form, too) so that we eventually do not have to touch the URIs which is how it should be. This means though that we have to deal with methods who do not support this yet (aka: all at the moment) for which we decode and encode while communicating with them.
* Do not hardcode (wrong) group and mode in setup warningDavid Kalnischkies2020-06-061-3/+3
| | | | | | | | | | | | | | Partial directories are created with 0700, but the parent is 0755, while the error message would report 0700 for both… that isn't right and can be pretty confusing. Turns out that the messages aren't marked for translation, so no unfuzzing is required & we just leave it as untranslated for now. Especially as the more detailed error strings derived from errno are translated. Reported-By: Wakko Warner <wakko@animx.eu.org> Closes: #962310
* Fix small memory leak in MethodConfigDavid Kalnischkies2020-05-271-1/+1
| | | | | | | | | We are leaking a d-pointer currently weighting a boolean in size and MethodConfig is instantiated in small numbers only, so nobody will actually notice a difference, but proper cleanup is important. Reported-By: clang LeakSanitizer References: 04ab37fecaf286f724bef2e0969d2b67ab5ac1b1
* Merge pkgAcquire::RunFdsSane back into RunFdsJulian Andres Klode2020-02-261-9/+3
|
* acquire: Move queue startup after calling log's Start()Julian Andres Klode2020-01-081-4/+4
| | | | | | | | | | Startup() was checking for bad items and failing them, but we did not actually call Start() in the log, so the log might not be setup correctly. This caused a crash in python-apt when items were being failed on queue startup, as it released the GIL when Start() is being called and re-acquires it when running callbacks.
* Fix some style warnings from cppcheckDavid Kalnischkies2019-11-261-2/+0
| | | | | | | | Unused variable, std::algorithms instead of raw for-loops. There should be no observeable difference in behaviour. Reported-By: cppcheck Gbp-Dch: Ignore
* Fix segfault in pkgAcquire::Enqueue() with Acquire::Queue-Mode=accessJulian Andres Klode2019-08-151-4/+6
| | | | | | | | | | | | | | | | | In commit 79b1a8298, QueueName() was changed, amongst other things, to exit early when the queue mode was single access, as single access does not need any fancy queue name. The exit became too early though, as Config was not initialized anymore, but the caller was relying on it. Fix QueueName() to always initialize Config and in Enqueue() initialize Config with a nullptr, so if this regresses it's guaranteed to fail harder. Also add a test case - this is very simple, but the first and only test case for access queue mode. Regression-Of: 79b1a82983e737e74359bc306d9edb357c5bdd46 LP: #1839714
* Distribute host-less work based on backlog of the queuesDavid Kalnischkies2019-07-081-44/+66
| | | | | | | | | | | | | | | | | Work like applying patches via rred can be performed by many concurrent rred processes, but we can't just spawn new ones forever: We limit us to the number of CPUs which can drive them and reuse existing ones if they have nothing to do at the moment. The problem arises if we have reached the limit of queues and all of them are busy which is more likely to happen on "slow" machines with few CPUs. In this case we opted for random distribution, but that can result in many big files (e.g. Contents) being added to one queue while the others get none or only small files. Ideally we would ask the methods how much they still have to do, but they only know that for the current item, not for all items in the queue, so we use the filesize of the expected result.
* apt-pkg: URI: Add 'explicit' to single argument constructorJulian Andres Klode2019-04-301-1/+1
| | | | | This needs a fair amount of changes elsewhere in the code, hence this is separate from the previous commits.
* acq: worker: Move CurrentSize, TotalSize, ResumePoint to CurrentItemJulian Andres Klode2019-04-301-3/+3
| | | | | These status fields belong to the current item, move them there. This prepares us for eventually having multiple current items.
* Don't limit cpu-limited queues to at most 10David Kalnischkies2019-04-161-5/+7
| | | | | | | | | | | Queues for processes like rred are not created by hostname but we spawn at most CPU*2 queues to place items in. The problem is that we then proceeded to limit it to at most 10 queues (via QueueHost::Limit) again at the end of the method so that all items (after the first 10 queues are busy) are forcibly placed into a generic catch-all instance which is bad because we don't keep all CPUs we have available busy and worse we end up sheduling the most work to a single one while random distribution was intended.
* acquire: Remove deprecated pkgAcquire::Setup() functionJulian Andres Klode2019-02-261-19/+0
|
* acquire: Fold pkgAcquireStatus2 into pkgAcquireStatusJulian Andres Klode2019-02-261-9/+2
| | | | Clean up the code, make it neat, lalala
* Fix calculation of elapsed usec in downloadsJulian Andres Klode2018-09-241-2/+2
| | | | | | A recent change to use chronos inadvertently replaced the difference of new usec - old usec with new sec - old usec, which is obviously wrong.
* Use a steady clock source for progress reportingDavid Kalnischkies2018-05-281-27/+35
| | | | | | Clock changes while apt is running can result in strange reports confusing (and amusing) users. Sadly, to keep the ABI for now the code is a bit more ugly than it would need to be.
* Remove obsolete RCS keywordsGuillem Jover2018-05-071-1/+0
| | | | Prompted-by: Jakub Wilk <jwilk@debian.org>
* ensure correct file permissions for auxfilesDavid Kalnischkies2018-02-191-0/+2
| | | | | | | | | | | | | | The interesting takeaway here is perhaps that 'chmod +w' is effected by the umask – obvious in hindsight of course. The usual setup helps with hiding that applying that recursively on all directories (and files) isn't correct. Ensuring files will not be stored with the wrong permissions even if in strange umask contexts is trivial in comparison. Fixing the test also highlighted that it wasn't bulletproof as apt will automatically fix the permissions of the directories it works with, so for this test we actually need to introduce a shortcut in the code. Reported-By: Ubuntu autopkgtest CI
* allow the apt/lists/auxfiles/ directory to be missingDavid Kalnischkies2018-01-191-17/+35
| | | | | | | | | | apt 1.6~alpha6 introduced aux requests to revamp the implementation of a-t-mirror. This already included the potential of running as non-root, but the detection wasn't complete resulting in errors or could produce spurious warnings along the way if the directory didn't exist yet. References: ef9677831f62a1554a888ebc7b162517d7881116 Closes: 887624
* simplify translating "lists directory missing" msgDavid Kalnischkies2018-01-041-6/+6
| | | | | | | | The appended "partial" should not be translated, but some translations got this wrong and now that there is also "auxfiles" we can just fix that problem by hiding these untranslatables from the translators. Gbp-Dch: Ignore
* require methods to request AuxRequest capability at startupDavid Kalnischkies2018-01-031-5/+18
| | | | | | Allowing a method to request work from other methods is a powerful capability which could be misused or exploited, so to slightly limited the surface let method opt-in into this capability on startup.
* allow a method to request auxiliary filesDavid Kalnischkies2018-01-031-15/+25
| | | | | | | | | | | | | | | | | | | | If a method needs a file to operate like e.g. mirror needs to get a list of mirrors before it can redirect the the actual requests to them. That could easily be solved by moving the logic into libapt directly, but by allowing a method to request other methods to do something we can keep this logic contained in the method and allow e.g. also methods which perform binary patching or similar things. Previously they would need to implement their own acquire system inside the existing one which in all likelyhood will not support the same features and methods nor operate with similar security compared to what we have already running 'above' the requesting method. That said, to avoid methods producing conflicts with "proper" files we are downloading a new directory is introduced to keep the auxiliary files in. [The message magic number 351 is a tribute to the german Grundgesetz article 35 paragraph 1 which defines that all authorities of the state(s) help each other on request.]
* remove pointless APT_PURE from void functionsDavid Kalnischkies2017-12-141-3/+3
| | | | | | | | | | | Earlier gcc versions used to complain that you should add them althrough there isn't a lot of point to it if you think about it, but now gcc (>= 8) complains about the attribute being present. warning: ‘pure’ attribute on function returning ‘void’ [-Wattributes] Reported-By: gcc -Wattributes Gbp-Dch: Ignore
* deal with floats without old-style castDavid Kalnischkies2017-12-131-6/+7
| | | | | | | | We have no speed problem with handling floats/doubles in our progress handling, but that shouldn't prevent us from cleaning up the handling slightly to avoid unclean casting to ints. Reported-By: gcc -Wdouble-promotion -Wold-style-cast
* Replace APT_CONST with APT_PURE everywhereJulian Andres Klode2017-08-241-3/+3
| | | | | As a follow up to the last commit, let's replace APT_CONST with APT_PURE everywhere to clean stuff up.
* don't hang if multiple sources use unavailable methodDavid Kalnischkies2017-08-041-3/+6
| | | | | | | | | | | | | | | APT clients always noticed if a method isn't supported and nowadays generate a message of the form: E: The method driver …/foobar could not be found. N: Is the package apt-transport-foobar installed? This only worked if a single source was using such an unavailable method through as we were registering the failed config the first round and the second would try to send requests to the not started method, which wouldn't work and hang instead (+ hiding the error messages as they would be shown only at the end of the execution). Closes: 870675
* allow the auth.conf to be root:root ownedDavid Kalnischkies2017-07-261-15/+0
| | | | | | | | | | | Opening the file before we drop privileges in the methods allows us to avoid chowning in the acquire main process which can apply to the wrong file (imagine Binary scoped settings) and surprises users as their permission setup is overridden. There are no security benefits as the file is open, so an evil method could as before read the contents of the file, but it isn't worse than before and we avoid permission problems in this setup.
* send weak-only hashes to methodsDavid Kalnischkies2017-07-261-15/+8
| | | | | | | | | | Weak hashes like filesize can be used by methods for basic checks and early refusals even if we can't use them for hard security proposes. Normal apt operations are not affected by this as they fail if no strong hash is available, but if apt is forced to work with weak-only files or e.g. in apt-helper context it can have benefits as weak is better than no hash for the methods.
* Reformat and sort all includes with clang-formatJulian Andres Klode2017-07-121-13/+13
| | | | | | | | | | | | | This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
* allow frontends to override releaseinfo change behaviourDavid Kalnischkies2017-06-281-1/+30
| | | | | | | | Having messages being printed on the error stack and confirm them by commandline flags is an okayish first step, but some frontends will probably want to have a more interactive feeling here with a proper question the user can just press yes/no for as for some frontends a commandline flag makes no sense…
* Avoid chdir in acquire clean with unlinkatDavid Kalnischkies2017-06-261-32/+24
| | | | | | | | | | | | POSIX.1-2008 gives us a range of *at calls to deal with files including the unlinkat so we can remove a file from a directory based on a path to the file relative to the directory. (In our case here the path we have is just the filename) We avoid changing directories in this way which e.g. fails if the directory we started in no longer exists or is otherwise inaccessible. Closes: 860738
* Only merge acquire items with the same meta keyJulian Andres Klode2017-01-281-1/+17
| | | | | | | | | | | | | | | Since the introduction of by-hash, two differently named files might have the same real URL. In our case, the files icons-64x64.tar.gz and icons-128x128.tar.gz of empty tarballs. APT would try to merge them and end with weird errors because it completed the first download and enters the second stage for decompressing and verifying. After that it would queue a new item to copy the original file to the location, but that copy item would be in the wrong stage, causing it to use the hashes for the decompressed item. Closes: #838441
* remove 'old' FAILED files in the next acquire callDavid Kalnischkies2017-01-191-0/+6
| | | | | | | | | | | | | | | If apt renames a file to .FAILED it leaves its namespace and is never touched again – expect since 1.1~exp4 in which "apt clean" will remove those files. The usefulness of these files rapidly degrades if you don't keep the update log itself (together with debug output in the best case) through and on 99% of all system they will be kept around forever just to collect dust over time and eat up space. With this commit an update call will remove all FAILED files of previous runs, so that the FAILED files you have on disk are always only the ones related to the last apt run stopping apt from hoarding files. Closes: 846476
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2026-07-18 21:32:10 +0000