apt/apt-pkg/acquire-worker.cc, branch 2.9.22

Allow HashSum mismatches to fallback to other mirrors

2024-11-22T12:40:10Z

Perhaps a common error in the past nowadays thanks to by-hash we rarely get mismatches due to our calling behaviour and more because the mirror is genuinely bad especially if this mirror didn't work with by-hash. Using our "normal" error handling allows falling back to other mirrors which hopefully do support by-hash and/or are not broken on a filesystem level making us more resistent to the occasional bad apple in a wild mirror forest.

Never consider file/copy/cdrom sources bad sites due to errors

2024-11-22T12:40:10Z

As the hostname for them is likely empty (for absolute or . or .. for relative paths) considering one a bad site causes all of the sites from this scheme to be considered bad. In a perfect world, we would figure out a good site-path to use for these sources, but as they are local sources which means we don't have to perform costly requests to the internet we just let them happen instead for now.

Clear previous ignored errors before processing acquire item again

2024-11-22T12:40:10Z

If an item failed previously, but that failure is ignored and a retry is triggered we might end up in a situation in which it is failing yet again for perhaps totally different reasons, but the error message displayed is the one we have shown (or, because it was ignored likely hidden) before which can be rather confusing in debugging.

Allow AutoDetectProxy to work with more than just http(s)

2024-11-22T12:40:10Z

It is a bit unfair for third-party methods wrapping e.g. http to not have access to such tools, same for our old ftp and e.g. our tor even if in practice this isn't used much even for http. At least that is one less FIXME in the code.

Show hashes calculated by main thread in error messages

2024-11-22T12:21:24Z

The main thread calculates the hash sums if the method hasn't to be able to verify the files, but while a failure is detected and reported, the error messages did not include the hash sums which looks at least puzzling. So to have a complete error message we ensure that the hashes we calculated are appended to the message from the method so that it looks for the higher level code as if the method had calculated the hashes.

Support uncompressed indexes from partial file:/ mirrors

2024-11-22T12:21:24Z

If a file:/ mirror does not contain the requested compressed file, but the uncompressed variant of that file the method will sent us a fitting Done message with data only about the Alt-Filename, but the central hash verification code in the acquire system does not expect that looking for the non-existent compressed file to compare its hashes with the expected result – which fails (if we are verifying at least). That file:/ as well as the item code deals with this suggests that this feature was broken by 448c38bdcd72b52f11ec5f326f822cf57653f81c, but I couldn't (easily) compile that to verify, so this is only a guess. Supporting this allows using index files from /var/lib/apt/lists – which are likely uncompressed or can at least made so easily – to construct a file:/ mirror.

Handle Retry-After by default

2024-11-18T13:54:19Z

Let's get some feedback here. This is capped to 30s per retry, so a maximum of 90s per item that we delay the failure case. In the worst case, we hence have 90n seconds delay for n items. In a more realistic worse case however, the server likely does not collapse per-item, and we end up trying all items in a short burst so it seems more realistic we hit a 90s total delay. The bursty behavior may be suboptimal; we don't quite know, but it seems the specification of Retry-After is per-entity rather than per-host, so technically speaking it seems correct.

http: use Retry-After HTTP header to optimize retries

2024-11-18T12:44:49Z

Some mirrors like snapshot.debian.org apply strict http rate limits. While apt already has an exponential backoff mechanism implemented, this is not sufficient due to the following reasons: 1. all retries happen roughly at the same time 2. the retry-after information from the server is not used We fix this by improving the algorithm: First, if present, the timestamp or duration of the Retry-After header is added to the exponential backoff, optimizing for success-on-second-try. Second, a random delay is added (delay += [0, delay]) to distribute the retries to not immediately run into the limit again. To avoid super-long delays, this is capped by the configured maximum delay. This logic only becomes active if the option Acquire::Retries::HandleRetryAfter is enabled. Signed-off-by: Felix Moessbauer

Remove ftp, rsh, ssh methods (disabled since 1.8)

2024-11-07T13:04:30Z

These methods have been unsupported and disabled since 1.8, remove them for the 3.0 cleanup. Please migrate to http instead. If you need ad-hoc access to a remote repository, you can run `python3 -m http.server` on that machine and use SSH port forwarding to run http over ssh.

worker: Add an audit level to log audit messages

2024-07-15T17:16:27Z