| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
Benchmark:
- https://stackoverflow.com/questions/62058906/why-my-performance-benchmark-gives-me-wrong-results
References:
- https://clang.llvm.org/extra/clang-tidy/checks/performance/prefer-single-char-overloads.html
|
| |
|
|
|
|
|
|
|
|
|
| |
Perhaps a common error in the past nowadays thanks to by-hash we rarely
get mismatches due to our calling behaviour and more because the mirror
is genuinely bad especially if this mirror didn't work with by-hash.
Using our "normal" error handling allows falling back to other mirrors
which hopefully do support by-hash and/or are not broken on a filesystem
level making us more resistent to the occasional bad apple in a wild
mirror forest.
|
| |
|
|
|
|
|
|
|
|
|
| |
As the hostname for them is likely empty (for absolute or . or .. for
relative paths) considering one a bad site causes all of the sites from
this scheme to be considered bad.
In a perfect world, we would figure out a good site-path to use for
these sources, but as they are local sources which means we don't have
to perform costly requests to the internet we just let them happen
instead for now.
|
| |
|
|
|
|
|
|
| |
If an item failed previously, but that failure is ignored and a retry is
triggered we might end up in a situation in which it is failing yet
again for perhaps totally different reasons, but the error message
displayed is the one we have shown (or, because it was ignored likely
hidden) before which can be rather confusing in debugging.
|
| |
|
|
|
|
|
|
| |
It is a bit unfair for third-party methods wrapping e.g. http to not
have access to such tools, same for our old ftp and e.g. our tor even
if in practice this isn't used much even for http.
At least that is one less FIXME in the code.
|
| |
|
|
|
|
|
|
|
|
|
| |
The main thread calculates the hash sums if the method hasn't to be able
to verify the files, but while a failure is detected and reported, the
error messages did not include the hash sums which looks at least
puzzling.
So to have a complete error message we ensure that the hashes we
calculated are appended to the message from the method so that it looks
for the higher level code as if the method had calculated the hashes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a file:/ mirror does not contain the requested compressed file, but
the uncompressed variant of that file the method will sent us a fitting
Done message with data only about the Alt-Filename, but the central hash
verification code in the acquire system does not expect that looking for
the non-existent compressed file to compare its hashes with the expected
result – which fails (if we are verifying at least).
That file:/ as well as the item code deals with this suggests that this
feature was broken by 448c38bdcd72b52f11ec5f326f822cf57653f81c, but I
couldn't (easily) compile that to verify, so this is only a guess.
Supporting this allows using index files from /var/lib/apt/lists –
which are likely uncompressed or can at least made so easily – to
construct a file:/ mirror.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let's get some feedback here. This is capped to 30s per retry, so
a maximum of 90s per item that we delay the failure case. In the
worst case, we hence have 90n seconds delay for n items.
In a more realistic worse case however, the server likely does
not collapse per-item, and we end up trying all items in a short
burst so it seems more realistic we hit a 90s total delay.
The bursty behavior may be suboptimal; we don't quite know, but
it seems the specification of Retry-After is per-entity rather than
per-host, so technically speaking it seems correct.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some mirrors like snapshot.debian.org apply strict http rate limits.
While apt already has an exponential backoff mechanism implemented, this
is not sufficient due to the following reasons:
1. all retries happen roughly at the same time
2. the retry-after information from the server is not used
We fix this by improving the algorithm: First, if present, the timestamp
or duration of the Retry-After header is added to the exponential
backoff, optimizing for success-on-second-try. Second, a random delay is
added (delay += [0, delay]) to distribute the retries to not immediately
run into the limit again. To avoid super-long delays, this is capped by
the configured maximum delay. This logic only becomes active if the
option Acquire::Retries::HandleRetryAfter is enabled.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
|
| |
|
|
|
|
|
|
|
| |
These methods have been unsupported and disabled since 1.8, remove
them for the 3.0 cleanup.
Please migrate to http instead. If you need ad-hoc access to a remote
repository, you can run `python3 -m http.server` on that machine and
use SSH port forwarding to run http over ssh.
|
| | |
|
| |
|
|
|
|
| |
This was automated with sed and git-clang-format, and then I had to
fix up the top of policy.cc by hand as git-clang-format accidentally
indented it by two spaces.
|
| |
|
|
|
|
| |
This is very basic support on the testing side, we just test
the debug output but not how long it actually took. Would be
nice to check time really.
|
| |
|
|
|
| |
This yields more accurate delays and avoids issues with clock
skew.
|
| |
|
|
|
|
|
| |
Add a new Item field called FetchAfter, which determines the earliest
time the item should be fetched at. Adjust insertion into queue to
take it into account alongside priority, and only fill pipelines
with items that are ready.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We do not deal a lot with URIs which need encoding, but then we do it is
a pain that we store it decoded in the acquire system as it means we
have to decode and reencode URIs eventually which is potentially giving
us slightly different URIs.
We see that in our own testing framework while setting up redirects as
the config options are effectively double-encoded and decoded to pass
them around successfully as otherwise %2f and / in an URI are treated
the same.
This commit adds the infrastructure for methods to opt into getting URIs
send in encoded form (and returning them to us in encoded form, too) so
that we eventually do not have to touch the URIs which is how it should
be. This means though that we have to deal with methods who do not
support this yet (aka: all at the moment) for which we decode and encode
while communicating with them.
|
| | |
|
| |
|
|
|
| |
This needs a fair amount of changes elsewhere in the code,
hence this is separate from the previous commits.
|
| |
|
|
|
| |
These status fields belong to the current item, move them there. This
prepares us for eventually having multiple current items.
|
| |
|
|
|
|
|
| |
These methods are not supposed to be used anymore, they are
not actively maintained and may hence contain odd bugs.
Fixes !49
|
| |
|
|
|
|
|
|
|
|
|
| |
If we got a file but it produced a hash error, mismatched size or
similar we shouldn't fallback to alternative URIs as they likely result
in the same error. If we can we should instead use another mirror.
We used to be a lot stricter by stopping all trys for this file if we
got a non-404 (or a hash-based) failure, but that is too hard as we
really want to try other mirrors (if we have them) in the hope that they
have the expected and correct files.
|
| |
|
|
| |
Prompted-by: Jakub Wilk <jwilk@debian.org>
|
| |
|
|
|
| |
Reported-By: codespell & spellintian
Gbp-Dch: Ignore
|
| |
|
|
|
|
| |
Allowing a method to request work from other methods is a powerful
capability which could be misused or exploited, so to slightly limited
the surface let method opt-in into this capability on startup.
|
| |
|
|
|
|
|
|
|
|
| |
Embedding an entire acquire stack and HTTP logic in the mirror method
made it rather heavy weight and fragile. This reimplement goes the other
way by doing only the bare minimum in the method itself and instead
redirect the actual download of files to their proper methods.
The reimplementation drops the (in the real world) unused query-string
feature as it isn't really implementable in the new architecture.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a method needs a file to operate like e.g. mirror needs to get a list
of mirrors before it can redirect the the actual requests to them. That
could easily be solved by moving the logic into libapt directly, but by
allowing a method to request other methods to do something we can keep
this logic contained in the method and allow e.g. also methods which
perform binary patching or similar things.
Previously they would need to implement their own acquire system inside
the existing one which in all likelyhood will not support the same
features and methods nor operate with similar security compared to what
we have already running 'above' the requesting method. That said, to
avoid methods producing conflicts with "proper" files we are downloading
a new directory is introduced to keep the auxiliary files in.
[The message magic number 351 is a tribute to the german Grundgesetz
article 35 paragraph 1 which defines that all authorities of the
state(s) help each other on request.]
|
| |
|
|
|
|
|
|
| |
For deb files we always supported falling back from one server to the
other if one failed to download the deb, but that was hardwired in the
handling of this specific item. Moving this alongside the retry
infrastructure we can implement it for all items and allow methods to
use this as well by providing additional URIs in a redirect.
|
| |
|
|
|
|
|
| |
Moving the Retry-implementation from individual items to the worker
implementation not only gives every file retry capability instead of
just a selected few but also avoids needing to implement it in each item
(incorrectly).
|
| |
|
|
|
|
|
| |
We would like to issue a warning about seccomp support in
Configuration(), but since the queue is empty, there is no
current item to show the URL for and we get a segfault. Show
the protocol instead.
|
| |
|
|
|
|
|
| |
This avoids running the Proxy-Auto-Detect script inside the
untrusted (well, less trusted for now) sandbox. This will allow
us to restrict the http method from fork()ing or exec()ing via
seccomp.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This makes it easier to see which headers includes what.
The changes were done by running
git grep -l '#\s*include' \
| grep -E '.(cc|h)$' \
| xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/'
To modify all include lines by adding a space, and then running
./git-clang-format.sh.
|
| |
|
|
|
|
|
| |
Any respective parser will do the right thing and grab the last value,
but its better for style to generate that field only once.
Gbp-Dch: Ignore
|
| |\ |
|
| | |
| |
| |
| |
| | |
This is needed on BSD where root's default group is wheel, not
root.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The bugreport shows a segfault caused by the code not doing the correct
magical dance to remove an item from inside a queue in all cases. We
could try hard to fix this, but it is actually better and also easier to
perform these checks (which cause instant failure) earlier so that they
haven't entered queue(s) yet, which in return makes cleanup trivial.
The result is that we actually end up failing "too early" as if we
wouldn't be careful download errors would be logged before that process
was even started. Not a problem for the acquire system, but likely to
confuse users and programs alike if they see the download process
producing errors before apt was technically allowed to do an acquire
(it didn't, so no violation, but it looks like it to the untrained eye).
Closes: 835195
|
| |
|
|
|
|
|
|
|
|
| |
Now that we have the redirections loopchecker centrally in our items we
can use it also to prevent internal redirections to loop caused by
bugs as in a few instances we get into the business of rewriting the URI
we will query by ourself as we predict we would see such a redirect
anyway. Our code has no bugs of course, hence no practical difference. ;)
Gbp-Dch: Ignore
|
| |
|
|
|
|
|
|
|
|
| |
The failure handling frequently changes URI & Description of the failed
item to try a slightly different combination which might work, but the
logging of the failure happens only afterwards as the same failure
handling decides if this is a critical error or not so we need a backup
here instead of potentially new content.
A purely cosmetic issue, but can still be confusing for humans.
|
| |
|
|
|
|
|
|
|
|
| |
To prevent accidents like adding http-sources while using tor+http it
can make sense to allow disabling methods. It might even make sense to
allow "redirections" and adding "symlinked" methods via configuration.
This could e.g. allow using different options for certain sources by
adding and configuring a "virtual" new method which picks up the config
based on the name it was called with like e.g. http does if called as
tor+http.
|
| |
|
|
|
|
|
| |
Having the detection handled in specific (http) workers means that a
redirection loop over different hostnames isn't detected. Its also not a
good idea have this implement in each method independently even if it
would work
|
| |
|
|
|
|
|
|
|
|
|
|
| |
apt-transports not shipped in apt directly are usually named
apt-transport-% with % being what is in the name of the transport.
tor additional introduced aliases via %+something, which isn't a bad
idea, so be strip the +something part from the method name before
suggesting the installation of an apt-transport-% package.
This avoids us the maintainance of a list of existing transports
creating a two class system of known and unknown transports which would
be quite arbitrary and is unfriendly to backports.
|
| |
|
|
|
|
|
|
| |
Weak had no dedicated option before and Insecure and Downgrade were both
global options, which given the effect they all have on security is
rather bad. Setting them for individual repositories only isn't great
but at least slightly better and also more consistent with other
settings for repositories.
|
| |
|
|
|
|
|
|
| |
Downloading and saying "Hash Sum mismatch" isn't very friendly from a
user POV, so with this change we try to detect such cases early on and
report it, preferably before download even started.
Closes: 827758
|
| |
|
|
|
|
|
|
|
|
| |
Broken in a4b8112b19763cbd2c12b81d55bc7d43a591d610.
If an item has a description which includes no space and is redirected
to another mirror the code which wants to rewrite the description
expects a space in there, but can't find it and the unguarded substr
command on the string will fail with an exception thrown…
Guarding it properly and everything is fine.
|
| |
|
|
|
|
| |
They are the small brothers of the hashsum mismatch, so they deserve a
similar treatment even through we have for architectual reasons not a
much to display as for hashsum mismatches for now.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Users tend to report these errors with just this error message… not very
actionable and hard to figure out if this is a temporary or 'permanent'
mirror-sync issue or even the occasional apt bug.
Showing the involved hashsums and modification times should help in
triaging these kind of bugs – and eventually we will have less of them
via by-hash.
The subheaders aren't marked for translation for now as they are
technical glibberish and probably easier to deal with if not translated.
After all, our iconic "Hash Sum mismatch" is translated at least.
These additions were proposed in #817240 by Peter Palfrader.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With the previous commit we track the state of transactions, so we can
now use our knowledge to avoid processing data for a transaction which
was already closed (via an abort in this case).
This is needed as multiple independent processes are interacting in the
process, so there isn't a simple immediate full-engine stop and it would
also be bad to teach each and every item how to check if its manager
has failed subordinate and what to do in that case.
In the pdiff case, which deals (potentially) with many items during its
lifetime e.g. a hashsum mismatch in another file can abort the
transaction the file we try to patch via pdiff belongs to. This causes
some of the items (which are already done) to be aborted with it, but
items still in the process of acquisition continue in the processing and
will later try to use all the items together failing in strange ways as
cleanup already happened.
The chosen solution is to dry up the communication channels instead by
ignoring new requests for data acquisition, canceling requests which are
not assigned to a queue and not calling Done/Failed on items anymore.
This means that e.g. already started or pending (e.g. pipelined)
downloads aren't stopped and continue as normal for now, but they remain
in partial/ and aren't processed further so the next update command will
pick them up and put them to good use while the current process fails
updating (for this transaction group) in an orderly fashion.
Closes: 817240
Thanks: Barr Detwix & Vincent Lefevre for log files
|
| |
|
|
|
|
|
|
| |
This makes the new GPG related warnings much nicer to read,
for example, the second one here replaces the first one:
W: gpgv:/var/lib/apt/lists/example.com_dists_stable_InRelease: Weak ...
W: http://example.com/dists/stable/InRelease: Weak ...
|
| |
|
|
|
|
|
|
|
| |
This can be used by workers to send warnings to the main
program. The messages will be passed to _error->Warning()
by APT with the URI prepended.
We are not going to make that really public now, as the
interface might change a bit.
|
| |
|
|
|
| |
Reported-By: cppcheck
Git-Dch: Ignore
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In 0940230d we started dropping privileges for file (and a bit later for
copy, too) with the intend of uniforming this for all methods. The
commit message says that the source will likely fail based on the
compressors already – and there isn't much secret in the repository
content. After all, after apt has run the update everyone can access the
content via apt anyway…
There are sources through which worked before which are mostly
single-deb (and those with the uncompressed files available).
The first one being especially surprising for users maybe, so instead of
failing, we make it so that apt detects that it can't access a source as
_apt and if so doesn't drop (for all sources!) privileges – but we limit
this to file/copy, so the uncompress which might be needed will still
fail – but that failed before this regression.
We display a notice about this, mostly so that if it still fails (e.g.
compressed) the user has some idea what is wrong.
Closes: 805069
|
