| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
| |
If apt update is interrupted during transaction commit, the system may
end up with updated InRelease that does not match the other list files,
which can cause apt to believe that the lists are already up-to-date when
they are in fact not.
Closes: #1078608
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Ongoing Run can be cancelled by returning false from ::Pulse(). In this case
the items would still be called pkgAcquire::Item::Finished() as usual, causing
pkgAcqMetaClearSig to commit the incomplete transaction anyway as at that
point it's still Started and has no errors.
Add a pkgAcquire::Item::Cancelled() method and use it in pkgAcqMetaClearSig
to abort the transaction.
Closes: #1078608
|
| | |
|
| |
|
|
|
|
|
|
| |
Benchmark:
- https://stackoverflow.com/questions/62058906/why-my-performance-benchmark-gives-me-wrong-results
References:
- https://clang.llvm.org/extra/clang-tidy/checks/performance/prefer-single-char-overloads.html
|
| |
|
|
|
| |
We configure all variants the CPU supports as active, but repositories
usually do not provide all of them, so let's not be noisy.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Architecture variants are children of an architecture that share
the same ABI but are optimized for different ISA levels. They
are available in Ubuntu 25.10 and newer, and not supported in
Debian or other distributions.
A deb built for a variant contains the Architecture-Variant field,
and the Architecture field points to the baseline, for example:
Architecture: amd64
Architecture-Variant: amd64v3
However, the apt-get indextargets command reports the variant in the
Architecture: field, and most of the code in APT presents the variant
as the architecture.
There are two types of variants:
1. Standalone variants are recorded in the Architectures field of the
Release file as if they were a real architecture:
Architectures: amd64 amd64v3
Standalone architecture variants only fetch the standalone
architecture variant's Packages file. To do this, this patch
changes the code such that the variants indextargets "supplant"
the base targets.
This may have complicated outcomes on the apt-get indextargets
command.
2. Other variants can only be identified by their files being recorded
with hashes in the Release file.
APT fetches both the base architecture's as well as the variant's
Packages file.
Variants are configured in the
APT::Architecture-Variants
list.
Image builders may want to build specific variant images using
APT::Architecture-Variants { "amd64v3"; }
But this commit also implements an automatic discovery mechanism
using the varianttable and /proc/cpuinfo.
APT::Architecture-Variants "auto";
|
| |
|
|
| |
Reported-by: cacin on irc
|
| |
|
|
| |
string{=> _view})
|
| | |
|
| |
|
|
|
|
|
|
|
| |
We were rather inconsistent in using it and as our public headers
contain deduction guides (a c++17 feature) it seems silly to try to hide
a c++11 feature in a macro, so lets stop this charade and drop the
macro and while we are changing all these lines lets apply [[nodiscard]]
(another c++17 feature) and other suggestions from clang-tidy and
formatting for a little more consistency.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new sqv method uses sequoia's sqv tool to verify files. The
tool's interface is quite simple: It returns 0 on success, and
prints one line per good signer with the fingerprint.
sqv has a configurable crypto policy. We have defined apt-specific
override mechanisms for sequoia's standard policy, allowing both
users, distributions, and apt package to provide overrides for
Sequoia's default policy in meaningful ways.
The sqv method will be built and be the standard method for
verifying OpenPGP signatures provided that `sqv` is in the
PATH during building. It is not built if there was no sqv
at build time, so you need a rebuild to enable sqv later
on.
On the flip side, the gpgv method is always built, but it
does need to be always built: If APT::Key::GPGVCommand is
set, we need to fallback to it - this is important to
support existing users of that interface such as
mmdebstrap. Also we want to fall back to it when /usr/bin/sqv
disappears - for example in our CI :D
A couple of concessions have been made for test suite purposes:
- Failure to split a clearsigned file only shows the summary,
as the gpgv method also only showed it, and no details why
it failed.
- We write "Got GOODSIG <fingerprint>" in debug mode to mimic
the gpgv code to keep the test suite happy.
- In various places in the test suite we assert minimal output
from sqv, but sqv's human output is not intended to be stable.
This will incur additional work when it breaks. However we do
not _parse_ the output, so actual operation of apt is unaffected.
A couple of things are suboptimal here:
- We are still doing clearsigned splitting ourselves. sqv only
has support for detached signatures right now, whereas sqopv
has support for clearsigned files as well (but sqopv does not
provide any reasons for why signature verification fails or
means to set a policy).
- Deprecation of algorithms happens on a timebomb basis in
sequoia. We have no means to give users warnings ahead of
time if their configuration is outdated.
We have implemented various bits that probably should be going away:
- Fallback to trusted.gpg
This is just annoying...
- Support for fingerprints in Signed-By (no subkey matching though)
The extra work here is arguably less compared to gpgv.
- Check the keyring for correctness.
With Signed-By everywhere, we should just error out rather than
skip broken keyrings.
Moo
|
| |\
| |
| |
| |
| | |
Support uncompressed indexes from partial file:/ mirrors
See merge request apt-team/apt!235
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Ideally copy and file mirrors would support by-hash as well, but its
harder to setup and maintain especially if you want to cache an online
mirror who has by-hash enabled.
We can avoid unneeded roundtrips (in the best case) and entire cache
misses (in the usual worst case) by "just" telling methods that the URI
we passed it has the requested file perhaps also in other paths.
This is done in pseudo-relative paths as we would otherwise need to
teach redirection code to rewrite those URIs as well. A method like http
can easily ignore this value and await explicit instructions to look at
that file, but inspecting the path in local sources via file or copy is
(comparatively) free, so we just do it immediately. If that ends up
being the wrong version of the file as by-hash would have protected us
from we are in this feature branch now falling back to other mirrors
which are like the ones online and in support of by-hash.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Perhaps a common error in the past nowadays thanks to by-hash we rarely
get mismatches due to our calling behaviour and more because the mirror
is genuinely bad especially if this mirror didn't work with by-hash.
Using our "normal" error handling allows falling back to other mirrors
which hopefully do support by-hash and/or are not broken on a filesystem
level making us more resistent to the occasional bad apple in a wild
mirror forest.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
As the hostname for them is likely empty (for absolute or . or .. for
relative paths) considering one a bad site causes all of the sites from
this scheme to be considered bad.
In a perfect world, we would figure out a good site-path to use for
these sources, but as they are local sources which means we don't have
to perform costly requests to the internet we just let them happen
instead for now.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
While we pick most files from the file:/ mirror directly, we explicitly
bring in the InRelease file with a copy one if-clause later, which
causes this code to be run again and ends up considering the copy:/ the
mirror the file was picked up from, which works, but isn't what we had
signed up for.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The main thread calculates the hash sums if the method hasn't to be able
to verify the files, but while a failure is detected and reported, the
error messages did not include the hash sums which looks at least
puzzling.
So to have a complete error message we ensure that the hashes we
calculated are appended to the message from the method so that it looks
for the higher level code as if the method had calculated the hashes.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If a file:/ mirror does not contain the requested compressed file, but
the uncompressed variant of that file the method will sent us a fitting
Done message with data only about the Alt-Filename, but the central hash
verification code in the acquire system does not expect that looking for
the non-existent compressed file to compare its hashes with the expected
result – which fails (if we are verifying at least).
That file:/ as well as the item code deals with this suggests that this
feature was broken by 448c38bdcd72b52f11ec5f326f822cf57653f81c, but I
couldn't (easily) compile that to verify, so this is only a guess.
Supporting this allows using index files from /var/lib/apt/lists –
which are likely uncompressed or can at least made so easily – to
construct a file:/ mirror.
|
| |/
|
|
|
|
|
| |
gcc reports `warning: Declaring type ‘class X’ final would
enable devirtualization of Y calls [-Wsuggest-final-types]`.
Not that this would massively improve performance, but it
shouldn't hurt telling the compiler what is obvious for a human.
|
| | |
|
| | |
|
| |
|
|
| |
now, UB in C++20)
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
We never used the debug level before, so we can do that. This
allows us to have the new audit level.
We did call DumpErrors() with DEBUG in two debug code paths,
so don't touch those.
debug
|
| |
|
|
|
|
| |
This was automated with sed and git-clang-format, and then I had to
fix up the top of policy.cc by hand as git-clang-format accidentally
indented it by two spaces.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Nowadays we only download the index file if we have a non-current file
on disk which we want to patch. If that is the case, any index file for
patches we could have stored is by definition outdated, so storing those
files just takes up disk space.
At least, that is the case if we have a Release file – if we don't this
commit introduces a needless redownload for such repositories but such
repositories are an error by default and if they can't be bothered to
provide a Release file its very unlikely they actually ship diffs, so
adding detection code for this seems pointless at best.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A source marked with trusted=yes can still fail verification of the
Release file, mostly for Date related issues, like being too new or too
old, which have other options to force them in.
The update code was not using the Release file (which was a InRelease
file but failed verification – which was overridden by trusted=yes) as
intended, but it marked it for storage, so that this "bad" Release file
would end up being moved into lists/, which is bad as the indexes it
refers to aren't updated while the next update run assumes that the
indexes are in the state the Release file claims them to be in.
Fixed simply by making the storage conditional on the usage as intended,
which also resolves a second issue: The verification can also detect that
a Release file we got is older than what we already have to avoid down-
grade attacks. The more likely explanation is a slightly outdated mirror
in a rotation/CDN through, so this gets the silent treatment to avoid
scaring users by handling it as if we had got the same Release file we
already have stored locally, removing the freshly received older file
in the process alongside setting some variables. Those variables were
already modified in the trusted=yes case though resulting in the stored
Release file being removed instead. Not modifying the variables too early
resolves this problem as well.
Both seem to exist since at least 2015 as traces are visible in 448c38bdcd
already, which shuffled lots of code around including the bad ones, but
as we are in trusted=yes land, security is of no concern here, this
"just" leads to failed pinning, hashsum mismatches and other strange
problems in follow-up calls depending on how out of sync the Release
file (if its still present) is with the rest of the trusted data.
Reported-By: Dima Kogan <dkogan@debian.org> on IRC
Tested-By: Dima Kogan <dkogan@debian.org>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We only check the start of these lines to avoid hard coding the exact
command and we pick 150 as maximum line length as the longest package
name on my system is apparently 75 characters long. We could choose
longer or shorter without much issue as over-length just means we
mishandle the rest of the line as a new line and it should be really
unlikely that a) lines are that long in this file and b) that such long
lines contain one of our trigger sequences – but even if, all we do is
start a download of an online file. Could be worse.
This auto-detection can be avoided by setting
Acquire::Changelogs::AlwaysOnline (or Origin specific sub options)
to "true" if you always want the changelog from an online source.
The reverse – setting it to "false" in the hope it would not get the
changelog from an online source – was not and is still not possible.
Closes: #1024457
|
| | |
|
| |
|
|
|
|
| |
FindS has a APT::StringView based API nowadays, so we can avoid these
explicit calls also allowing us to avoid the std::string in input or
output entirely or at least move it a few branches down.
|
| | |
|
| |
|
|
|
|
| |
This currently has no effect, as there are no quotable characters
inside it, but it will allow us to send embedded keys through to
the method.
|
| |
|
|
|
| |
This yields more accurate delays and avoids issues with clock
skew.
|
| |
|
|
|
|
|
| |
Add a new Item field called FetchAfter, which determines the earliest
time the item should be fetched at. Adjust insertion into queue to
take it into account alongside priority, and only fill pipelines
with items that are ready.
|
| |
|
|
|
|
|
|
| |
Enqueuing by calling Init(), and then resetting the status to
idle means that the item can get enqueued twice if we call
Cycle() from inside pkgAcquire::Run().
Reset the status to StatIdle before calling Init()
|
| |
|
|
|
|
|
|
| |
The error says the repository is broken but doesn't mention which one it
is. The item description gives us at least all the information, but is
not as nicely formatted. As this message is not even marked for
translation this is a rather temporary affair and we can survive without
the eye candy for a while.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Keeping URIs encoded in the acquire system depends on having them
encoded in the first place. While many other places got the encoding
2 out of 3 ArchiveURI implementations were missed which are in practice
responsible for nearly all of the URI building, just that index filename
do not contain characters to escape and the Filename fields in Packages
files usually aren't. Usually. Except if you happen to have e.g. an epoch
featuring package with the colon encoded in the filename. On the upside,
in most repositories the epoch isn't part of the filename.
Reported-By: Johannes 'josch' Schauer on IRC
References: e6c55283d235aa9404395d30f2db891f36995c49
|
| |
|
|
|
|
|
|
|
| |
Enable the Acquire::Retries option by default, set to 3.
This will help with slightly unreliable networking; future
work is needed for adding backoff and SRV/IP rotation.
LP: #1876035
Gbp-Dch: full
|
| |
|
|
|
|
|
|
|
| |
Repositories without Size information for packages are not
proper and need fixing. This ensures people see an error in
CI, and get notifications and hence the ability to fix it.
It can be turned off by setting Acquire::AllowUnsizedPackages
to true.
|
| |
|
|
|
|
|
|
|
|
|
| |
The mirror method can distribute requests for files based on various
metadata bits, but some – the main index files – weren't actually
passing those on to the methods as advertised in the manpage.
This is hidden both by mirror usually falling back to other sources
which will eventually hit the right one and that if the repository does
not support by-hash apt will automatically stick to the mirror which was
used for the Release file.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Especially in small sections of an archive it can happen that an index
returns to a previous state (e.g. if a package was first added and then
removed with no other changes happening in between). The result is that
we have multiple patches which start from the same hash which if we
perform clientside merging is no problem although not ideal as we
perform needless work.
For serverside merging it would not matter, but due to rred previously
refusing to merge zero-size patches but dak ignoring failure letting it
carry these size-zero patches until they naturally expire we run into a
problem as these broken patches won't do and force us to fall back to
downloading the entire index. By always starting from the last patch
instead of the first with the starter hash we can avoid this problem
and behave optimally in clientside merge cases, too.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The rred method expects the patches to have a certain name, which we
have to rename the file to before calling the method, but by delaying
the rename we ensure that if the download of one of them fails and a
successful fallback occurs they are all properly cleaned up as no longer
useful while in the error case the next apt run can potentially pick
them up as already downloaded.
Our test-pdiff-usage test was encountering this every other run, but did
not fail as the check for unaccounted files in partial/ was wrapped
in a subshell so that the failure produced failing output, but did not
change the exit code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
APT tries to detect if applying patches is more costly than just
downloading the complete index by combining the size of the patches.
That is correct for client-side merging, but for server-side merging
we actually don't know if we will jump directly from old to current or
have still intermediate steps in between.
With this commit we assume it will be a jump from old to current through
as that is what dak implements and it seems reasonable if you go to
the trouble of server side merging that the server does the entire
merging in one file instead of leaving additional work for the client
to do.
Note that this just changes the heuristic to prevent apt from discarding
patches as uneconomic in the now more common one merged-patch style, it
still supports multiple merged patches as before.
To resolve this cleanly we would need another field in the index file
declaring which hash we will arrive at if a patch is applied (or a field
differentiating between these merged-patch styles at least), but that
seems like overkill for now.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This commit potentially breaks code feeding apt an encoded URI using a
method which does not get URIs send encoded. The webserverconfig
requests in our tests are an example for this – but they only worked
before if the server was expecting a double encoding as that was what
was happening to an encoded URI: so unlikely to work as expected in
practice.
Now with the new methods we can drop this double encoding and rely on
the URI being passed properly (and without modification) between the
layers so that passing in encoded URIs should now work correctly.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We do not deal a lot with URIs which need encoding, but then we do it is
a pain that we store it decoded in the acquire system as it means we
have to decode and reencode URIs eventually which is potentially giving
us slightly different URIs.
We see that in our own testing framework while setting up redirects as
the config options are effectively double-encoded and decoded to pass
them around successfully as otherwise %2f and / in an URI are treated
the same.
This commit adds the infrastructure for methods to opt into getting URIs
send in encoded form (and returning them to us in encoded form, too) so
that we eventually do not have to touch the URIs which is how it should
be. This means though that we have to deal with methods who do not
support this yet (aka: all at the moment) for which we decode and encode
while communicating with them.
|
| |
|
|
| |
Closes: #931566
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
This experiment did not turn out sensibly, as some servers do not
accept credentials when none are expected and fail, so you cannot
mirror such a repository.
This reverts commit c2b9b0489538fed4770515bd8853a960b13a2618.
|
