| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The new sqv method uses sequoia's sqv tool to verify files. The
tool's interface is quite simple: It returns 0 on success, and
prints one line per good signer with the fingerprint.
sqv has a configurable crypto policy. We have defined apt-specific
override mechanisms for sequoia's standard policy, allowing both
users, distributions, and apt package to provide overrides for
Sequoia's default policy in meaningful ways.
The sqv method will be built and be the standard method for
verifying OpenPGP signatures provided that `sqv` is in the
PATH during building. It is not built if there was no sqv
at build time, so you need a rebuild to enable sqv later
on.
On the flip side, the gpgv method is always built, but it
does need to be always built: If APT::Key::GPGVCommand is
set, we need to fallback to it - this is important to
support existing users of that interface such as
mmdebstrap. Also we want to fall back to it when /usr/bin/sqv
disappears - for example in our CI :D
A couple of concessions have been made for test suite purposes:
- Failure to split a clearsigned file only shows the summary,
as the gpgv method also only showed it, and no details why
it failed.
- We write "Got GOODSIG <fingerprint>" in debug mode to mimic
the gpgv code to keep the test suite happy.
- In various places in the test suite we assert minimal output
from sqv, but sqv's human output is not intended to be stable.
This will incur additional work when it breaks. However we do
not _parse_ the output, so actual operation of apt is unaffected.
A couple of things are suboptimal here:
- We are still doing clearsigned splitting ourselves. sqv only
has support for detached signatures right now, whereas sqopv
has support for clearsigned files as well (but sqopv does not
provide any reasons for why signature verification fails or
means to set a policy).
- Deprecation of algorithms happens on a timebomb basis in
sequoia. We have no means to give users warnings ahead of
time if their configuration is outdated.
We have implemented various bits that probably should be going away:
- Fallback to trusted.gpg
This is just annoying...
- Support for fingerprints in Signed-By (no subkey matching though)
The extra work here is arguably less compared to gpgv.
- Check the keyring for correctness.
With Signed-By everywhere, we should just error out rather than
skip broken keyrings.
Moo
|
| |
|
|
|
| |
OpenSSL is mandatory now, it is no longer possible to build
without https support either.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce an OpenSSL::Crypto backend for the hashes library
and an OpenSSL::SSL backend for the TLS support in our https
method.
Many thanks to curl for showing the way with how to handle
a CRL file. There are some memory leaks here with the
TlsFd itself as well as the proxy support; and we should
reorganize the code to generate the ssl object as late
as possible.
A peculiar aspect of OpenSSL is that SSL_has_pending() returns
1 even if SSL_read() will fail to read anything and return the
equivalent of EAGAIN. We work around this here by also peeking
ahead 1 byte. I was running a very high RTT connection from
Germany to Australia for testing, and with the peeking it's
using negligible amounts of CPU; before that, it was busy
looping at 100%. Bad OpenSSL!
|
| |
|
|
|
|
|
| |
The GnuTLS backend avoids the need to link in libgcrypt, and
gets us another certified library people will be happy with.
This should be an intermediate step on the road to OpenSSL.
|
| |
|
|
| |
Everything is abstracted away now.
|
| | |
|
| |
|
|
|
| |
This is a pretty standalone function, extract it for later
reuse.
|
| |
|
|
|
|
|
|
| |
We tried to access str.size() - 1 on a 0-sized string, which
does not work, and now that we build with assertions triggers
it.
Closes: #1090068
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
gpgv is quite picky, it takes at most 40 keyrings but it also doesn't
even take keyrings with : in the name, generating an ERROR message
instead.
So instead let's always merge the configured keyrings for the source
into a single one.
While we are at it, since we are merging now, let's also add some
rudimentary error checking to what we write into the merged keyring:
If copying into it fails, we should abort.
Reported-by: SuShY on IRC
|
| |
|
|
|
|
|
|
| |
Reimplement strict pinning by rejecting the non-candidates when
translating the problem from the depcache to the solver. This is
substantially better than restricting the list of alternatives for
an or group to only include allowed ones for debugging purposes,
albeit a bit slower.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We only compared candidate to installed version, but candidate
should dominate all versions, otherwise we end up in the fancy
problem of elpa-notmuch in
upgrade-noble-t64-remove-desktop-2024-03-29.edsp
Where we had three versions:
not-installable 0.38.3-1ubuntu1
candidate 0.38.2-1.1ubuntu2
installed 0.38.2-1ubuntu2
And received an ordering:
installed > non-installable > candidate
despite
candidate > installed
This is only visible with no-strict-pinning right now, as we are
otherwise filtering out invalid choices (and hence we only have
candidate and installed otherwise).
|
| |
|
|
|
|
|
|
|
|
|
|
| |
We only ever documented .gpg and .asc and have ensured that
only such files exist in the keyring, but people can set the
Signed-By option to keys with any random extension, as that
was not validated before by apt-key.
This doesn't change the behavior for trusted.gpg.d: We filter
the directory listing and only look at .gpg and .asc files.
Closes: #1088656
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This gains us support for not using an absolute path, which
is nice, and we can now cache the invocation across fork() in
the method.
Still keep preferring absolute paths, this avoids execvp() having
to iterate the path and reduces bug potential with custom gpgv in
/usr/local.
Fix the last entry in the list of gpgvs to say gpgv1
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
Lifted from the dpkg mtree code that mjg59 submitted in
https://lists.debian.org/debian-dpkg/2018/05/msg00005.html
Which as dpkg is GPL-2.0+ also is GPL-2.0+
|
| |
|
|
|
| |
This is a bit more overhead here, but if we need to build temporary
strings this is a lot cleaner.
|
| | |
|
| |
|
|
| |
Converting it to a comma-separated string and back loses precision.
|
| |
|
|
| |
This allows printing warnings and errors,
|
| | |
|
| |
|
|
|
| |
The version that accepted APT::StringView was marked
public.
|
| |\
| |
| |
| |
| | |
Support uncompressed indexes from partial file:/ mirrors
See merge request apt-team/apt!235
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Ideally copy and file mirrors would support by-hash as well, but its
harder to setup and maintain especially if you want to cache an online
mirror who has by-hash enabled.
We can avoid unneeded roundtrips (in the best case) and entire cache
misses (in the usual worst case) by "just" telling methods that the URI
we passed it has the requested file perhaps also in other paths.
This is done in pseudo-relative paths as we would otherwise need to
teach redirection code to rewrite those URIs as well. A method like http
can easily ignore this value and await explicit instructions to look at
that file, but inspecting the path in local sources via file or copy is
(comparatively) free, so we just do it immediately. If that ends up
being the wrong version of the file as by-hash would have protected us
from we are in this feature branch now falling back to other mirrors
which are like the ones online and in support of by-hash.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Perhaps a common error in the past nowadays thanks to by-hash we rarely
get mismatches due to our calling behaviour and more because the mirror
is genuinely bad especially if this mirror didn't work with by-hash.
Using our "normal" error handling allows falling back to other mirrors
which hopefully do support by-hash and/or are not broken on a filesystem
level making us more resistent to the occasional bad apple in a wild
mirror forest.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
As the hostname for them is likely empty (for absolute or . or .. for
relative paths) considering one a bad site causes all of the sites from
this scheme to be considered bad.
In a perfect world, we would figure out a good site-path to use for
these sources, but as they are local sources which means we don't have
to perform costly requests to the internet we just let them happen
instead for now.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
If an item failed previously, but that failure is ignored and a retry is
triggered we might end up in a situation in which it is failing yet
again for perhaps totally different reasons, but the error message
displayed is the one we have shown (or, because it was ignored likely
hidden) before which can be rather confusing in debugging.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
While we pick most files from the file:/ mirror directly, we explicitly
bring in the InRelease file with a copy one if-clause later, which
causes this code to be run again and ends up considering the copy:/ the
mirror the file was picked up from, which works, but isn't what we had
signed up for.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
It is a bit unfair for third-party methods wrapping e.g. http to not
have access to such tools, same for our old ftp and e.g. our tor even
if in practice this isn't used much even for http.
At least that is one less FIXME in the code.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The previous version allowed only foo::*::bar as a pattern, now you can
have as many starred parts as you want allowing for a finer control
instead of hardcoding specific common star-values or giving up with **.
It also adds a new INVALID type for an option to mark options which
should not be used, but are covered by a pattern which would normally
allow it.
Our tests are more or less the only users of this hidden feature making
this change not very risky even through it has no dedicated test itself.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The main thread calculates the hash sums if the method hasn't to be able
to verify the files, but while a failure is detected and reported, the
error messages did not include the hash sums which looks at least
puzzling.
So to have a complete error message we ensure that the hashes we
calculated are appended to the message from the method so that it looks
for the higher level code as if the method had calculated the hashes.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
If a file:/ mirror does not contain the requested compressed file, but
the uncompressed variant of that file the method will sent us a fitting
Done message with data only about the Alt-Filename, but the central hash
verification code in the acquire system does not expect that looking for
the non-existent compressed file to compare its hashes with the expected
result – which fails (if we are verifying at least).
That file:/ as well as the item code deals with this suggests that this
feature was broken by 448c38bdcd72b52f11ec5f326f822cf57653f81c, but I
couldn't (easily) compile that to verify, so this is only a guess.
Supporting this allows using index files from /var/lib/apt/lists –
which are likely uncompressed or can at least made so easily – to
construct a file:/ mirror.
|
| | |
| |
| |
| |
| |
| |
| | |
gcc reports `warning: Declaring type ‘class X’ final would
enable devirtualization of Y calls [-Wsuggest-final-types]`.
Not that this would massively improve performance, but it
shouldn't hurt telling the compiler what is obvious for a human.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
| |
gcc produces a length warning starting with:
| warning: ignoring return value of ‘_FIter std::remove_if(_FIter, _FIter, _Predicate) [with …]’, declared with attribute ‘nodiscard’ [-Wunused-result]
which is usually correct, but in the usage here we don't want to call an
erase as we operate on a c-style new[] array here which contains pointers
into argv and has an explicit sentinel (nullptr) – FileSize() is based
on this sentinel.
Casting to void silences the warning as is standard practice and
hopefully indicates better that this is intended than ignoring the
warning for the casual on-looker.
|
| |\
| |
| |
| |
| | |
Add a --comment option to record Comment: in history
See merge request apt-team/apt!401
|
| | |
| |
| |
| | |
Suggested-by: mhoye on IRC
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pattern tree parser was written using string_views in a no-copy
fashion. However, at various places we assume that we can index one
byte after the end of the string_view and get a NUL-byte there.
This works fine in practice because we only ever pass it zero-terminated
string_views, but it fails if you compile with -Wp,-D_GLIBCXX_ASSERTIONS
because it will do bounds checks.
So instead, do our own bounds checks here and always return '\0'
if requesting one byte too many.
Reported-by: Christian Heusel <christian@heusel.eu>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let's get some feedback here. This is capped to 30s per retry, so
a maximum of 90s per item that we delay the failure case. In the
worst case, we hence have 90n seconds delay for n items.
In a more realistic worse case however, the server likely does
not collapse per-item, and we end up trying all items in a short
burst so it seems more realistic we hit a 90s total delay.
The bursty behavior may be suboptimal; we don't quite know, but
it seems the specification of Retry-After is per-entity rather than
per-host, so technically speaking it seems correct.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some mirrors like snapshot.debian.org apply strict http rate limits.
While apt already has an exponential backoff mechanism implemented, this
is not sufficient due to the following reasons:
1. all retries happen roughly at the same time
2. the retry-after information from the server is not used
We fix this by improving the algorithm: First, if present, the timestamp
or duration of the Retry-After header is added to the exponential
backoff, optimizing for success-on-second-try. Second, a random delay is
added (delay += [0, delay]) to distribute the retries to not immediately
run into the limit again. To avoid super-long delays, this is capped by
the configured maximum delay. This logic only becomes active if the
option Acquire::Retries::HandleRetryAfter is enabled.
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
|
| |
|
|
|
|
|
|
|
| |
Specifically the lack of the `string_view` include is causing
compilation failures for rust-rust-apt.
Running iwyu also adds `algorithm` for our use of `std::min`,
and `iosfwd` for our use of `std::ostream` none of which seem
to be causing build failures so far.
|
| |\
| |
| |
| |
| | |
DEFER
See merge request apt-team/apt!392
|
| | | |
|
| | | |
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | | |
std::string_view const & -> std::string_view
See merge request apt-team/apt!397
|
| | |/
| |
| |
| | |
Fixes: 2c03eed649068c8838d971ceeb1a09603dcc8e9c
|
