summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'pu/uriencode' into 'master'Julian Andres Klode2020-12-1832-121/+276
|\ | | | | | | | | Use encoded URIs in the acquire system See merge request apt-team/apt!139
| * Don't re-encode encoded URIs in pkgAcqFileDavid Kalnischkies2020-12-182-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | This commit potentially breaks code feeding apt an encoded URI using a method which does not get URIs send encoded. The webserverconfig requests in our tests are an example for this – but they only worked before if the server was expecting a double encoding as that was what was happening to an encoded URI: so unlikely to work as expected in practice. Now with the new methods we can drop this double encoding and rely on the URI being passed properly (and without modification) between the layers so that passing in encoded URIs should now work correctly.
| * Implement encoded URI handling in all methodsDavid Kalnischkies2020-12-1813-37/+77
| | | | | | | | | | | | | | | | Every method opts in to getting the encoded URI passed along while keeping compat in case we are operated by an older acquire system. Effectively this is just a change for the http-based methods as the others just decode the URI as they work with files directly.
| * Keep URIs encoded in the acquire systemDavid Kalnischkies2020-12-1816-75/+174
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We do not deal a lot with URIs which need encoding, but then we do it is a pain that we store it decoded in the acquire system as it means we have to decode and reencode URIs eventually which is potentially giving us slightly different URIs. We see that in our own testing framework while setting up redirects as the config options are effectively double-encoded and decoded to pass them around successfully as otherwise %2f and / in an URI are treated the same. This commit adds the infrastructure for methods to opt into getting URIs send in encoded form (and returning them to us in encoded form, too) so that we eventually do not have to touch the URIs which is how it should be. This means though that we have to deal with methods who do not support this yet (aka: all at the moment) for which we decode and encode while communicating with them.
| * Proper URI encoding for config requests to our test webserverDavid Kalnischkies2020-12-184-14/+29
|/ | | | | | Our http method encodes the URI again which results in the double encoding we have unwrap in the webserver (we did already, but we skip the filename handling now which does the first decode).
* Do not require libxxhash-dev for including pkgcachegen.hJulian Andres Klode2020-12-171-1/+3
|
* Unroll pkgCache::sHash 8 time, break up dependencyJulian Andres Klode2020-12-151-2/+16
| | | | | | | | | | | | | | | | Unroll pkgCache::sHash 8 times and break up the dependency between the iterations by expanding the calculation H(n) = 33 * H(n-1) + c 8 times rather than performing it 8 times. This seems to yield about a 0.4% performance improvement. I tried unrolling 4 and 2 bytes as well, those only having 3 ifs at the end rather than 1 small loop; but that was actually slower - potentially the code got to large and the cache went bonkers. I also tried unrolling 4 times instead of 8, thinking that smaller code might yield better results overall then, but that was slower as well.
* Release 2.1.142.1.14Julian Andres Klode2020-12-155-6/+13
|
* Use XXH3 for cache, hash table hashingJulian Andres Klode2020-12-156-64/+47
| | | | | | XXH3 is faster than both our CRC32c implementation as well as DJB hash for hash table hashing, so meh, let's switch to it.
* test: fixup for hash table size increase (changed output order)Julian Andres Klode2020-12-155-13/+12
|
* Release 2.1.132.1.13Julian Andres Klode2020-12-105-8/+32
|
* Raise APT::Cache-HashtableSize to 196613Julian Andres Klode2020-12-101-1/+1
| | | | | | | We now have over 100k package names, my Ubuntu system has 125k arleady, so increase the hash table size to match, this will cost us about a MB in cache size, but give a very nice speed up somewhere around 3%-4% or so.
* Merge branch 'pu/cve-2020-27350'Julian Andres Klode2020-12-096-2/+400
|\
| * CVE-2020-27350: tarfile: integer overflow: Limit tar items to 128 GiBJulian Andres Klode2020-12-093-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The integer overflow was detected by DonKult who added a check like this: (std::numeric_limits<decltype(Itm.Size)>::max() - (2 * sizeof(Block))) Which deals with the code as is, but also still is a fairly big limit, and could become fragile if we change the code. Let's limit our file sizes to 128 GiB, which should be sufficient for everyone. Original comment by DonKult: The code assumes that it can add sizeof(Block)-1 to the size of the item later on, but if we are close to a 64bit overflow this is not possible. Fixing this seems too complex compared to just ensuring there is enough room left given that we will have a lot more problems the moment we will be acting on files that large as if the item is that large, the (valid) tar including it probably doesn't fit in 64bit either.
| * CVE-2020-27350: debfile: integer overflow: Limit control size to 64 MiBJulian Andres Klode2020-12-093-0/+22
| | | | | | | | | | | | | | | | | | Like the code in arfile.cc, MemControlExtract also has buffer overflows, in code allocating memory for parsing control files. Specify an upper limit of 64 MiB for control files to both protect against the Size overflowing (we allocate Size + 2 bytes), and protect a bit against control files consisting only of zeroes.
| * tarfile: OOM hardening: Limit size of long names/links to 1 MiBJulian Andres Klode2020-12-093-2/+99
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Tarballs have long names and long link targets structured by a special tar header with a GNU extension followed by the actual content (padded to 512 bytes). Essentially, think of a name as a special kind of file. The limit of a file size in a header is 12 bytes, aka 10**12 or 1 TB. While this works OK-ish for file content that we stream to extractors, we need to copy file names into memory, and this opens us up to an OOM DoS attack. Limit the file name size to 1 MiB, as libarchive does, to make things safer.
| * CVE-2020-27350: arfile: Integer overflow in parsingJulian Andres Klode2020-12-094-1/+263
|/ | | | | | | | | | | | | | | | | | | | | | GHSL-2020-169: This first hunk adds a check that we have more files left to read in the file than the size of the member, ensuring that (a) the number is not negative, which caused the crash here and (b) ensures that we similarly avoid other issues with trying to read too much data. GHSL-2020-168: Long file names are encoded by a special marker in the filename and then the real filename is part of what is normally the data. We did not check that the length of the file name is within the length of the member, which means that we got a overflow later when subtracting the length from the member size to get the remaining member size. The file createdeb-lp1899193.cc was provided by GitHub Security Lab and reformatted using apt coding style for inclusion in the test case, both of these issues have an automated test case in test/integration/test-ubuntu-bug-1899193-security-issues. LP: #1899193
* patterns: Terminate short pattern by ~ and !Julian Andres Klode2020-12-072-1/+6
| | | | | | | | | | This allows patterns like ~nalpha~nbeta and ~nalpha!~nbeta to work like they do in APT. Also add a comment to remind readers that everything in START should be in short too. Cc: stable >= 2.0
* HexDigest: Silence -Wstringop-overflowJulian Andres Klode2020-12-041-0/+1
| | | | | | | | | | | | | | | | | | | | | The compiler does not know that the size is small and thinks we might be doing a stack buffer overflow of the vla: Add APT_ASSUME macro and silence -Wstringop-overflow in HexDigest() The compiler does not know that the size of a hash is at most 512 bit, so tell it that it is. ../apt-pkg/contrib/hashes.cc: In function ‘std::string HexDigest(gcry_md_hd_t, int)’: ../apt-pkg/contrib/hashes.cc:415:21: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=] 415 | Result[(Size)*2] = 0; | ~~~~~~~~~~~~~~~~~^~~ ../apt-pkg/contrib/hashes.cc:414:9: note: at offset [-9223372036854775808, 9223372036854775807] to an object with size at most 4294967295 declared here 414 | char Result[((Size)*2) + 1]; | ^~~~~~ Fix this by adding a simple assertion. This generates an extra two instructions in the normal code path, so it's not exactly super costly.
* test-method-rred: Use apthelper instead of apt-helperJulian Andres Klode2020-12-021-1/+1
| | | | | | Fixes lookup in as-installed testing Gbp-Dch: ignore
* gitignore: Add .*.swp filesJulian Andres Klode2020-12-021-0/+1
|
* gitignore: Add /build and /obj-* build dirsJulian Andres Klode2020-12-021-0/+3
| | | | This is more accurate
* Merge branch 'multiarch-fixes' into 'master'Julian Andres Klode2020-11-252-0/+9
|\ | | | | | | | | Apply hints suggested by the multi-arch hinter See merge request apt-team/apt!137
| * Apply multi-arch hints.Debian Janitor2020-11-232-0/+9
| | | | | | | | | | | | + apt-doc, libapt-pkg-doc: Add Multi-Arch: foreign. Changes-By: apply-multiarch-hints
* | Merge branch 'patch-1' into 'master'Julian Andres Klode2020-11-251-1/+1
|\ \ | | | | | | | | | | | | Fix typo in Catalan translation. See merge request apt-team/apt!132
| * | Fix typo in Catalan translation.Jordi Mallach2020-08-311-1/+1
| | |
* | | Merge branch 'feature/rred' into 'master'Julian Andres Klode2020-11-258-47/+192
|\ \ \ | |_|/ |/| | | | | | | | Enhance rred for possible external usage See merge request apt-team/apt!136
| * | Support compressed output from rred similar to apt-helper cat-filefeature/rredDavid Kalnischkies2020-11-074-4/+49
| | |
| * | Support reading compressed patches in rred direct call modesDavid Kalnischkies2020-11-072-1/+4
| | | | | | | | | | | | | | | | | | The acquire system mode does this for a long time already and as it is easy to implement and handy for manual testing as well we can support it in the other modes, too.
| * | Prepare rred binary for external usageDavid Kalnischkies2020-11-078-47/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Merging patches is a bit of non-trivial code we have for client-side work, but as we support also server-side merging we can export this functionality so that server software can reuse it. Note that this just cleans up and makes rred behave a bit more like all our other binaries by supporting setting configuration at runtime and supporting --help and --version. If you can make due without this, the now advertised functionality is provided already in earlier versions.
* | | Release 2.1.122.1.12Julian Andres Klode2020-11-235-6/+26
| | |
* | | Do not immediately configure m-a: same packages in lockstepJulian Andres Klode2020-11-062-3/+4
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In LP#835625, it was reported that apt did not unpack multi-arch packages in the correct order, and dpkg did not like that. The fix also made apt configure packages together, which is not strictly necessary. This turned out to cause issues now, because of dependencies on libc6:i386 that caused immediate configuration of that to not work. Work around the issue by not configuring multi-arch: same packages in lockstep if they have the immediate flag set. This will be the pseudo-essential set, and given how essential works, we mostly need the native arch to work correctly anyway. LP: #1871268 Regression-Of: 30426f4822516bdd26528aa2e6d8d69c1291c8d3
* | Refresh lintian-overrides of apt and libapt-pkg-docDavid Kalnischkies2020-11-052-4/+8
| |
* | Update libapt-pkg6.0 symbols fileDavid Kalnischkies2020-11-051-140/+6
| | | | | | | | | | That mostly means deleting symbols which went private or have disappeared and were previously compiler artefacts.
* | Remove ancient versions support from apts postinstDavid Kalnischkies2020-11-051-78/+2
| | | | | | | | | | | | | | | | The versions "needing" these fixes are at least five years old, so in an effort to save massive amounts of runtime and disk space (on aggregate at least) we can drop these lines. Reported-By: lintian maintainer-script-supports-ancient-package-version
* | Install translated apt-patterns(7) man pagesDavid Kalnischkies2020-11-051-4/+3
| | | | | | | | Reported-By: dh_missing
* | Rename CMake find_package helpers to avoid developer warningsDavid Kalnischkies2020-11-056-17/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | CMake Warning (dev) at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:273 (message): | The package name passed to `find_package_handle_standard_args` (Berkeley) | does not match the name of the calling package (BerkeleyDB). This can lead | to problems in calling code that expects `find_package` result variables | (e.g., `_FOUND`) to follow a certain pattern. | Call Stack (most recent call first): | CMake/FindBerkeleyDB.cmake:57 (find_package_handle_standard_args) | CMakeLists.txt:83 (find_package) | This warning is for project developers. Use -Wno-dev to suppress it. And indeed, we checked for BERKLEY_DB_FOUND which was not defined so our HAVE_BDB was not set – just that it is never used, so it wasn't noticed.
* | Portuguese manpages translation updateAmérico Monteiro2020-11-041-10/+17
| | | | | | | | Closes: #968414
* | Remove expired domain that became nsfw from debian/changelogJulian Andres Klode2020-10-301-1/+0
| | | | | | | | | | | | mirror.fail points to porn now apparently. Cc: stable
* | pkgnames: Do not exclude virtual packages with --all-namesJulian Andres Klode2020-10-262-8/+5
| | | | | | | | | | | | | | | | | | We accidentally excluded virtual packages by excluding every group that had a package, but where the package had no versions. Rewrite the code so the lookup consistently uses VersionList() instead of FirstVersion and FindPkg("any") - those are all the same, and this is easier to read.
* | pkgnames: Correctly set the default for AllNames to falseJulian Andres Klode2020-10-262-1/+24
| | | | | | | | | | | | | | We passed "false" instead of false, and that apparently got cast to bool, because it's a non-null pointer. LP: #1876495
* | Release 2.1.112.1.11Julian Andres Klode2020-10-215-6/+21
| |
* | Do not produce late error if immediate configuration fails, just warnJulian Andres Klode2020-10-211-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We are seeing more and more installations fail due to immediate configuration issues related to libc6. Immediate configuration is supposed to ensure that an essential package is configured immediately, just in case some other packages use a part of the essential package that only works if that package is configured. This used to be a warning, it was turned into an error in some commit I can't remember right now, but importantly, the error missed a return, which means that ordering completed succesfully and packages were being installed anyway; and after all that happened successfully, we'd print an error at the end and exit with an error code, which is not super useful. Revert the error back to a warning such that the behavior stays the same but we do not fail (unless we mess up ordering which then gets caught by a consistency check later on. Closes: #953260 Closes: #972552 LP: #1871268
* | Dutch manpages translation updateFrans Spiesschaert2020-09-102-35/+29
| | | | | | | | | | | | Closes: #970037 [jak: Fix typo extended_status -> extended_states]
* | doc: Bump Ubuntu release from focal to groovyJulian Andres Klode2020-09-091-1/+1
|/
* Fix "extended_states" typo in apt-mark(8)JCGoran2020-08-2711-19/+19
| | | | Closes: #969086
* Release 2.1.102.1.10Julian Andres Klode2020-08-1148-402/+191
|
* Merge branch 'pu/http-debug' into 'master'Julian Andres Klode2020-08-114-66/+58
|\ | | | | | | | | Add better acquire debugging support See merge request apt-team/apt!130
| * Rewrite HttpServerState::Die()Julian Andres Klode2020-08-111-28/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old code was fairly confusing, and contradictory. Notably, the second `if` also only applied to the Data state, whereas we already terminated the Data state earlier. This was bad. The else fallback applied in three cases: (1) We reached our limit (2) We are Persistent (3) We are headers Now, it always failed as a transient error if it had nothing left in the buffer. BUT: Nothing left in the buffer is the correct thing to happen if we were fetching content. Checking all combinations for the flags, we can compare the results of Die() between 2.1.7 - the last "known-acceptable-ish" version and this version: 2.1.7 this Data !Persist !Space !Limit OK (A) OK Data !Persist !Space Limit OK (A) OK Data !Persist Space !Limit OK (C) OK Data !Persist Space Limit OK OK Data Persist !Space !Limit ERR ERR * Data Persist !Space Limit OK (B) OK Data Persist Space !Limit ERR ERR Data Persist Space Limit OK OK => Data connections are OK if they have not reached their limit, or are persistent (in which case they'll probably be chunked) Header !Persist !Space !Limit ERR ERR Header !Persist !Space Limit ERR ERR Header !Persist Space !Limit OK OK Header !Persist Space Limit OK OK Header Persist !Space !Limit ERR ERR Header Persist !Space Limit ERR ERR Header Persist Space !Limit OK OK Header Persist Space Limit OK OK => Common scheme here is that header connections are fine if they have read something into the input buffer (Space). The rest does not matter. (A) Non-persistent connections with !space always enter the else clause, hence success (B) no Space means we enter the if/else, we go with else because IsLimit(), and we succeed because we don't have space (C) Having space we do enter the while (WriteSpace()) loop, but we never reach IsLimit(), hence we fall through. Given that our connection is not persistent, we fall through to the else case, and there we win because we have data left to write.
| * http: Fully flush local file both before/after server readJulian Andres Klode2020-08-113-19/+26
| | | | | | | | | | | | | | | | | | | | | | We do not want to end up in a code path while reading content from the server where we have local data left to write, which can happen if a previous read included both headers and content. Restructure Flush() to accept a new argument to allow incomplete flushs (which do not match our limit), so that it can flush as far as possible, and modify Go() and use that before and after reading from the server.
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-01 17:22:59 +0000