summaryrefslogtreecommitdiff
path: root/apt-pkg
Commit message (Collapse)AuthorAgeFilesLines
* Do not sync the cache fileJulian Andres Klode2015-12-291-2/+0
| | | | Integrity is taken care of by the checksum now.
* Add support for calculating hashes over the entire cacheJulian Andres Klode2015-12-293-5/+43
|
* pkgCacheGenerator: Allow passing down an already created cacheJulian Andres Klode2015-12-293-5/+19
| | | | | If we already have opened a cache, there is no point in having to open it again.
* pkgTagSection::Scan: Fix read of uninitialized valueJulian Andres Klode2015-12-291-1/+1
| | | | | We ignored the boundary of the buffer we were reading in while scanning for spaces.
* strutl.cc: Add declarations for the compat _ascii() functionsJulian Andres Klode2015-12-291-0/+2
| | | | | | This shuts up gcc Gbp-Dch: ignore
* Turn tolower_ascii() and isspace_ascii() into inline functionsJulian Andres Klode2015-12-292-11/+21
| | | | | | | | | | To preserve compatibility, the new inline functions have _inline as a suffix, and a macro defines the old names to refer to the inline variants. The old functions are still preserved for binary compatibility. Also simplify the implementation of both functions.
* Switch to DJB hashing and use prime number as table sizeJulian Andres Klode2015-12-291-7/+7
| | | | | | On my testing system, consisting of unstable and experimental, this reduces the average chain from 6.5 to 4.5, and the longest chain from 17 to 15.
* BufferedFileFdPrivate: Make InternalFlush() save against errorsJulian Andres Klode2015-12-281-8/+8
| | | | | | | | | | | Previously, if flush errored inside the loop, data could have already been written to the wrapped descriptor without having been removed from the buffer. Also try to work around EINTR here. A better solution might be to have the individual privates detect an interrupt and return 0 in such a case, instead of relying on errno being untouched in between the syscall and the return from InternalWrite.
* aptconfiguration: Set default compression level to 6Julian Andres Klode2015-12-281-5/+5
| | | | | | | | | | | | | | | | | | Since commit 7a68effcb904b4424b54a30e448b6f2560cd1078, the xz and lzma compressors read the level of compression they shall use. A default of -9 is too much for them, this will use 674 MB, according to the xz manual page. Level -6 on the other hand only needs 94 MB memory for compression. This causes autopkgtest failures in the test-compressed-indexes test, as not enough memory exists to proceed. Change the other compression levels to 6 as well: The gzip and bzip2 FileFd backends do not read them, and use their code's default level which is 6, so do the same for external methods.
* BufferedWriter: flushing: Check for written < size instead of <=Julian Andres Klode2015-12-281-3/+1
| | | | | This avoids some issues with InternalWrite returning 0 because it just cannot write stuff at the moment.
* deal with empty values properly in deb822 parserDavid Kalnischkies2015-12-271-1/+3
| | | | | | | | Regression introduced in 8710a36a01c0cb1648926792c2ad05185535558e, but such fields are unlikely in practice as it is just as simple to not have a field at all with the same result of not having a value. Closes: 808102
* allow repositories to forbid arch:all for specific index targetsDavid Kalnischkies2015-12-275-3/+30
| | | | | | | | | | | | | | | | | | | | | | | | Debian has a Packages file for arch:all already, but the arch:any files contain arch:all packages as well, so downloading it would be a total waste of resources. Getting this solved is on the list of things to do, but it is also the hardest part – for index targets like Contents the situation is much easier and less server/client implementations are involved so we might not want to stall them. A repository can now declare via: No-Support-for-Architecture-all: Packages that even if an arch:all Packages exists, it shouldn't be downloaded, so that support for Contents files can be added now. See also 1dd20368486820efb6ef4476ad739e967174bec4 for the implementation of downloading arch:all index targets, which this is limiting. The field uses the name of the target from the apt configuration for simplicity and is negative by design as this field is intended to be supported/needed only for a "short" time (one or two Debian releases). While this commit theoretically supports any target, its expected to only see "Packages" as a value in reality.
* pkgcachegen.h: Hack around unordered_map not existing before C++11Julian Andres Klode2015-12-271-0/+5
| | | | | This is for public users only, which cannot use the class at all, except for the static methods.
* FileFd: Add a buffered writing modeJulian Andres Klode2015-12-272-0/+153
| | | | | This is somewhat experimental right now, and might not work for everyone, so it is on an opt-in basis.
* FildFd: Introduce a Flush() function and call it from Close()Julian Andres Klode2015-12-272-0/+16
| | | | The flush function can be used for buffered writers.
* FileFdPrivate: Add getter and setter for fieldsJulian Andres Klode2015-12-271-9/+42
| | | | | We will soon implement a buffered writing decorator and we will need to forward attribute changes to those.
* fileutl: simple_buffer: Add write() and full() methodsJulian Andres Klode2015-12-271-0/+11
| | | | | | These can be used to implement write buffering Gbp-Dch: ignore
* fileutl: simple_buffer: Mark accessors as constJulian Andres Klode2015-12-271-2/+3
| | | | | | Suggested by David. Gbp-Dch: ignore
* FileFdPrivate: Extract SimpleBuffer and mark it as hiddenJulian Andres Klode2015-12-271-21/+24
| | | | Gbp-Dch: ignore
* ParseDepends: Mark branches for build-dep parsing as unlikelyJulian Andres Klode2015-12-271-2/+2
| | | | | | We do not see those branches at all during normal mode of operation (that is, during cache generation), so tell the compiler about it.
* debListParser: Do not validate Description-md5 for correctness twiceJulian Andres Klode2015-12-271-2/+4
| | | | | The Set() method returns false if the input is no hex number, so simply use that.
* Hex2Digit: Do not use isxdigit()Niels Thykier2015-12-271-4/+9
| | | | | | | We directly check if we are a hex digit in HexDigit, so use that information. [jak@debian.org: Commit message wording]
* debListParser: ParseDepends: Only query native arch if neededJulian Andres Klode2015-12-271-1/+2
| | | | | | This makes the code parsing architecture lists slower, but on the other hand, improves the more generic case of reading dependencies from Packages files.
* pkgcachegen: Use std::unordered_map instead of std::mapJulian Andres Klode2015-12-272-7/+7
| | | | | std::unordered_map is faster than std::map in our use case, reducing cache generation time by about 10% in my benchmark.
* Convert most callers of isspace() to isspace_ascii()Julian Andres Klode2015-12-275-32/+32
| | | | | This converts all callers that read machine-generated data, callers that might work with user input are not converted.
* Introduce isspace_ascii() for use by parsersJulian Andres Klode2015-12-272-0/+19
| | | | This is like isspace(), but ignores the current locale.
* Refactor InternalReadLine to not unroll Size == 0 caseJulian Andres Klode2015-12-261-5/+4
| | | | | | There is not much point and this is more readable. Gbp-Dch: ignore
* Change InternalReadLine to always use buffer.read() return valueJulian Andres Klode2015-12-261-12/+8
| | | | | | | | | This is mostly a documentation issue, as the size we want to read is always less than or equal to the size of the buffer, so the return value will be the same as the size argument. Nonetheless, people wondered about it, and it seems clearer to just always use the return value.
* Get rid of memmove() in our read bufferingJulian Andres Klode2015-12-261-76/+57
| | | | | | This further improves our performance, and rred on uncompressed files now spents 78% of its time in writing. Which means that we should really look at buffering those.
* Use a hardcoded buffer size of 4096 to fix performanceJulian Andres Klode2015-12-261-4/+2
| | | | | | | | | | | | | | The code uses memmove() to move parts of the buffer to the front when the buffer is only partially read. By simply reading one page at a time, the maximum size of bytes that must be moved has a hard limit, and performance improves: In one test case, consisting of a 430 MB Contents file, and a 75K PDiff, applying the PDiff previously took about 48 seconds and now completes in 2 seconds. Further speed up can be achieved by buffering writes, they account for about 60% of the run-time now.
* Mark all FileFdPrivate classes as hidden1.1.6Julian Andres Klode2015-12-241-6/+6
| | | | Gbp-Dch: ignore
* fix new[] vs delete mismatch introduced by b3db9d81David Kalnischkies2015-12-231-7/+7
| | | | | | | | And as we are at it lets fix the 'style' issue I introduced with the filefd changes as well. Reported-By: gcc -fsanitize's & cppcheck Git-Dch: Ignore
* use a dynamic buffer for ReadLineDavid Kalnischkies2015-12-231-15/+22
| | | | | | | | | We don't need the buffer that often - only for ReadLine - as it is only occasionally used, so it is actually more efficient to allocate it if needed instead of statically by default. It also allows the caller to influence the buffer size instead of hardcoding it. Git-Dch: Ignore
* implement a buffer system for FileFd::ReadLineDavid Kalnischkies2015-12-231-27/+140
| | | | | | | | | | | | | | | The default implementation of ReadLine was very naive by just reading each character one-by-one. That is kinda okay for libraries implementing compression as they have internal buffers (but still not great), but while working with files directly or via a pipe as there is no buffer there so all those reads are in fact system calls. This commit introduces an internal buffer in the FileFd implementation which is only used by ReadLine. The more low-level Read and all other actions remain unbuffered – they just changed to deal with potential "left-overs" in the buffer correctly. Closes: 808579
* parse xz-compression level from configurationDavid Kalnischkies2015-12-221-2/+28
| | | | | | | If we use the library to compress xz, still try to understand and pick up the arguments we would have used to call xz to figure out which level the user wants us to use instead of defaulting to level 6 (which is the default level of xz).
* follow dpkg and xz and use CRC64 for xz compressionDavid Kalnischkies2015-12-221-1/+1
| | | | | | | | dpkg switched from CRC32 to CRC64 in 777915108d9d36d022dc4fc4151a615fc95e5032 with the message: | This is the default CRC used by the xz command-line tool, align with | it and switch from CRC32 to CRC64. It should provide slightly better | detection against damaged data, at a negligible speed difference.
* shuffle compressor-specific code into private subclassesDavid Kalnischkies2015-12-222-635/+692
| | | | | | | | | | | | | | This isn't implementing any new features, it is "just" moving code around from FileFd methods which decided on each call how to handle the request by including all logic for all possible compressor backends in the method body to a model in which backend-specifics are implemented in a FileFdPrivate subclass. This avoids a big chunk of #ifdef's and should make it a tiny bit more obvious which backend uses which code. The execution of the idea is slightly uglified by the need to preserve ABI and API which causes liberal befriending. Git-Dch: Ignore
* Do not try to read in FileFd::Read() if Size is 0Julian Andres Klode2015-12-191-3/+2
| | | | | | | There's no point trying to read 0 bytes, so let's just not do this and switch to a while loop like in Write(). Gbp-Dch: ignore
* Do nothing in FileFd::Write() if Size is 0Julian Andres Klode2015-12-191-7/+5
| | | | | | | | | | | Turn the do-while loop into while loops, so it simply does nothing if the Size is already 0. This reverts commit c0b271edc2f6d9e5dea5ac82fbc911f0e3adfa7a which introduced a fix for a specific instance of the issue in the CopyFile() function. Closes: #808381
* CopyFile: avoid failing on EOF on some systemsPino Toscano2015-12-191-1/+1
| | | | | | | | | On EOF, ToRead will be 0, which might trigger on some systems (e.g. on the Hurd) an error due to the invalid byte count passed to write(). The whole loop already checks for ToRead != 0, so perform the writing step only when there was actual data read. Closes: #808381
* CopyFile: fix BufSize to a sane valuePino Toscano2015-12-191-2/+2
| | | | | | | | | | | | | | Commit e977b8b9234ac5db32f2f0ad7e183139b988340d tries to make BufSize calculated based on the size of the buffer; the problem is that std::unique_ptr::size() returns a pointer to the data, so sizeof() equals to the size of a pointer (later divided by sizeof(char), which is 1). The result is that the CopyFile copies in chunks of 8 bytes, which is not exactly ideal... As solution, declare BufSize in advance, and use its value to allocate the Buf array. Closes: #808381
* pkgcache: Make hash arch-independent using fixed size integerJulian Andres Klode2015-12-141-2/+2
| | | | | | | | This helps writing test cases. Also adapt the test case that expected 64-bit. Nothing changes performance wise, the distribution of the hash values remains intact.
* tagfile: Hardcode error message for out of range integer valuesJulian Andres Klode2015-12-141-4/+3
| | | | | | This makes the test suite work on 32 bit-long platforms. Gbp-Dch: ignore
* non-existing directories don't need to be cleanedDavid Kalnischkies2015-12-141-0/+5
| | | | | | | | | | Trying to clean up directories which do not exist seems rather silly if you think about it, so let apt think about it and stop it. Depends a bit on the caller if this is fixing anything for them as they might try to acquire a lock or doing other clever things as apt does. Closes: 807477
* support regex and co in 'apt-cache policy $pkg' againDavid Kalnischkies2015-12-141-1/+1
| | | | | | | | | | Regression of 1e064088bf7b3e29cd36d30760fb3e4143a1a49a (1.1~exp4) which moved code around and renamed methods heavily ending up calling the wrong method matching packagenames only instead of calling the full array. Most commands work with versions, so this managed to fly under the radar for quite a while. Closes: 807870
* show a more descriptive error for weak Release filesDavid Kalnischkies2015-12-142-4/+25
| | | | | | | | | | | | | | If we can't work with the hashes we parsed from the Release file we display now an error message if the Release file includes only weak hashes instead of downloading the indexes and failing to verify them with "Hash Sum mismatch" even through the hashes didn't mismatch (they were just weak). If for some (unlikely) reason we have got weak hashes only for individual targets we will show a warning to this effect (again, befor downloading and failing the index itself). Closes: 806459
* parse .diff/Index hashes in reverse orderDavid Kalnischkies2015-12-131-3/+11
| | | | | | | | | | | | | | | | Reversing the parsing order ensures that we parse weaker hashes (like SHA1) before we touch newer/stronger hashes (like SHA256) as the weaker ones will usually be there for a longer time already with data already present, which we would discard if we start with the strong one first. The discarding is visible in the debug logs: File X wasn't in the list for the first parsed hash! (history) File X wasn't in the list for the first parsed hash! (patches) which if file X is part of the patch-path means apt will not find a path and fallback to acquire the whole file instead needlessly. If file X isn't part of the patch-path that is no problem, so that effects only the update-call which updates with patches coming from before and after the addition of a new hash.
* fix typos and docs in GlobalError documentationDavid Kalnischkies2015-12-131-10/+9
| | | | | Reported-By: Manuel A. Fernandez Montecelo <mafm@debian.org> Git-Dch: Ignore
* mmap: Define _DEFAULT_SOURCE instead of _BSD_SOURCEJulian Andres Klode2015-12-111-1/+1
| | | | | | Fixes a warning reported by gcc. Gbp-Dch: ignore
* Bump cache minor version to 2 to trigger rebuildsJulian Andres Klode2015-12-111-1/+1
| | | | | | | | With the package names now normalized to lower case, the caches of affected systems need to be rebuild. Adjust the minor version to trigger such a rebuild. Gbp-Dch: ignore
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2026-10-19 13:50:48 +0000