summaryrefslogtreecommitdiff
path: root/apt-pkg/pkgcachegen.cc
Commit message (Collapse)AuthorAgeFilesLines
* Fix wrongful std::make_unique conversionJulian Andres Klode2026-05-171-5/+5
| | | | | | | We inadvertently changed a CreateDynamicMMap call. Fix that function instead. Regression-of: 23dba65f031954df896bc3c6dfb1a9705574886b
* apt: modernize to make_unique C++17Herman Semenoff2026-04-081-5/+5
| | | | | | | References: - https://stackoverflow.com/questions/79700634/pros-and-cons-of-make-unique-vs-direct-constructor-call-in-c17 - https://towardsdev.com/why-std-make-unique-beats-new-in-modern-c-7e2ba653737e - https://www.sololearn.com/en/Discuss/3334779/where-make_unique-is-better-than-unique_ptrraw-pointer
* Correctly copy provided version in m-a lazy-initJulian Andres Klode2025-11-101-1/+1
| | | | | | | | When a provides was copied, we inadvertently copied the package version instead of the provided version. Copy the provided version instead. Closes: #1120463
* Implement architecture variantsJulian Andres Klode2025-10-251-1/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Architecture variants are children of an architecture that share the same ABI but are optimized for different ISA levels. They are available in Ubuntu 25.10 and newer, and not supported in Debian or other distributions. A deb built for a variant contains the Architecture-Variant field, and the Architecture field points to the baseline, for example: Architecture: amd64 Architecture-Variant: amd64v3 However, the apt-get indextargets command reports the variant in the Architecture: field, and most of the code in APT presents the variant as the architecture. There are two types of variants: 1. Standalone variants are recorded in the Architectures field of the Release file as if they were a real architecture: Architectures: amd64 amd64v3 Standalone architecture variants only fetch the standalone architecture variant's Packages file. To do this, this patch changes the code such that the variants indextargets "supplant" the base targets. This may have complicated outcomes on the apt-get indextargets command. 2. Other variants can only be identified by their files being recorded with hashes in the Release file. APT fetches both the base architecture's as well as the variant's Packages file. Variants are configured in the APT::Architecture-Variants list. Image builders may want to build specific variant images using APT::Architecture-Variants { "amd64v3"; } But this commit also implements an automatic discovery mechanism using the varianttable and /proc/cpuinfo. APT::Architecture-Variants "auto";
* Add 'Include'/'Exclude' options to limit packages used from a repositoryJulian Andres Klode2025-05-191-0/+26
| | | | | | | | | | | | | 'Include' allows limiting a repository to a allowlist of packages, 'Exclude' allows removing packages from a repository. Implementation wise this is not the nicest as it goes via the IndexTarget options, so we first parse our option value into a vector, then we turn the vector into a string, and then vectorize it again. That said, we support both "," and " " as separators due to the construction (we split by "," then we join using " "). Gbp-Dch: full
* Fix crash in VersionExtra using --with-source etcJulian Andres Klode2025-02-241-2/+4
| | | | | | | | | | | | | | | | | | | | VersionExtra holds, at runtime, the SHA256 of the packages being added to try to avoid deduplicating obviously different packages; the SHA256 itself is too large to store in the cache. When volatile sources are used, the VersionExtra array is not initialized, or rather it used to be initialized to 32*1024 zero elements as a performance optimization. This meant that specifying --with-source with a package that looks like a duplicate of a version with an ID >= 32*1024 triggered the assertion; so we could not reproduce it easily in the test suite. Remove the optimized initialization to make the behavior more uniform; and treat too large version IDs as having no SHA256. Closes: #1098702
* cache: Introduce partial SourceVersion supportJulian Andres Klode2025-02-151-0/+5
| | | | | | | | | | | | This is the first step that introduces a 1:1 mapping between version and source version. In a future version this can use the fields currently marked unavailable to deduplicate the SourceVersion objects across the group. The policy gains a member for storing pins for sourceversions. Together, in the future we should be able to determine candidates for source versions.
* Run unifdef -DAPT_PKG_ABI=700 againJulian Andres Klode2025-02-151-10/+0
| | | | | main picked up a couple new ifdefs since the branch started; so clean them up again.
* Drop APT::StringViewнаб2025-02-141-1/+1
|
* Record all available sources for descriptionsDavid Kalnischkies2025-01-301-53/+55
| | | | | | | | | | | | Avoiding duplications is one thing, but it could also be handy to know all sources for a description so that you can e.g. avoid opening another file or similar such if its also present in one you already have open. Given how the structures were set up this was probably intended all along and either worked at some point or was abandoned for the duplicates that we started culling heavily with the introduction of MultiArch as at that point they blew up the cache and needed to be dealt with.
* Do not create descriptions structs for each architectureDavid Kalnischkies2025-01-301-6/+9
| | | | | | | | The architectures are supposed to share the descriptions as they are (very usually) the same, but the code was happily creating structs for every architecture of a package which is especially pointless as for e.g. Debian those will be the short description likely superseded in output and all uses by the 'en' "translation".
* Drop unused version stanza size info from binary cacheDavid Kalnischkies2025-01-301-3/+8
| | | | | | | | | | | | | | Same reason as with the description size before, but this one had actually one extremely dated user in src:apt left with dumpavail, which is changed to an implementation that is (in comparison) a bit slower, but should be good enough for a command that is ideally not used anymore (even if codesearch suggests otherwise). After all, what makes this a bit slower is not that we don't know the size, but that this one does additional work preparing for accessing the fields which we don't do. In exchange pkgTagFile has a better implementation for "Jitter" than the old code that we would "just" need to copy here if speed were really an issue.
* Drop unused description size info from binary cacheDavid Kalnischkies2025-01-301-3/+8
| | | | | | | | | | | Nothing in src:apt uses this information and that isn't all to surprising as it is usually not that interesting to know how big a specific stanza of a description is as the max size is available and you are usually working with more than one description. So this information just takes up space in our binary cache… 64 bits per structure which can quickly add up to whole MBs depending on how many (and which) sources you have enabled.
* std::string_view const & -> std::string_viewнаб2024-11-141-2/+2
| | | | Fixes: 2c03eed649068c8838d971ceeb1a09603dcc8e9c
* apt-pkg/pkgcachegen.cc, apt-pkg/pkgcachegen.h: APT::StringView -> ↵наб2024-11-121-66/+67
| | | | std::string_view
* apt-pkg/edsp/edsplistparser.cc: APT::StringView -> std::string_viewнаб2024-11-121-2/+2
|
* pkgcachegen: Do not try to chmod or write to /dev/nullJulian Andres Klode2024-05-221-0/+4
| | | | | | | When -o Dir::Cache::pkgcache or -o Dir::Cache is set to /dev/null we accidentally changed its mode to 0666; which is certainly not intended. Avoid this issue by exiting the function early if the file specified is /dev/null.
* Modernize standard library includesJulian Andres Klode2024-02-201-2/+2
| | | | | | This was automated with sed and git-clang-format, and then I had to fix up the top of policy.cc by hand as git-clang-format accidentally indented it by two spaces.
* pkgcachegen: Use placement new to construct headerJulian Andres Klode2024-01-121-1/+1
| | | | | | | | | | | | | | | | | Avoid copying the header from a stack allocated object as this will copy uninitialized padding bytes into the cache, triggering valgrind errors which people then use as a strawman for unrelated errors on armhf. In an optimal world we should annotate the allocator however such that valgrind actually does treat those bytes as uninitialized and then supress warnings in the harmless places, such that when you then go and try to access it in a place that matters, you do get an error for uninitialized memory. Currently any access within the pool will be considered initialized which is clearly suboptimal. But this is very much a TBD topic and involves annotating the allocator everywhere.
* Compare SHA256 to check if versions are really the sameJulian Andres Klode2023-08-021-2/+25
| | | | | | | | | | | | | If we know both SHA256, and they're different, the packages are. This approach stores the SHA256 only at runtime, avoiding the overhead of storing it on-disk, because when we update repositories we update all of them anyhow. Note that pkgCacheGenerator is hidden, so we can just modify its ABI, hooray. Closes: #931175 LP: #2029268
* Do not make provides of M-A:allowed implicit M-A:foreignDavid Kalnischkies2021-09-041-5/+5
| | | | | | | | | | | | | | | | | | As we don't know which architectures we will deal with and to avoid creating many "unneeded" packages (and provides) the cache generation uses a scheme of on-demand creation (see ecc138f858). This assumed a particular handling of :any which got changed later (3addaba1ff) making this code path not only no longer needed for M-A:allowed, but actually wrong as it would go on and create provides for the explicit Provides of a package as if the package would be M-A:foreign. The result was that a package A:amd64 providing B tagged as M-A:allowed would satisfy a "C:armel depends on B". Note that this bug does NOT effect "C:armel depends on A" which is (correctly) not satisfied as before. References: ecc138f858, 3addaba1ff
* Use size of the old cache as APT::Cache-Start defaultDavid Kalnischkies2021-02-041-1/+10
| | | | | | | Depending on your configured source 25 MB is hardly enough, so the mmap housing the cache while it is build has to grow. Repeatedly. We can cut down on the repeats of this by keeping a record of the size of the old cache assuming the sizes will remain roughly in the same ballpark.
* Avoid undefined pointer arithmetic while growing mmapDavid Kalnischkies2021-02-041-40/+23
| | | | | | | | | The undefined behaviour sanitizer complains with: runtime error: addition of unsigned offset to 0x… overflowed to 0x… Compilers and runtime do the right thing in any case and it is a codepath that can (and ideally should) be avoided for speed reasons alone, but fixing it can't hurt (too much).
* pkgcachegen: Avoid write to old cache for Version::ExtraJulian Andres Klode2021-01-131-1/+2
| | | | | | | | Assigning the result of AllocateInMap directly to Ver->d caused Ver->d to be resolved first, and hence if Ver was remapped during the AllocateInMap, we were trying to assign to the old value. Closes: #980037
* Add support for Phased-Update-PercentageJulian Andres Klode2021-01-081-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | This adds support for Phased-Update-Percentage by pinning upgrades that are not to be installed down to 1. The output of policy has been changed to add the level of phasing, and documentation has been improved to document how phased updates work. The patch detects if it is running in a chroot, and if so, always includes phased updates, restoring classic apt behavior to avoid behavioral changes on buildd chroots. Various options are added to control this all: * APT::Get::{Always,Never}-Include-Phased-Updates and their legacy update-manager equivalents to always or never include phased updates * APT::Machine-ID can be set to a UUID string to have all machines in a fleet phase the same * Dir::Etc::Machine-ID is weird in that it's default is sort of like ../machine-id, but not really, as ../machine-id would look up $PWD/../machine-id and not relative to Dir::Etc; but it allows you to override the path to machine-id (as opposed to the value) * Dir::Bin::ischroot is the path to the ischroot(1) binary which is used to detect whether we are running in a chroot.
* apt-pkg: default visibility to hiddenJulian Andres Klode2020-02-261-1/+0
|
* Silence narrow conversion warnings, add error checksJulian Andres Klode2020-02-251-7/+20
| | | | | | When converting a long offset to a uint32_t to be stored in the map, check that this is safe to do. If the offset is negative, or we lose data in the conversion, we lost.
* Make map_pointer<T> typesafeJulian Andres Klode2020-02-241-21/+22
| | | | | | | | | | | Instead of just using uint32_t, which would allow you to assign e.g. a map_pointer<Version> to a map_pointer<Package>, use our own smarter struct that has strict type checking. We allow creating a map_pointer from a nullptr, and we allow comparing map_pointer to nullptr, which also deals with comparisons against 0 which are often used, as 0 will be implictly converted to nullptr.
* Wrap AllocateInMap with a templated versionJulian Andres Klode2020-02-241-14/+16
|
* Replace map_pointer_t with map_pointer<T>Julian Andres Klode2020-02-241-36/+36
| | | | | | This is a first step to a type safe cache, adding typing information everywhere. Next, we'll replace map_pointer<T> implementation with a type safe one.
* Use a 32-bit djb VersionHash instead of CRC-16Julian Andres Klode2020-02-181-3/+3
|
* NewGroup: Create GrpIterator after allocation (fix segfault)Julian Andres Klode2020-01-271-1/+2
| | | | | | | NewGroup created a GrpIterator and then called WriteStringInMap() which might remap the cache, causing the iterator to go invalid. Avoid this simply by creating the iterator later on.
* NewProvidesAllArch: Check if group is empty before using itJulian Andres Klode2020-01-161-1/+1
| | | | | | | | | | | | | APT 1.9.6 introduced empty groups by making use of groups to deduplicate package names. This is not normally a problem, but here we assumed that every group has at least one package. This caused a problem because automake was providing automake-1.16 while having the source package automake-1.16. So we found the automake-1.16 group, iterated over its empty package list, trying to store the provides (which hence never happened). LP: #1859952
* Remove includes of (md5|sha1|sha2).h headersJulian Andres Klode2020-01-141-1/+0
| | | | Remove it everywhere, except where it is still needed.
* Avoid extra out-of-cache hash table deduplication for package namesJulian Andres Klode2020-01-081-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We were de-duplicating package name strings in StoreString, but also deduplicating most of them by them being in groups, so we had extra hash table lookups that could be avoided in NewGroup(). To continue deduplicating names across binary packages and source packages, insert groups for source packages as well. This is also a good first step in allowing efficient lookup of packages by source package - we can extend Group later by a list of SourceVersion objects, or alternatively, simply add a by-source chain into pkgCache::Version. This change improves performance by about 10% (913 to 814 ms), while having no significant overhead on the cache size: --- before +++ after @@ -1,7 +1,7 @@ -Total package names: 109536 (2.191 k) -Total package structures: 118689 (4.748 k) +Total package names: 119642 (2.393 k) +Total package structures: 118687 (4.747 k) Normal packages: 83309 - Pure virtual packages: 3365 + Pure virtual packages: 3363 Single virtual packages: 17811 Mixed virtual packages: 1973 Missing: 12231 @@ -10,21 +10,21 @@ Total distinct descriptions: 149291 (3.583 k) Total dependencies: 484135/156650 (12,2 M) Total ver/file relations: 57421 (1.378 k) Total Desc/File relations: 18219 (437 k) -Total Provides mappings: 29963 (719 k) +Total Provides mappings: 29959 (719 k) Total globbed strings: 226993 (5.332 k) Total slack space: 26,8 k -Total space accounted for: 38,1 M +Total space accounted for: 38,3 M Total buckets in PkgHashTable: 50503 - Unused: 5727 - Used: 44776 - Utilization: 88.6601% - Average entries: 2.65073 + Unused: 5728 + Used: 44775 + Utilization: 88.6581% + Average entries: 2.65074 Longest: 60 Shortest: 1 Total buckets in GrpHashTable: 50503 - Unused: 5727 - Used: 44776 - Utilization: 88.6601% - Average entries: 2.44631 - Longest: 10 + Unused: 4649 + Used: 45854 + Utilization: 90.7946% + Average entries: 2.60919 + Longest: 11 Shortest: 1
* pkgcachegen: Remove deprecated functionsJulian Andres Klode2019-02-261-5/+0
|
* pkgcache: Remove deprecated bitsJulian Andres Klode2019-02-261-3/+0
|
* Fix typo reported by codespell in code commentsDavid Kalnischkies2018-11-251-1/+1
| | | | | | | | No user visible change expect for some years old changelog entries, so we don't really need to add a new one for this… Reported-By: codespell Gbp-Dch: Ignore
* Remove obsolete RCS keywordsGuillem Jover2018-05-071-1/+0
| | | | Prompted-by: Jakub Wilk <jwilk@debian.org>
* Fix various typos reported by spellcheckersDavid Kalnischkies2018-05-051-1/+1
| | | | | Reported-By: codespell & spellintian Gbp-Dch: Ignore
* do not remap current files if nullptrs in cache generationDavid Kalnischkies2017-12-241-10/+11
| | | | | | | | | | | | | | If the cache needs to grow to make room to insert volatile files like deb files into the cache we were remapping null-pointers making them non-null-pointers in the process causing trouble later on. Only the current Releasefile pointer can currently legally be a nullpointer as volatile files have no release file they belong to, but for safety the pointer to the current Packages file is equally guarded. The option APT::Cache-Start can be used to workaround this problem. Reported-By: Mattia Rizzolo on IRC
* convert various c-style casts to C++-styleDavid Kalnischkies2017-12-131-4/+4
| | | | | | | | | | gcc was warning about ignored type qualifiers for all of them due to the last 'const', so dropping that and converting to static_cast in the process removes the here harmless warning to avoid hidden real issues in them later on. Reported-By: gcc Gbp-Dch: Ignore
* deprecate the single-line deprecation ignoring macroDavid Kalnischkies2017-12-131-1/+3
| | | | | | | | | | gcc has problems understanding this construct and additionally thinks it would produce multiple lines and stuff, so to keep using it isn't really worth it for the few instances we have: We can just write the long form there which works better. Reported-By: gcc Gbp-Dch: Ignore
* Reformat and sort all includes with clang-formatJulian Andres Klode2017-07-121-12/+12
| | | | | | | | | | | | | This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
* Drop cacheiterators.h includeJulian Andres Klode2017-07-121-1/+0
| | | | | Including cacheiterators.h before pkgcache.h fails because pkgcache.h depends on cacheiterators.h.
* avoid validate/delete/load race in cache generationDavid Kalnischkies2017-01-191-28/+31
| | | | | | | Keeping the Fd of the cache file we have validated around to later load it into the mmap ensures not only that we load the same file (which wouldn't really be a problem in practice), but that this file also still exists and wasn't deleted e.g. by a 'apt clean' call run in parallel.
* Do not use MD5SumValue for Description_md5()Julian Andres Klode2016-11-221-13/+13
| | | | | | | | | | | Our profile says we spend about 5% of the time transforming the hex digits into the binary format used by HashsumValue, all for comparing them against the other strings. That makes no sense at all. According to callgrind, this reduces the overall instruction count from 5,3 billion to 5 billion in my example, which roughly matches the 5%.
* Compare size before data when ordering cache bucket entriesJulian Andres Klode2016-11-221-2/+2
| | | | | | | This has the effect of significantly reducing actual string comparisons, and should improve the performance of FindGrp a bit, although it's hardly measureable (callgrind says it uses 10% instructions less now).
* do not treat same-version local debs as downgradeDavid Kalnischkies2016-07-011-2/+15
| | | | | | As the volatile sources are parsed last they were sorted behind the dpkg/status file and hence are treated as a downgrade, which isn't really what you want to happen as from a user POV its an upgrade.
* Prevent double remapping of iterators and string viewsJulian Andres Klode2016-03-061-8/+22
| | | | | | | | | | | If an iterator or a stringview has multiple dynamic objects registered with it, it may be remapped twice. Prevent that by noting which iterators/views we have seen and not remapping one if we have already seen it. We most likely do not have any instance of multiple dynamics on a single object, but let's play safe - the overhead is not high.
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2026-07-18 10:02:06 +0000