summaryrefslogtreecommitdiff
path: root/apt-pkg/pkgcachegen.cc
Commit message (Collapse)AuthorAgeFilesLines
* Fix typo reported by codespell in code commentsDavid Kalnischkies2018-11-251-1/+1
| | | | | | | | No user visible change expect for some years old changelog entries, so we don't really need to add a new one for this… Reported-By: codespell Gbp-Dch: Ignore
* Remove obsolete RCS keywordsGuillem Jover2018-05-071-1/+0
| | | | Prompted-by: Jakub Wilk <jwilk@debian.org>
* Fix various typos reported by spellcheckersDavid Kalnischkies2018-05-051-1/+1
| | | | | Reported-By: codespell & spellintian Gbp-Dch: Ignore
* do not remap current files if nullptrs in cache generationDavid Kalnischkies2017-12-241-10/+11
| | | | | | | | | | | | | | If the cache needs to grow to make room to insert volatile files like deb files into the cache we were remapping null-pointers making them non-null-pointers in the process causing trouble later on. Only the current Releasefile pointer can currently legally be a nullpointer as volatile files have no release file they belong to, but for safety the pointer to the current Packages file is equally guarded. The option APT::Cache-Start can be used to workaround this problem. Reported-By: Mattia Rizzolo on IRC
* convert various c-style casts to C++-styleDavid Kalnischkies2017-12-131-4/+4
| | | | | | | | | | gcc was warning about ignored type qualifiers for all of them due to the last 'const', so dropping that and converting to static_cast in the process removes the here harmless warning to avoid hidden real issues in them later on. Reported-By: gcc Gbp-Dch: Ignore
* deprecate the single-line deprecation ignoring macroDavid Kalnischkies2017-12-131-1/+3
| | | | | | | | | | gcc has problems understanding this construct and additionally thinks it would produce multiple lines and stuff, so to keep using it isn't really worth it for the few instances we have: We can just write the long form there which works better. Reported-By: gcc Gbp-Dch: Ignore
* Reformat and sort all includes with clang-formatJulian Andres Klode2017-07-121-12/+12
| | | | | | | | | | | | | This makes it easier to see which headers includes what. The changes were done by running git grep -l '#\s*include' \ | grep -E '.(cc|h)$' \ | xargs sed -i -E 's/(^\s*)#(\s*)include/\1#\2 include/' To modify all include lines by adding a space, and then running ./git-clang-format.sh.
* Drop cacheiterators.h includeJulian Andres Klode2017-07-121-1/+0
| | | | | Including cacheiterators.h before pkgcache.h fails because pkgcache.h depends on cacheiterators.h.
* avoid validate/delete/load race in cache generationDavid Kalnischkies2017-01-191-28/+31
| | | | | | | Keeping the Fd of the cache file we have validated around to later load it into the mmap ensures not only that we load the same file (which wouldn't really be a problem in practice), but that this file also still exists and wasn't deleted e.g. by a 'apt clean' call run in parallel.
* Do not use MD5SumValue for Description_md5()Julian Andres Klode2016-11-221-13/+13
| | | | | | | | | | | Our profile says we spend about 5% of the time transforming the hex digits into the binary format used by HashsumValue, all for comparing them against the other strings. That makes no sense at all. According to callgrind, this reduces the overall instruction count from 5,3 billion to 5 billion in my example, which roughly matches the 5%.
* Compare size before data when ordering cache bucket entriesJulian Andres Klode2016-11-221-2/+2
| | | | | | | This has the effect of significantly reducing actual string comparisons, and should improve the performance of FindGrp a bit, although it's hardly measureable (callgrind says it uses 10% instructions less now).
* do not treat same-version local debs as downgradeDavid Kalnischkies2016-07-011-2/+15
| | | | | | As the volatile sources are parsed last they were sorted behind the dpkg/status file and hence are treated as a downgrade, which isn't really what you want to happen as from a user POV its an upgrade.
* Prevent double remapping of iterators and string viewsJulian Andres Klode2016-03-061-8/+22
| | | | | | | | | | | If an iterator or a stringview has multiple dynamic objects registered with it, it may be remapped twice. Prevent that by noting which iterators/views we have seen and not remapping one if we have already seen it. We most likely do not have any instance of multiple dynamics on a single object, but let's play safe - the overhead is not high.
* deal better with (very) small apt::cache-start valuesDavid Kalnischkies2016-01-271-16/+24
| | | | | | | | | It is a bit academic to support values which aren't big enough to fit even the hashtables without resizing, but cleaning up ensures that we do the right thing (aka not segfaulting) even if something goes wrong in these deep layers. You still can't have very very small values through… Git-Dch: Ignore
* convert Version() and Architecture() to APT::StringViewDavid Kalnischkies2016-01-261-12/+15
| | | | | | Part of hidden classes, so conversion is abi-free. Git-Dch: Ignore
* reimplement build-dep via apts normal resolverDavid Kalnischkies2016-01-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | build-dep was implemented by parsing the build-dependencies of a package and figuring out which packages to install/remove based on this. That means that for the first level of dependencies build-dep was implementing its very own resolver with all the benefits (aka: bugs) this gives us for not using the existing resolver for all levels. Making this work involves generating a dummy binary package with fitting Depends and Conflicts and as we can't create them out of thin air the cache generation needs to be involved so we end up writing a Packages file which we want to parse – after we have parsed the other Packages files already. With .dsc/.deb files we could add them before we started parsing anything. With a bit of care we can avoid generating too much data we have to throw away again (as many parts assume that e.g. the count of packages doesn't change midair), so that on a speed front there shouldn't be much of a difference, but output can be slightly confusing as if we have a completely valid cache on disk the "Reading package lists... Done" is printed two times – but apt is pretty quick about it in that case. Closes: #137560, #444930, #489911, #583914, #728317, #812173
* use consistently the last : as name:arch separatorDavid Kalnischkies2016-01-251-1/+1
| | | | | | | | Proper debian packages do not contain ':' in the package name, so for real packages this is a non-issue, but apt itself frequently makes use of packages with such an illegal name for internal proposes. Git-Dch: Ignore
* always create pkg at the time pkg:arch is createdDavid Kalnischkies2016-01-251-16/+29
| | | | | | | | | To resolve dependencies like "pkg:arch" we create a package with the name "pkg:arch" and the architecture "any". We create these packages only if a dependency needs it as these kind of dependencies aren't that common. This commit ensured that in the even this architecture specific dependency is the only relation this package has we still create the underlying package to have them available in provides resolution.
* Remap another (non-parameter) StringViewJulian Andres Klode2016-01-231-1/+3
| | | | | | | | | I only looked at parameters in the previous commit, which was not enough: One place also generated local string views. In this case, we only need to make ArchA dynamic, as NameA is not used after the FindPkg() call. Gbp-Dch: ignore
* Remap StringView instances pointing into the cacheJulian Andres Klode2016-01-231-0/+19
| | | | | | | | | | | | | | | | | | | | It turns out that StringViews might need to be remapped in some places because they come from the cache. For example, some sites pass a Ver.VerStr() to NewProvides(). Such a StringView would become invalid during the duration of the call if the cache is remapped, causing the program to die with a segmentation fault. We can take care of those issues by remapping string views in the same way we remap all the iterators. String views are only remapped if they point into the cache though, this allows us to write more generic code on the callee site without having to check whether the view points into the cache or not. That's not as efficient as possible, but the overhead does not appear to be measurable. Closes: #812251
* Pass the old map size to ReMap()Julian Andres Klode2016-01-231-4/+7
| | | | | | | This allows us to check if a value to be remapped was inside the cache or not, which will become useful at a later point. Gbp-Dch: ignore
* fix M-A:foreign provides creation for unknown archsDavid Kalnischkies2016-01-141-4/+12
| | | | | | | | Architectures for packages which do not belong to the native nor a foreign architecture (dubbed barbarian for now) which are marked M-A:foreign still provide in their own architecture even if not for others. Also, other M-A:foreign (and allowed) packages provide in these barbarian architectures.
* Store the size of strings in the cacheJulian Andres Klode2016-01-081-1/+2
| | | | | By storing the size of the string in the cache, we can make use of it when comparing the names in the hashtable in pkgCache::FindGrp.
* pkgCacheGenerator: CurMd5.Value() cannot be emptyJulian Andres Klode2016-01-081-4/+0
| | | | | It makes no sense to check if the value is empty, as it cannot be. It will always be a hexstring of exactly 32 bytes.
* pkgCacheGenerator::StoreString: Get rid of std::stringJulian Andres Klode2016-01-081-7/+5
| | | | | | | | | Instead of storing a string -> map_stringitem_t mapping, create our own data type that can point to either a normal string or a string inside the cache. This avoids the creation of any string and improves performance slightly (about 4%).
* Replace compare() == 0 checks with this == other checksJulian Andres Klode2016-01-081-1/+1
| | | | | | | This improves performance, as we now can ignore unequal strings based on their length already. Gbp-Dch: ignore
* pkgCacheGenerator: Use StringView for toStringJulian Andres Klode2016-01-081-5/+5
| | | | | | This removes some minor overhead. Gbp-Dch: ignore
* pkgCacheGenerator::StoreString: Move the string into the mapJulian Andres Klode2016-01-081-2/+2
| | | | | | | | Moving the string is likely faster than copying it. We could probably avoid strings alltogether in the future using some more crazy code, but I have not looked at that yet. Gbp-Dch: ignore
* Switch performance critical code to use APT::StringViewJulian Andres Klode2016-01-071-21/+22
| | | | | | This improves performance of the cache generation on my ARM platform (4x Cortex A15) by about 10% to 20% from 2.35-2.50 to 2.1 seconds.
* Do not sync the cache fileJulian Andres Klode2015-12-291-2/+0
| | | | Integrity is taken care of by the checksum now.
* Add support for calculating hashes over the entire cacheJulian Andres Klode2015-12-291-1/+7
|
* pkgCacheGenerator: Allow passing down an already created cacheJulian Andres Klode2015-12-291-3/+13
| | | | | If we already have opened a cache, there is no point in having to open it again.
* pkgcachegen: Use std::unordered_map instead of std::mapJulian Andres Klode2015-12-271-2/+2
| | | | | std::unordered_map is faster than std::map in our use case, reducing cache generation time by about 10% in my benchmark.
* add messages to our deprecation warnings in libaptDavid Kalnischkies2015-11-271-2/+2
| | | | Git-Dch: Ignore
* do not segfault in cache generation on mmap failureDavid Kalnischkies2015-11-201-1/+9
| | | | | | | | | | | | | | | Out of memory and similar circumstanzas could cause MMap::Map to fail and especially the mmap/malloc calls in it. With some additional checking we can avoid segfaults and similar in such situations – at least in theory as if this is a real out of memory everything we do to handle the error could just as well run into a memory problem as well… But at least in theory (if MMap::Map is made to fail always) we can deal with it so good that a user actually never sees a failure (as the cache it tries to load with it fails and is discarded, so that DynamicMMap takes over and a new one is build) instead of segfaulting. Closes: 803417
* apply various suggestions made by cppcheckDavid Kalnischkies2015-11-051-1/+0
| | | | | Reported-By: cppcheck Git-Dch: Ignore
* do not ignore differently versioned self-providesDavid Kalnischkies2015-09-141-1/+2
| | | | Reported-By: Konomi on IRC
* avoid using global PendingError to avoid failing too often too soonDavid Kalnischkies2015-09-141-19/+32
| | | | | | | | | | | | | | | | | | | Our error reporting is historically grown into some kind of mess. A while ago I implemented stacking for the global error which is used in this commit now to wrap calls to functions which do not report (all) errors via return, so that only failures in those calls cause a failure to propergate down the chain rather than failing if anything (potentially totally unrelated) has failed at some point in the past. This way we can avoid stopping the entire acquire process just because a single source produced an error for example. It also means that after the acquire process the cache is generated – even if the acquire process had failures – as we still have the old good data around we can and should generate a cache for (again). There are probably more instances of this hiding, but all these looked like the easiest to work with and fix with reasonable (aka net-positive) effects.
* implement dpkgs vision of interpreting pkg:<arch> dependenciesDavid Kalnischkies2015-09-141-0/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | How the Multi-Arch field and pkg:<arch> dependencies interact was discussed at DebConf15 in the "MultiArch BoF". dpkg and apt (among other tools like dose) had a different interpretation in certain scenarios which we resolved by agreeing on dpkg view – and this commit realizes this agreement in code. As was the case so far libapt sticks to the idea of trying to hide MultiArch as much as possible from individual frontends and instead translates it to good old SingleArch. There are certainly situations which can be improved in frontends if they know that MultiArch is upon them, but these are improvements – not necessary changes needed to unbreak a frontend. The implementation idea is simple: If we parse a dependency on foo:amd64 the dependency is formed on a package 'foo:amd64' of arch 'any'. This package is provided by package 'foo' of arch 'amd64', but not by 'foo' of arch 'i386'. Both of those foo packages provide each other through (assuming foo is M-A:foreign) to allow a dependency on 'foo' to be satisfied by either foo of amd64 or i386. Packages can also declare to provide 'foo:amd64' which is translated to providing 'foo:amd64:any' as well. This indirection over provides was chosen as the alternative would be to teach dependency resolvers how to deal with architecture specific dependencies – which violates the design idea of avoiding resolver changes, especially as architecture-specific dependencies are a cornercase with quite a few subtil rules. Handling it all over versioned provides as we already did for M-A in general seems much simpler as it just works for them. This switch to :any has actually a "surprising" benefit as well: Even frontends showing a package name via .Name() [which doesn't show the architecture] will display the "architecture" for dependencies in which it was explicitely requested, while we will not show the 'strange' :any arch in FullName(true) [= pretty-print] either. Before you had to specialcase these and by default you wouldn't get these details shown. The only identifiable disadvantage is that this complicates error reporting and handling. apt-get's ShowBroken has existing problems with virtual packages [it just shows the name without any reason], so that has to be worked on eventually. The other case is that detecting if a package is completely unknown or if it was at least referenced somewhere needs to acount for this "split" – not that it makes a practical difference which error is shown… but its one of the improvements possible.
* ignore AllowMem parameter in cache generationDavid Kalnischkies2015-08-271-11/+2
| | | | | | | | | | | | | | | | The parameter name suggests that it should forbid the building of the entire cache in memory, but this isn't how it was previously and as AllowMem is false by default it actually prevents previous usecases from working like being root and configuring apt to build no caches at all. This should be fixed at some point to actually work, but that is hard to pull off as it means switching the default and some callers (including apt itself) actually did call it explicitly with false in certain cases for no apparent reason (at least now where it is common to have enough memory to throw at every problem and even if not is a slow apt usally better than an apt erroring out). Closes: 796459
* Fix more instances of missing remapping handlingJulian Andres Klode2015-08-271-7/+22
| | | | | | | | | | | | After fixing Bug#796999, we noticed that there were some more instances of iterators which had no associated Dynamic object, causing them to not be updated when the cache was remapped. This happened in two places: In NewPackage() and in NewProvidesAllArch(). Gbp-Dch: ignore
* pkgcachegen: Account for remapping when parsing depends from NewPackageJulian Andres Klode2015-08-271-2/+6
| | | | | | | | | | | | | | | In both the Ver and Dep variables, we need to account for remapping, as otherwise we would still reference the old bug. Reproduction environment: * An i386 system with amd64 foreign architecture * A sources.list with deb http://snapshot.debian.org/archive/debian/20150826T102846Z/ unstable main deb http://snapshot.debian.org/archive/debian/20150826T102846Z/ experimental main Thanks: Jakub Wilk for the bug report and the backtraces Closes: #796999
* Cleanup includes after running iwyuMichael Vogt2015-08-171-2/+0
|
* Deprecate SPtrArray<T> and convert everyone to unique_ptr<T[]>Julian Andres Klode2015-08-131-4/+4
| | | | More standardization
* Mark SPtr as deprecated, and convert users to std::unique_ptrJulian Andres Klode2015-08-131-14/+14
| | | | Switch to std::unique_ptr, as this is safer than SPtr.
* Drop C++11 elements from headersJulian Andres Klode2015-08-111-1/+2
|
* drop obsolete explicit :none handling in pkgCacheGenDavid Kalnischkies2015-08-101-50/+0
| | | | | | | We archieve the same without the special handling now, so drop this code. Makes supporting this abdomination a little longer bearable as well. Git-Dch: Ignore
* eliminate dead file-provides code in cache generationDavid Kalnischkies2015-08-101-89/+2
| | | | | | | | | | The code was never active in production, it just sits there collecting dust and given that it is never tested probably doesn't even work anymore the way it was supposed to be (whatever that was exactly in the first place). So just remove it before I have to "fix" it again next time. Git-Dch: Ignore
* elimate duplicated code in pkgIndexFile subclassesDavid Kalnischkies2015-08-101-9/+9
| | | | | | | | Trade deduplication of code for a bunch of new virtuals, so it is actually visible how the different indexes behave cleaning up the interface at large in the process. Git-Dch: Ignore
* add volatile sources support in libapt-pkgDavid Kalnischkies2015-08-101-213/+188
| | | | | | | | | | | | | | | | | | | Sources are usually defined in sources.list (and co) and are pretty stable, but once in a while a frontend might want to add an additional "source" like a local .deb file to install this package (No support for 'real' sources being added this way as this is a multistep process). We had a hack in place to allow apt-get and apt to pull this of for a short while now, but other frontends are either left in the cold by this and/or the code for it looks dirty with FIXMEs plastering it and has on top of this also some problems (like including these 'volatile' sources in the srcpkgcache.bin file). So the biggest part in this commit is actually the rewrite of the cache generation as it is now potentially a three step process. The biggest problem with adding support now through is that this makes a bunch of previously mostly unusable by externs and therefore hidden classes public, so a bit of further tuneing on this now public API is in order…
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-01 08:32:26 +0000