summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* tests: set source directory for gdbDavid Kalnischkies2016-08-171-1/+1
| | | | | | Helps interactive gdb calls find the source code. Gbp-Dch: Ignore
* default to Dir=/ in dpkg/status file finding magicDavid Kalnischkies2016-08-171-12/+10
| | | | | | | | | | | | | Seen in cme #833656 if Dir isn't set (yet) we end up later absoluting a path which was supposed to be absolute already, so if Dir is empty we assume it to be '/' instead. In practice this is a bug in the software using libapt, but for maxium compatibility lets explicitly set the default value here to be safe. Reported-By: Paul Wise <pabs@debian.org> Inspired-By: Brendan O'Dea <bod@debian.org> Fixes-Regression: 475f75506db48a7fa90711fce4ed129f6a14cc9a Shadows-Bug: #833656
* support compression and by-hash for .diff/Index filesDavid Kalnischkies2016-08-174-114/+151
| | | | | | | | | | | In af81ab9030229b4ce6cbe28f0f0831d4896fda01 by-hash got implemented as a special compression type for our usual index files like Packages. Missing in this scheme was the special .diff/Index index file containing the info about individual patches for this index file. Deriving from the index file class directly we inherent the compression handling infrastructure and in this way also by-hash nearly for free. Closes: #824926
* support getting only-uncompressed files via by-hashDavid Kalnischkies2016-08-171-0/+2
| | | | | | The URI we later want to modify to get the file via by-hash was unset in case a file was only available uncompressed (which is usually not the case) causing an acquire error.
* set the correct item FileSize in by-hash caseDavid Kalnischkies2016-08-171-4/+3
| | | | | | | | | | In af81ab9030229b4ce6cbe28f0f0831d4896fda01 we implement by-hash as a special compression type, which breaks this filesize setting as the code is looking for a foobar.by-hash file then. Dealing this slightly gets us the intended value. Note that this has no direct effect as this value will be set in other ways, too, and could only effect progress reporting. Gbp-Dch: Ignore
* retry without same redirection mirror on 404 errorsDavid Kalnischkies2016-08-172-5/+41
| | | | | | | | If 9b8034a9fd40b4d05075fda719e61f6eb4c45678 serves the Release files from a partial mirror we will end up getting 404 for some of the indexes. Instead of giving up, we will instead ignore our same redirection mirror constrain and ask the redirection service as a potential hashsum mismatch is better than keeping the certain 404 error.
* check internal redirections for loops, tooDavid Kalnischkies2016-08-172-0/+22
| | | | | | | | | | Now that we have the redirections loopchecker centrally in our items we can use it also to prevent internal redirections to loop caused by bugs as in a few instances we get into the business of rewriting the URI we will query by ourself as we predict we would see such a redirect anyway. Our code has no bugs of course, hence no practical difference. ;) Gbp-Dch: Ignore
* log with the failed item description, not with next tryDavid Kalnischkies2016-08-161-3/+4
| | | | | | | | | | The failure handling frequently changes URI & Description of the failed item to try a slightly different combination which might work, but the logging of the failure happens only afterwards as the same failure handling decides if this is a critical error or not so we need a backup here instead of potentially new content. A purely cosmetic issue, but can still be confusing for humans.
* don't try pipelining if server closes connectionsDavid Kalnischkies2016-08-161-1/+9
| | | | | | | | | | | | | | | | | | | | | | | | If a server closes a connection after sending us a file that tends to mean that its a type of server who always closes the connection – it is therefore relatively pointless to try pipelining with it even if it isn't a problem by itself: apt is just restarting the pipeline each time after it got served one file and the connection is closed. The problem starts if one or more proxies are between the server and apt and they disagree about how the connection should be as in the bugreporters case where the responses apt gets contain both Keep-Alive and Proxy-Connection headers (which apt both ignores) indicating a proxy is trying to keep a connection open while the response also contains "Connection: close" indicating the opposite which apt understands and respects as it is required to do. We avoid stepping into this abyss by not performing pipelining anymore if we got a respond with the indication to close connection if the response was otherwise a success – error messages are sent by some servers via this method as their pages tend to be created dynamically and hence their size isn't known a priori to them. Closes: #832113
* don't sent Range requests if we know its not acceptedDavid Kalnischkies2016-08-169-13/+37
| | | | | | If the server told us in a previous request that it isn't supporting Ranges with bytes via an Accept-Ranges header missing bytes, we don't try to formulate requests using Ranges.
* reorganize server-states resetting in http/httpsDavid Kalnischkies2016-08-165-17/+26
| | | | | | | | | | | | | We keep various information bits about the server around, some only effecting the currently handled file (like sizes) while others should be persistent (like pipeline detections). http used to reset all file-related manually, which is a bit silly if we already have a Reset() method – which does reset all through –, so extending it with a parameter for reuse and calling it from https too (as this was previously resetting by just creating a new state struct – it uses no value of the persistent state-keeping yet as it supports no pipelining). Gbp-Dch: Ignore
* CMake: Install bash completions via cmakeJulian Andres Klode2016-08-153-1/+6
| | | | | | | Having the completions installed only by the packaging was an oversight. Gbp-Dch: ignore
* Change anonscm.d.o links to /git/apt/apt.git and httpsJulian Andres Klode2016-08-132-3/+3
| | | | | | This also fixes Debian/apt#20, but is slightly more complete. I think /git also looks better than /cgit, so let's switch the Vcs entry in control over too.
* http(s): allow empty values for header fieldsDavid Kalnischkies2016-08-133-19/+20
| | | | | | | | | | | | It seems completely pointless from a server-POV to sent empty header fields, so most of them don't do it (simply proven by this limitation existing since day one) – but it is technically allowed by the RFC as the surounding whitespaces are optional and Github seems to like sending "X-Geo-Block-List:\r\n" since recently (bug reports in other http clients indicate July) at least sometimes as the reporter claims to have seen it on https only even through it can happen with both. Closes: 834048
* drop incorrect const attribute from DirectoryExistsDavid Kalnischkies2016-08-121-1/+1
| | | | | | | | | | | | | | | | | Since its existence in 2010 DirectoryExists was always marked with this attribute, but for no real reason. Arguably a check for the existence of the file is not modifying global state, so theoretically this shouldn't be a problem. It is wrong from a logical point of view through as between two calls the directory could be created so the promise we made to the compiler that it could remove the second call would be wrong, so API wise it is wrong. It's a bit mysterious that this is only observeable on ppc64el and can be fixed by reordering code ever so slightly, but in the end its more our fault for adding this attribute than the compilers fault for doing something silly based on the attribute. LP: 1473674
* fileutl: empty file support: Avoid fstat() on -1 fd and check resultJulian Andres Klode2016-08-121-2/+3
| | | | | When checking if a file is empty, we forget to check that fstat() actually worked.
* tests: don't do boundless string compares with data()David Kalnischkies2016-08-121-9/+11
| | | | Git-Dch: Ignore
* ensure a good clock() value for usage and testsDavid Kalnischkies2016-08-122-2/+10
| | | | | | | | | | | We use clock() as a very cheap way of getting a "random" value, but the manpage warns that this could return -1, so we should be dealing with this. Additionally, e.g. on hurd-i386 the value increases only slowly – to slow for our fast running tests for randomness hence producing the same range in both samples, so we introduce a simple busy-wait loop (as clock is counting processor time used by the program) in the test which delays the second sample just enough making our randomness a bit more predictable.
* don't perform int<float in progress bar drawingDavid Kalnischkies2016-08-122-13/+14
| | | | | | | | Comparing floating numbers is always fun and in this instance a 9 < 9.0 is "somehow" true on hurd-i386 letting the tests fail by reporting that too much progress achieved. A bit mysterious, but with some rework we can use code which avoids dealing with the floats in this way entirely and make our testcases happy.
* ctest: show test output in case of failuresDavid Kalnischkies2016-08-123-3/+7
| | | | | | | | | | ctest as run by cmake by default does not show the output of the tests even if the tests failed. In terms of our tests it could be handy to set it always, but unfortunately it seems like cmake doesn't allow it if the internet is to be believed, so lets enable it at least while building packages and on travis. Gbp-Dch: Ignore
* CMake: Exclude .md5 and .map doxygen files from installJulian Andres Klode2016-08-112-3/+4
| | | | | | This is much better than removing them in debian/rules. Gbp-Dch: ignore
* CMake: Use COPYONLY instead of @ONLYJulian Andres Klode2016-08-112-2/+2
| | | | | | I don't know what happened back in 2009 when I wrote this, but it seems I used the wrong option. These files should not have any variable substitution done to them.
* CMake: Mark Doxygen as required and use DOXYGEN_EXECUTABLEJulian Andres Klode2016-08-111-2/+2
| | | | | | Seems like I missed that when adding doxygen support. Gbp-Dch: ignore
* debian/NEWS: Get rid of 1.3~pre3+cmake1 entryJulian Andres Klode2016-08-111-14/+0
| | | | | | This was only needed temporarily Thanks: Axel Beckert for reporting
* Release 1.3~rc11.3_rc1Julian Andres Klode2016-08-1158-85292/+85630
| | | | | | | | This commit looks heavy. Most of that comes from the fact that the ordering of files in the translations changed with the switch to CMake. I could have gone the extra mile to figure out the original ordering and replicate it, but I have chosen to re-order everything by file and line number, as that's easier.
* doc/po: Adjust translations for programmatic typo fixJulian Andres Klode2016-08-1110-10/+10
| | | | | | This enhances commit b9e6db821a6ddbc5bf6a90c80c296d4e91283a63. Gbp-Dch: ignore
* tests: copy 01autoremove from the right placeDavid Kalnischkies2016-08-111-2/+2
| | | | | | | | | | | | With cmake using BUILDDIRECTORY at this place is not only as wrong as it was before, but it might not even work always copying the system provided one which might or might not be current and hence fails tests needing it to be current like ./test-apt-move-and-forget-manual-sections We don't want to always use the one from the source directory through either like in autopkgtests. Gbp-Dch: Ignore
* Merge branch 'feature/apt-dpkg-comm'David Kalnischkies2016-08-1128-246/+703
|\
| * disable explicit configuration of all packages at the endDavid Kalnischkies2016-08-1016-164/+105
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With b4450f1dd6bca537e60406b2383ab154a3e1485f we dropped what we calculated here later on and now that we don't need it in the meantime either we can just skip the busy work by default and expect dpkg to do the right thing dropping also our little "last explicit configures" removal trick introduced in b4450f1dd6bca537e60406b2383ab154a3e1485f. This enables the last of a bunch of previously experimental options, some of them existing still, but are very special and hence not really worth documenting anymore (especially as it would need to be rewritten now entirely) which is why the documentation is nearly completely dropped. The order of configuration stanzas in the simulation code changes slightly as it isn't concerning itself with finding the 'right' order, but any order is valid anyhow as long as the entire set happens in the same call.
| * simulate all package manager actions explicitlyDavid Kalnischkies2016-08-106-31/+111
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a planner lets actions to be figured out by dpkg in pending calls these actions aren't mentioned in a simulation. While that might be a good thing for debugging, it would be a change in behavior and especially if a planner avoids explicit removals could be confusing for users. As such we perform the same 'trick' as in the dpkg implementation by performing explicitly what would be done by the pending calls. To save us some work and avoid desyncs we perform a layer violation by using deb/ code in the generic simulation – and further we perform ugly dynamic_cast to avoid breaking the ABI for nothing; aptitude is the only other user of the simulation class according to codesearch.d.n and for that our little trick works. It just isn't working if you happen to extend pkgSimulate or otherwise manage to call the protected Go methods directly – which isn't very realistic/practical.
| * try to avoid removal of crossgraded packagesDavid Kalnischkies2016-08-102-26/+189
| | | | | | | | | | | | | | | | The user has to approve the removal of a crossgraded package as it might be needed to remove it (temporarily) in the process, but in most cases we can happily avoid it and let dpkg unpack over it skipping the remove. This has some effects on progress reporting and how deal with selections through which makes this a tiny bit complicated.
| * ensure all removes are reported to hook scriptsDavid Kalnischkies2016-08-101-0/+13
| | | | | | | | Same reason and implementation as for configure.
| * ensure all configures are reported to hook scriptsDavid Kalnischkies2016-08-101-0/+17
| | | | | | | | | | | | | | A planner might not explicitly configure all packages, but we need to know all packages which will be configured for progress reporting and to tell the hook scripts about them as they rely on this for their own functionality.
| * don't purge directly, but remove and do purge at the endDavid Kalnischkies2016-08-102-63/+88
| | | | | | | | | | | | | | | | | | | | If we want a package to be purged from the system tell dpkg in the ordering (if it has to touch it explicitly) to remove it and cover the purging of the config files at the end with a --purge --pending call. That should help packages move conffiles around between packages correctly even if the user is purging packages directly in big actions like dist-upgrades involving many packages.
| * call dpkg with --no-triggers by defaultDavid Kalnischkies2016-08-101-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | Implemented a long while ago now with relatively good progress reporting involving triggers is a good time to try delaying the execution of triggers across dpkg invocations finally by default. Note: The bugreport talks also about 'smarter' configuration which is a much bigger part and approached from multiple directions, but doesn't really involve triggers per-se so considering it decoupled should help in getting it done… Closes: #626599
| * select remove/purge packages early on for dpkgDavid Kalnischkies2016-08-104-16/+88
| | | | | | | | | | | | | | Telling dpkg early on that we are going to remove these packages later helps it with auto-deconfiguration decisions and its another area where a planner can ignore the nitty gritty details and let dpkg decide the course of action if there are no special requirements.
| * save and restore selection states before/after calling dpkgDavid Kalnischkies2016-08-103-8/+47
| | | | | | | | | | | | | | | | | | | | | | | | dpkg decides certain things on its own based on selections and especially if we want to call --pending on purge/remove actions, we need to ensure a clean slate or otherwise we surprise the user by removing packages we weren't allowed to remove by the user in this run (the selection might be an overarching plan for the not-yet "future"). Ideally dpkg would have some kind of temporal selection interface for this case, but it hasn't, so we make it temporal with the risk of loosing state if we don't manage to restore them.
| * use dpkg --unpack --recursive to avoid long cmdlinesDavid Kalnischkies2016-08-102-8/+96
| | | | | | | | | | | | | | | | | | | | Having long commandlines split into two is a huge problem if it happens and additionally if we want to introduce planners which perform less micromanagment its a good idea to leave the details for dpkg to decide. In practice this doesn't work yet unconditionally as a bug is hiding in the ordering code of dpkg, but it works if apt imposes its ordering so this commit allows for now at least to solve the first problem.
| * pass --force-remove-essential to dpkg only if neededDavid Kalnischkies2016-08-103-2/+20
| | | | | | | | | | | | APT (usually) knows which package is essential or not, so we can avoid passing this force flag to dpkg unconditionally if the user hasn't chosen a non-default essential handling obscuring the information.
* | Merge branch 'feature/methods'David Kalnischkies2016-08-1131-439/+1028
|\ \
| * | http: auto-configure for local Tor proxy if called as 'tor'David Kalnischkies2016-08-114-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With apts http transport supporting socks5h proxies and all the work in terms of configuration of methods based on the name it is called with it becomes surprisingly easy to implement Tor support equally (and perhaps even a bit exceeding) what is available currently in apt-transport-tor. How this will turn out to be handled packaging wise we will see in https://lists.debian.org/deity/2016/08/msg00012.html , but until this is resolved we can add the needed support without actively enabling it for now, so that this can be tested better.
| * | block direct connections to .onion domains (RFC7687)David Kalnischkies2016-08-112-1/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Doing a direct connect to an .onion address (if you don't happen to use it as a local domain, which you shouldn't) is bound to fail and does leak the information that you do use Tor and which hidden service you wanted to connect to to a DNS server. Worse, if the DNS is poisoned and actually resolves tricking a user into believing the setup would work correctly… This does block also the usage of wrappers like torsocks with apt, but with native support available and advertised in the error message this shouldn't really be an issue. Inspired-by: https://bugzilla.mozilla.org/show_bug.cgi?id=1228457
| * | allow methods to be disabled and redirected via configDavid Kalnischkies2016-08-105-44/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To prevent accidents like adding http-sources while using tor+http it can make sense to allow disabling methods. It might even make sense to allow "redirections" and adding "symlinked" methods via configuration. This could e.g. allow using different options for certain sources by adding and configuring a "virtual" new method which picks up the config based on the name it was called with like e.g. http does if called as tor+http.
| * | implement socks5h proxy support for http methodDavid Kalnischkies2016-08-105-23/+318
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Socks support is a requested feature in sofar that the internet is actually believing Acquire::socks::Proxy would exist. It doesn't and this commit isn't adding it as that isn't how our configuration works, but it allows Acquire::http::Proxy="socks5h://…". The HTTPS method was changed already to support socks proxies (all versions) via curl. This commit implements only SOCKS5 (RFC1928) with no auth or pass&user auth (RFC1929), but not GSSAPI which is required by the RFC. The 'h' in the protocol name further indicates that DNS resolution is delegated to the socks proxy rather than performed locally. The implementation works and was tested with Tor as socks proxy for which implementing socks5h only can actually be considered a feature. Closes: 744934
| * | implement generic config fallback for methodsDavid Kalnischkies2016-08-1018-205/+303
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The https method implemented for a long while now a hardcoded fallback to the same options in http, which, while it works, is rather inflexible if we want to allow the methods to use another name to change their behavior slightly, like apt-transport-tor does to https – most of the diff being s#https#tor#g which then fails to do the full circle fallthrough tor -> https -> http for https sources. With this config infrastructure this could be implemented now.
| * | use the same redirection handling for http and httpsDavid Kalnischkies2016-08-107-101/+102
| | | | | | | | | | | | | | | | | | | | | | | | cURL which backs our https implementation can handle redirects on its own, but by dealing with them on our own we gain finer control over which redirections will be performed (we don't like https → http) and by whom so that redirections to other hosts correctly spawn a new https method dealing with these instead of letting the current one deal with it.
| * | detect redirection loops in acquire instead of workersDavid Kalnischkies2016-08-1010-65/+129
| | | | | | | | | | | | | | | | | | | | | Having the detection handled in specific (http) workers means that a redirection loop over different hostnames isn't detected. Its also not a good idea have this implement in each method independently even if it would work
| * | fail on unsupported http/https proxy settingsDavid Kalnischkies2016-08-104-6/+45
| | | | | | | | | | | | Closes: #623443
| * | suggest transport-packages based on established nameschemeDavid Kalnischkies2016-08-101-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | apt-transports not shipped in apt directly are usually named apt-transport-% with % being what is in the name of the transport. tor additional introduced aliases via %+something, which isn't a bad idea, so be strip the +something part from the method name before suggesting the installation of an apt-transport-% package. This avoids us the maintainance of a list of existing transports creating a two class system of known and unknown transports which would be quite arbitrary and is unfriendly to backports.
| * | support all socks-proxy known to curl in https methodDavid Kalnischkies2016-08-101-1/+12
| |/