summaryrefslogtreecommitdiff
path: root/methods/http.cc
Commit message (Collapse)AuthorAgeFilesLines
* http: don't hang on redirect with length + connection closeDavid Kalnischkies2016-06-151-4/+4
| | | | | | | | Most servers who close the connection do not send a content-length as this is redundant information usually, but some might and while testing with our server and with 'aptwebserver::response-header::Connection' set to 'close' I noticed that http hangs after a redirect in such cases, so if we have the information, just use it instead of discarding it.
* use std::locale::global instead of setlocaleDavid Kalnischkies2016-05-281-1/+0
| | | | | | We use a wild mixture of C and C++ ways of generating output, so having a consistent world-view in both styles sounds like a good idea and should help in preventing regressions.
* prevent C++ locale number formatting in text APIsDavid Kalnischkies2016-05-271-2/+2
| | | | | | | | | | | Setting the C++ locale via std::locale::global(std::locale("")); which would otherwise default to the default C locale (aka: unaffected by setlocale) effects the formatting of numeric types in IO streams, which for output for humans is perfectly sensible, but breaks our many text interfaces used and parsed by us and others without expecting the numbers to be formatted. Closes: #825396
* fix two memory leaks reported by gccDavid Kalnischkies2015-09-141-2/+2
| | | | | Reported-By: gcc -fsanitize=address -fno-sanitize=vptr Git-Dch: Ignore
* Merge branch 'debian/sid' into debian/experimentalMichael Vogt2015-05-221-1/+1
|\ | | | | | | | | | | | | | | | | Conflicts: apt-pkg/pkgcache.h debian/changelog methods/https.cc methods/server.cc test/integration/test-apt-download-progress
| * Fix endless loop in apt-get update that can cause disk fillupMichael Vogt2015-05-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The apt http code parses Content-Length and Content-Range. For both requests the variable "Size" is used and the semantic for this Size is the total file size. However Content-Length is not the entire file size for partital file requests. For servers that send the Content-Range header first and then the Content-Length header this can lead to globbing of Size so that its less than the real file size. This may lead to a subsequent passing of a negative number into the CircleBuf which leads to a endless loop that writes data. Thanks to Anton Blanchard for the analysis and initial patch. LP: #1445239
| * dispose http(s) 416 error page as non-contentDavid Kalnischkies2014-12-221-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. (The tests and their binary helpers had to be slightly modified to apply, but the patch to fix the issue itself is unchanged.) Closes: 768797
* | calculate hashes while downloading in httpsDavid Kalnischkies2015-04-191-5/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | We do this in HTTP already to give the CPU some exercise while the disk is heavily spinning (or flashing?) to store the data avoiding the need to reread the entire file again later on to calculate the hashes – which happens outside of the eyes of progress reporting, so you might ended up with a bunch of https workers 'stuck' at 100% while they were busy calculating hashes. This is a bummer for everyone using apt as a connection speedtest as the https method works slower now (not really, it just isn't reporting done too early anymore).
* | calculate only expected hashes in methodsDavid Kalnischkies2015-04-191-7/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Methods get told which hashes are expected by the acquire system, which means we can use this list to restrict what we calculate in the methods as any extra we are calculating is wasted effort as we can't compare it with anything anyway. Adding support for a new hash algorithm is therefore 'free' now and if a algorithm is no longer provided in a repository for a file, we automatically stop calculating it. In practice this results in a speed-up in Debian as we don't have SHA512 here (so far), so we practically stop calculating it.
* | handle servers closing encoded connections correctlyDavid Kalnischkies2015-04-191-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Servers who advertise that they close the connection get the 'Closes' encoding flag, but this conflicts with servers who response with a transfer-encoding (e.g. encoding) as it is saved in the same flag. We have a better flag for the keep-alive (or not) of the connection anyway, so we check this instead of the encoding. This is in practice not much of a problem as real servers we talk to are HTTP1.1 servers (with keep-alive) and there isn't much point in doing chunked encoding if you are going to close anyway, but our simple testserver stumbles over this if pressed and its a bit cleaner, too. Git-Dch: Ignore
* | derive more of https from http methodDavid Kalnischkies2015-03-161-2/+0
| | | | | | | | | | | | | | | | | | Bug #778375 uncovered that https wasn't properly integrated in the class family tree of http as it was supposed to be leading to a NULL pointer dereference. Fixing this 'properly' was deemed to much diff for practically no gain that late in the release, so commit 0c2dc43d4fe1d026650b5e2920a021557f9534a6 just fixed the synptom, while this commit here is fixing the cause plus adding a test.
* | dispose http(s) 416 error page as non-contentDavid Kalnischkies2014-12-091-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Real webservers (like apache) actually send an error page with a 416 response, but our client didn't expect it leaving the page on the socket to be parsed as response for the next request (http) or as file content (https), which isn't what we want at all… Symptom is a "Bad header line" as html usually doesn't parse that well to an http-header. This manifests itself e.g. if we have a complete file (or larger) in partial/ which isn't discarded by If-Range as the server doesn't support it (or it is just newer, think: mirror rotation). It is a sort-of regression of 78c72d0ce22e00b194251445aae306df357d5c1a, which removed the filesize - 1 trick, but this had its own problems… To properly test this our webserver gains the ability to reply with transfer-encoding: chunked as most real webservers will use it to send the dynamically generated error pages. Closes: 768797
* | Fix backward compatiblity of the new pkgAcquireMethod::DropPrivsOrDie()Michael Vogt2014-10-131-0/+2
| | | | | | | | | | | | | | | | Do not drop privileges in the methods when using a older version of libapt that does not support the chown magic in partial/ yet. To do this DropPrivileges() now will ignore a empty Apt::Sandbox::User. Cleanup all hardcoded _apt along the way.
* | Send "Fail-Reason: MaximumSizeExceeded" from the methodMichael Vogt2014-10-071-0/+1
| | | | | | | | | | Communicate the fail reason from the methods to the parent and Rename() failed files.
* | make expected-size a maximum-size check as this is what we want at this pointMichael Vogt2014-10-071-2/+2
| |
* | make http size check workMichael Vogt2014-10-061-63/+7
|\|
| * Make Proxy-Auto-Detect check for each hostMichael Vogt2014-09-021-60/+2
| | | | | | | | | | | | | | | | | | When doing Acquire::http{,s}::Proxy-Auto-Detect, run the auto-detect command for each host instead of only once. This should make using "proxy" from libproxy-tools feasible which can then be used for PAC style or other proxy configurations. Closes: #759264
| * methods/http.cc: use Req.str() in debug outputMichael Vogt2014-06-241-1/+1
| |
* | Pass ExpectedSize to tthe backend methodMichael Vogt2014-08-261-1/+9
|/ | | | | This ensures that we can stop downloading if the server send too much data by accident (or by a malicious attempt)
* build http request in a stringstreamDavid Kalnischkies2014-04-261-57/+31
| | | | | beside reducing code a bit, it avoids oddball problems while building the string and doesn't trigger static analyse warnings.
* follow method attribute suggestions by gccDavid Kalnischkies2014-03-131-2/+2
| | | | | Git-Dch: Ignore Reported-By: gcc -Wsuggest-attribute={pure,const,noreturn}
* cleanup headers and especially #includes everywhereDavid Kalnischkies2014-03-131-8/+5
| | | | | | | | Beside being a bit cleaner it hopefully also resolves oddball problems I have with high levels of parallel jobs. Git-Dch: Ignore Reported-By: iwyu (include-what-you-use)
* StartPos is always positive for http/httpsDavid Kalnischkies2014-03-131-7/+2
| | | | | | | | | server.cc: In member function ‘bool ServerState::HeaderLine(std::string)’: server.cc:198:72: warning: format ‘%llu’ expects argument of type ‘long long unsigned int*’, but argument 3 has type ‘long long int*’ [-Wformat=] else if (sscanf(Val.c_str(),"bytes %llu-%*u/%llu",&StartPos,&Size) != 2) Git-Dch: Ignore Reported-By: gcc -Wpedantic
* warning: extra ‘;’ [-Wpedantic]David Kalnischkies2014-03-131-3/+3
| | | | | Git-Dch: Ignore Reported-By: gcc -Wpedantic
* Fix typos in documentation (codespell)Michael Vogt2014-02-221-2/+2
|
* fix various style/performance warnings in rredDavid Kalnischkies2014-01-301-1/+0
| | | | | Reported-By: cppcheck Git-Dch: Ignore
* correct some style/performance/warnings from cppcheckDavid Kalnischkies2014-01-161-3/+1
| | | | | | | | The most "visible" change is from utime to utimensat/futimens as the first one isn't part of POSIX anymore. Reported-By: cppcheck Git-Dch: Ignore
* add Acquire::http::Proxy-Auto-Detect to the apt.conf.5 manpage (closes: 726597)Michael Vogt2013-10-221-1/+5
|
* handle complete responses to https range requestsDavid Kalnischkies2013-10-011-0/+5
| | | | | | | | | | | | | Servers might respond with a complete file either because they don't support Ranges at all or the If-Range condition isn't statisfied, so we have to parse the headers curl gets ourself to seek or truncate the file we have so far. This also finially adds the testcase testing a bunch of partial situations for both, http and https - which is now all green. Closes: 617643, 667699 LP: 1157943
* refactor http client implementationDavid Kalnischkies2013-10-011-790/+201
| | | | | | | | | | | | | No effective behavior change, just shuffling big junks of code between methods and classes to split them into those strongly related to our client implementation and those implementing HTTP. The idea is to get HTTPS to a point in which most of the implementation can be shared even though the client implementations itself is completely different. This isn't anywhere near yet though, but it should beenough to reuse at least a few lines from http in https now. Git-Dch: Ignore
* replace "filesize - 1" trick in http with proper 416 handlingDavid Kalnischkies2013-10-011-6/+21
| | | | | | | | Our http client requests the "filesize - 1" for the small edgecase of handling a file which was completely downloaded, but not yet moved to the correct place as we get 416 errors in that case, but as we can handle 416 returns now we just special-case the situation of requesting the exact filesize and handle it as a 200 without content instead.
* retry without partial data after a 416 responseDavid Kalnischkies2013-10-011-2/+17
| | | | | | | | | | | | | | | If we get a 416 from the server it means the Range we asked for is above the real filesize of the file on the server. Mostly this happens if the server isn't supporting If-Range, but regardless of how we end up with the partial data, the data is invalid so we discard it and retry with a fresh plate and hope for the best. Old behavior was to consider 416 an error and retry with a different compression until we ran out of compression and requested the uncompressed file (which doesn't exist on most mirrors) with an accept line which server answered with "406 Not Acceptable". Closes: 710924
* Merge remote-tracking branch 'mvo/bugfix/coverity' into debian/sidMichael Vogt2013-07-281-1/+1
|\
| * fix off-by-one error in HttpMethod::​AutoDetectProxy()Michael Vogt2013-07-251-1/+1
| |
* | request absolute URIs from proxies again (0.9.9.3 regession)David Kalnischkies2013-07-261-4/+16
| | | | | | | | | | | | | | | | Commit 2b9c9b7f28b18f6ae3e422020e8934872b06c9f3 not only removes keep-alive, but also changes the request URI send to proxies which are required to be absolute URIs rather than the usual absolute paths. Closes: 717891
* | Do not send a connection: keep-alive, at allRaphael Geissert2013-07-141-21/+8
|/
* merged patch from Daniel Hartwig to fix URI and proxy releated issuesMichael Vogt2013-05-081-7/+7
|
* quote plus in filenames to work around a bug in the S3 serverMichael Vogt2013-01-241-1/+5
| | | (LP: #1003633)
* * methods/http.cc:Michael Vogt2013-01-111-1/+1
| | | | | | - quote spaces in filenames to ensure as the http method is also (potentially) used for non deb,dsc content that may contain spaces, thanks to Daniel Hartwig and Thomas Bushnell (LP: #1086997)
* add spaces around PACKAGE_VERSION to fix FTBFS with -std=c++11David Kalnischkies2012-05-171-1/+1
|
* * methods/http.cc:David Kalnischkies2012-05-151-1/+1
| | | | | - after many years of pointless discussions disable http/1.1 pipelining by default as many webservers and proxies seem to be unable to conform to specification must's (rfc2616 section 8.1.2.2) (LP: #996151)
* * apt-pkg/acquire*.cc:Raphael Geissert2012-05-141-1/+4
| | | | | | | - handle redirections in the worker with the right method instead of in the method the redirection occured in (Closes: #668111) * methods/http.cc: - forbid redirects to change protocol
* newer gcc versions seems to have no problem with that, but while workingDavid Kalnischkies2012-04-171-1/+1
| | | | with g++-4.1 it complains about this so lets be extra clear
* the previously used VERSION didn't work everywhere so we are switchingDavid Kalnischkies2012-03-221-1/+1
| | | | to the more standard PACKAGE_VERSION and make it work in every file
* fix "(performance) Possible inefficient checking for 'R' emptiness."David Kalnischkies2012-03-051-1/+1
|
* fix "(style) Checking if unsigned variable 'Minor' is less than zero."David Kalnischkies2012-03-051-1/+1
|
* ensure that (s)scanf doesn't parse a too long Code now that a previousDavid Kalnischkies2012-03-051-5/+7
| | | | commit lifted the Line-length limit
* remove the arbitrary MAXLEN limit for response lines (Closes: #658346)David Kalnischkies2012-02-111-4/+0
|
* * methods/http{s,}.cc:David Kalnischkies2012-01-301-1/+13
| | | | | - if a file without an extension is requested send an 'Accept: text/*' header to avoid that the server chooses unsupported compressed files in a content-negotation attempt (Closes: #657560)
* try to avoid direct usage of .Fd() if possible and do read()s and coDavid Kalnischkies2011-12-171-17/+7
| | | | on the FileFd instead
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-01 11:30:05 +0000