if insecure repo is allowed continue on all http errors

If a InRelease file fails to download with a non-404 error
we assumed there is some general problem with repository like
a webportal or your are blocked from access (wrong auth, Tor, …).

Turns out some server like S3 return 403 if a file doesn't exist.
Allowing this in general seems like a step backwards as 403 is a
reasonable response if auth failed, so failing here seems better
than letting those users run into problems.

What we can do is show our insecure warnings through and allow the
failures for insecure repos: If the repo is signed it is easy to add
an InRelease file and if not you are setup for trouble anyhow.

References: cbbf185c3c55effe47f218a07e7b1f324973a8a6

3 files changed, 39 insertions, 8 deletions

diff --git a/apt-pkg/acquire-item.cc b/apt-pkg/acquire-item.cc
index 1b0d6c592..b13bd203b 100644
--- a/apt-pkg/acquire-item.cc
+++ b/apt-pkg/acquire-item.cc
@@ -1784,18 +1784,27 @@ void pkgAcqMetaClearSig::Failed(string const &Message,pkgAcquire::MethodConfig c
 
    if (AuthPass == false)
    {
+      if (Status == StatTransientNetworkError)
+      {
+	 TransactionManager->AbortTransaction();
+	 return;
+      }
       auto const failreason = LookupTag(Message, "FailReason");
       auto const httperror = "HttpError";
-      if (Status == StatAuthError || Status == StatTransientNetworkError ||
-	    (strncmp(failreason.c_str(), httperror, strlen(httperror)) == 0 &&
-	     failreason != "HttpError404"))
+      if (Status == StatAuthError ||
+	  (strncmp(failreason.c_str(), httperror, strlen(httperror)) == 0 &&
+	   failreason != "HttpError404"))
       {
 	 // if we expected a ClearTextSignature (InRelease) but got a network
 	 // error or got a file, but it wasn't valid, we end up here (see VerifyDone).
 	 // As these is usually called by web-portals we do not try Release/Release.gpg
 	 // as this is gonna fail anyway and instead abort our try (LP#346386)
-	 TransactionManager->AbortTransaction();
-	 return;
+	 _error->PushToStack();
+	 _error->Error(_("Failed to fetch %s  %s"), Target.URI.c_str(), ErrorText.c_str());
+	 if (AllowInsecureRepositories(InsecureType::UNSIGNED, Target.Description, TransactionManager->MetaIndexParser, TransactionManager, this) == true)
+	    _error->RevertToStack();
+	 else
+	    return;
       }
 
       // Queue the 'old' InRelease file for removal if we try Release.gpg
diff --git a/test/integration/test-apt-update-nofallback b/test/integration/test-apt-update-nofallback
index 47adff33d..d7e30ba20 100755
--- a/test/integration/test-apt-update-nofallback
+++ b/test/integration/test-apt-update-nofallback
@@ -178,8 +178,10 @@ test_subvert_inrelease()
     # replace InRelease with something else
     mv "$APTARCHIVE/dists/unstable/Release" "$APTARCHIVE/dists/unstable/InRelease"
 
-    testfailuremsg "E: Failed to fetch file:${APTARCHIVE}/dists/unstable/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
-E: Some index files failed to download. They have been ignored, or old ones used instead." aptget update
+    testfailuremsg "E: Failed to fetch file://${APTARCHIVE}/dists/unstable/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
+E: The repository 'file:${APTARCHIVE} unstable InRelease' is no longer signed.
+N: Updating from such a repository can't be done securely, and is therefore disabled by default.
+N: See apt-secure(8) manpage for repository creation and user configuration details." aptget update
 
     # ensure we keep the repo
     testfileequal lists.before "$(listcurrentlistsdirectory)"
diff --git a/test/integration/test-ubuntu-bug-346386-apt-get-update-paywall b/test/integration/test-ubuntu-bug-346386-apt-get-update-paywall
index 46c7c5672..3571a9f25 100755
--- a/test/integration/test-ubuntu-bug-346386-apt-get-update-paywall
+++ b/test/integration/test-ubuntu-bug-346386-apt-get-update-paywall
@@ -78,4 +78,24 @@ testfailureequal "Err:1 http://localhost:${APTHTTPPORT} unstable InRelease
   511  Network Authentication Required
 Reading package lists...
 E: Failed to fetch http://localhost:${APTHTTPPORT}/dists/unstable/InRelease  511  Network Authentication Required
-E: Some index files failed to download. They have been ignored, or old ones used instead." apt update
+E: The repository 'http://localhost:${APTHTTPPORT} unstable InRelease' is not signed.
+N: Updating from such a repository can't be done securely, and is therefore disabled by default.
+N: See apt-secure(8) manpage for repository creation and user configuration details." aptget update
+
+# on S3 all files get a 403. If we accept unsigned, lets be liberal in non-existence acceptance
+webserverconfig 'aptwebserver::httpcode::404' '403 Forbidden'
+rm -rf rootdir/var/lib/apt/lists
+testfailureequal "Err:1 http://localhost:${APTHTTPPORT} unstable InRelease
+  403  Forbidden
+Reading package lists...
+E: Failed to fetch http://localhost:${APTHTTPPORT}/dists/unstable/InRelease  403  Forbidden
+E: The repository 'http://localhost:${APTHTTPPORT} unstable InRelease' is not signed.
+N: Updating from such a repository can't be done securely, and is therefore disabled by default.
+N: See apt-secure(8) manpage for repository creation and user configuration details." apt update
+
+sed -i 's#^deb\(-src\)\? #deb\1 [allow-insecure=yes] #' rootdir/etc/apt/sources.list.d/*
+testfailure apt update
+testequal "Ign:1 http://localhost:${APTHTTPPORT} unstable InRelease
+  403  Forbidden
+Ign:2 http://localhost:${APTHTTPPORT} unstable Release
+  403  Forbidden" head -n 4 rootdir/tmp/testfailure.output