FTP upload causes squid hang

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Is this bug fixed in upstream release 3.4.8? If so, then I will be fixing the development release of Ubuntu shortly.

If someone can prepare backports, please follow the steps at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure to have Trusty, Utopic and Vivid updated.

All the steps documented there need to be followed. In particular, I'm concerned that we have a working test case that can be followed by someone not familiar with the package for SRU verification purposes.

I my opinion this change is simple (but critical, any user can break the service) and the patch should be applied for squid in the main repository of Ubuntu LTS.

That's fine, but we need to take care to avoid regressing existing users not affected by this bug, and we need a test case to exercise the bug to verify that it is fixed before issuing the update. Please see my link above if you can assist with this.

Thank you for reporting this bug to Ubuntu. Utopic reached EOL on July 23, 2015.
See this document for currently supported Ubuntu releases: https://wiki.ubuntu.com/Releases

Vivid has also reached EOL, but I have provided a debdiff for Precise. Please test the package (once built) at the PPA: https://launchpad.net/~nacc/+archive/ubuntu/lp1423498.

Thanks,
Nish

@Stanislav, if you could provide a testcase still, that would be ideal.

Err, sorry, wires crossed on bugs, I'll update my debdiff for Trusty.

@Robie, do we also want to fix in Wily, then?

@Nish

If applicable and someone is affected then yes please. If nobody is reported as affected then I'd say no point though (so speak up now!) so maybe not? If we don't think anyone will benefit from an SRU then I'd say no point. We'll probably stop doing lower priority SRUs for Wily soon anyway as it's EOL in July.

I have updated the debdiff to the Trusty version and submitted it to the same PPA. @Stanislav, if you could test squid3 version 3.3.8-1ubuntu6.8 from the same, I'd appreciate it.

Yakkety and Xenial are up to 3.5.12-1ubuntu8 / 3.5.12-1ubuntu7 respectively.

@Stanislav, I've unassigned myself while we wait on testing results, but will watch the bug for updates.

I'm so sorry for not being here for a while. Now I have time again to work on this a little bit.

I've tested 3.3.8-1ubuntu6.8 and it is still crashes.

Here is how to test:

sudo apt-get install squid3 lftp
echo test > /tmp/file1
touch /tmp/file2
lftp -e "set ftp:proxy http://localhost:3128; set net:max-retries 1; open $USER@localhost; put /tmp/file1; exit"

sudo cat /var/log/squid3/cache.log

Oops, once again:

sudo apt-get install squid3 lftp
echo test > /tmp/file1
touch /tmp/file2
lftp -e "set ftp:proxy http://localhost:3128; set net:max-retries 1; open $USER@localhost; put /tmp/file1; exit"
lftp -e "set ftp:proxy http://localhost:3128; set net:max-retries 1; open $USER@localhost; put /tmp/file2; exit"
sudo cat /var/log/squid3/cache.log

And once again (missed proftpd):

sudo apt-get install squid3 lftp proftpd-basic
echo test > /tmp/file1
touch /tmp/file2
lftp -e "set ftp:proxy http://localhost:3128; set net:max-retries 1; open $USER@localhost; put /tmp/file1; exit"
lftp -e "set ftp:proxy http://localhost:3128; set net:max-retries 1; open $USER@localhost; put /tmp/file2; exit"
sudo cat /var/log/squid3/cache.log

Confirmed on trusty, happens with zero sized files. Checking the rest

Indeed it only happens with trusty at this point.

Updated test case:

sudo apt install squid3 proftpd-basic curl
echo -e "ubuntu\nubuntu" | sudo passwd ubuntu
touch /tmp/zero
ftp_proxy=http://localhost:3128/ curl --upload-file /tmp/zero ftp://ubuntu:ubuntu@localhost/

On affected versions, that will return an error from curl:
curl: (52) Empty reply from server

Triage note:

https://github.com/squid-cache/squid/commit/ccc927f9204159637bc77820fbb3f36521a54c86

commit ccc927f9204159637bc77820fbb3f36521a54c86
Author: Alex Rousskov <email address hidden>
Date: Sun Dec 29 07:56:02 2013 -0800

    Bug 3498: FTP PUT assertion Server.cc:246: "r->body_pipe != NULL"

diff --git a/src/ftp.cc b/src/ftp.cc
index f4b108080..71f3d3035 100644
--- a/src/ftp.cc
+++ b/src/ftp.cc
@@ -3051,6 +3051,13 @@ void FtpStateData::readStor()
     debugs(9, 3, HERE);

     if (code == 125 || (code == 150 && Comm::IsConnOpen(data.conn))) {
+ if (!originalRequest()->body_pipe) {
+ debugs(9, 3, "zero-size STOR?");
+ state = WRITING_DATA; // make ftpWriteTransferDone() responsible
+ dataComplete(); // XXX: keep in sync with doneSendingRequestBody()
+ return;
+ }
+
         if (!startRequestBodyFlow()) { // register to receive body data
             ftpFail(this);
             return;

Hello Stanislav, or anyone else affected,

Accepted squid3 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squid3/3.3.8-1ubuntu6.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Trusty verification

Confirming the problem with 3.3.8-1ubuntu6.9
  Version table:
 *** 3.3.8-1ubuntu6.9 0
        500 http://br.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages

ubuntu@trusty-squid-ftp-upload-1423498:~$ echo -e "ubuntu\nubuntu" | sudo passwd ubuntu
Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully

ubuntu@trusty-squid-ftp-upload-1423498:~$ truncate -s 0 /tmp/zero

ubuntu@trusty-squid-ftp-upload-1423498:~$ ftp_proxy=http://localhost:3128/ curl --upload-file /tmp/zero ftp://ubuntu:ubuntu@localhost/
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (52) Empty reply from server

ubuntu@trusty-squid-ftp-upload-1423498:~$ sudo grep assertion /var/log/squid3/cache.log
2017/10/27 12:21:38| assertion failed: Server.cc:244: "r->body_pipe != NULL"

Installing the package from proposed:
  Version table:
 *** 3.3.8-1ubuntu6.10 0
        500 http://br.archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages

ubuntu@trusty-squid-ftp-upload-1423498:~$ ftp_proxy=http://localhost:3128/ curl --upload-file /tmp/zero ftp://ubuntu:ubuntu@localhost/
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>FTP PUT Successful.</title>
<style type="text/css"><!--
 /*
 Stylesheet for Squid Error pages
(...)

Nothing of note in the cache log, where the assertion error was before.

And the zero file was uploaded:
ubuntu@trusty-squid-ftp-upload-1423498:~$ l zero
-rw-r--r-- 1 ubuntu ubuntu 0 Oct 27 12:25 zero

This bug was fixed in the package squid3 - 3.3.8-1ubuntu6.10

---------------
squid3 (3.3.8-1ubuntu6.10) trusty; urgency=medium

  * debian/patches/fix-assertion-ftp-put-empty-file.patch: Fix ftp
    assertion error when uploading empty file. Thanks to Alex Rousskov
    <email address hidden>. Closes LP: #1423498.

 -- Andreas Hasenack <email address hidden> Thu, 28 Sep 2017 12:23:01 -0400

The verification of the Stable Release Update for squid3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.