zsync 32bit cannot download DVD

I retested with Ubuntu 9.04 amd64 and zsync 0.5. I can download DVD image in 64-bit Linux, bug is isoloated to 32-bit code. Thisis good news...

I tried to generate zsync file for DVD ISO image under Ubuntu 9.04 386 and Ubuntu 9.04 amd64 and I get the same file in both cases. This is great.

My conlusison is that only zsync client in 32-bit Linux has some bug that limits it use to some small ISO images (CD is OK, DVD is too big).

Thanks psl

I think this bug is might be related to bug 420931 which seems to get all the attention anyway.

A new package has been made available:

>Scott Kitterman wrote on 2009-10-02:
>Accepted into jaunty-proposed, the package will build now and be available in a few hours. Please test and >give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable >and use -proposed. Thank you in advance!

Maybe you find the time to verify again.

This bug is not related to #420931. I still cannot download DVD image at 32-bit system.

I have just replicated this bug successfully on intrepid:

$ uname -m
i686
$ apt-cache policy zsync
zsync:
  Installed: 0.5-1ubuntu3.8.10.1
  Candidate: 0.5-1ubuntu3.8.10.1
  Version table:
 *** 0.5-1ubuntu3.8.10.1 0
        500 http://at.archive.ubuntu.com intrepid-proposed/universe Packages
        100 /var/lib/dpkg/status
     0.5-1ubuntu3 0
        500 http://at.archive.ubuntu.com intrepid/universe Packages
$ zsync -i karmic-desktop-amd64.iso http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso.zsync
#################### 100.0% 771.4 kBps DONE

reading seed file karmic-desktop-amd64.iso: *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************Read karmic-desktop-amd64.iso. Target -26.2% complete. ********************************************************************************************************************
downloading from http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso:
-------------------- -51.5% 594.6 kBps ETA

downloading from http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso:
-------------------- -77.8% 548.7 kBps -115:-57 ETA

zsync works by writing an output file of final size at the beginning, and it does not seem to plan to get the whole DVD here:
$ls -n
total 2542652
-rwx------ 1 1001 1001 729829376 2009-10-15 10:30 karmic-desktop-amd64.iso
-rw------- 1 1001 1001 2083430400 2009-10-16 08:50 karmic-dvd-amd64.iso.part

it aborts:
$zsync -i karmic-desktop-amd64.iso http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso.zsync
#################### 100.0% 771.4 kBps DONE

reading seed file karmic-desktop-amd64.iso: *************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************Read karmic-desktop-amd64.iso. Target -26.2% complete. ********************************************************************************************************************
downloading from http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso:
-------------------- -51.5% 594.6 kBps ETA

downloading from http://cdimage.ubuntu.com/dvd/current/karmic-dvd-amd64.iso:
#################### 100.0% 683.3 kBps DONE 0 ETA

Aborting, download available in karmic-dvd-amd64.iso.part
verifying download...

zsync ...

I had this same problem when trying to update my Edubuntu DVD. I had to use rsync instead.

Wolfgang, thanks for testing. As I mentioned in bug 420931, I was going to prepare a backport of zsync 0.6 in karmic, but it turns out from testing that version that it doesn't handle filesizes larger than 2GB on 32bit platforms either.

Same on fully updated/upgraded karmic...

ubuntu@ubuntu:~$ apt-cache policy zsync
zsync:
  Installato: 0.6-1ubuntu2
  Candidato: 0.6-1ubuntu2
  Tabella versione:
 *** 0.6-1ubuntu2 0
        500 http://archive.ubuntu.com karmic/universe Packages
        100 /var/lib/dpkg/status

The dl of dvd stuck at same file size:

2147483647 2009-10-21 20:58 karmic-dvd-amd64.iso.part
2147483647 2009-10-22 07:45 karmic-dvd-i386.iso.part

Hope this helps

Side effects of this bug https://bugs.launchpad.net/ubuntu-cdimage/+bug/461973

Hth

I replicate the issue at my test server. I have a huge test file with random data at my local WWW server and I try to get test file (filesize 2202009600) with zsync running at 32-bit Ubuntu system. From http traffic, it looks that zsync fails to request more than 2147483647 bytes from the server. Is the bug in embedded www client in zsync? I think that 64-bit system will request wider range (Range: bytes=0-2202009599). Is this hint useful to fix this bug in zsync?

GET /zsync/random.iso HTTP/1.1

User-Agent: zsync/0.5

Host: server

Referer: http://server/zsync/random.iso.2k.zsync

Range: bytes=0-2147483647

Connection: close

HTTP/1.1 206 Partial Content

Date: Wed, 28 Oct 2009 02:11:50 GMT

Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.7 with Suhosin-Patch

Last-Modified: Fri, 14 Aug 2009 23:04:00 GMT

ETag: "ba37c204-83400000-47122114e1800"

Accept-Ranges: bytes

Content-Length: 2147483648

Content-Range: bytes 0-2147483647/2202009600

Connection: close

Content-Type: application/x-iso9660-image

..,C.0Li.S*..c...H..RyW.....'......3.........#.=<........M...Wf....O...[.-..3y..6.Y...I..2:..dk. f
..<.l.o.i...).J..+.y%....xt.nY.^...\w..Kw ...S...T..>....xY.Y|b.5..N....LQ..u.].g....>...Q.W.......
t...k.....>............$j...y'........m.y..x...gb...=$jv......H.......Q..{....I..Ij.fG...?.u....W_b......k..z.z...._..OX........-tl"..q"qGb<8.~%.S....l..~...r....Z........R..%.cV....}~.J.L....Z.u.$9..kSM..n..ELI..)~..BH...Ei........>...J..g=z.....{...X...
.......

zsync client exits after downloading 2147483647 bytes from www server, it doesn't try again and again. End user has to download the rest of file (or the whole file) in other way. :-(

I've prepared a fix for this issue in zsync and uploaded it to the unapproved queue. Subscribing motu-sru for approval. debdiff is attached.

Steve, let's version it to 0.6-1ubuntu2.1 by convention; otherwise consider this an ACK from motu-sru.

Hi John,

On Thu, Nov 05, 2009 at 03:21:02PM -0000, John Dong wrote:
> Steve, let's version it to 0.6-1ubuntu2.1 by convention; otherwise
> consider this an ACK from motu-sru.

The intent was that the package could be copied from karmic-proposed to
lucid once built, since there isn't likely to be any other upload of zsync
to lucid before then. In that case, do you still object to the version
number?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Safety wise I don't think there's much of an issue with that plan; as long as the archive magicians don't have an objection to it, I don't either :)

Accepted into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

This still aborts for me on i386 when initially downloading a large file, like so:

  $ zsync -k .ubuntu-9.04-dvd-i386.iso.zsync http://denisovich/iso/releases/ubuntu-9.04-dvd-i386.iso.zsync
  No relevent local data found - I will be downloading the whole file. If that's not what you want, CTRL-C out. You should specify the local file is the old version of the file to download with -i (you might have to decompress it with gzip -d first). Or perhaps you just have no data that helps download the file
  downloading from http://denisovich/iso/releases/ubuntu-9.04-dvd-i386.iso:
  #------------------- 5.5% 26166.4 kBps

  downloading from http://denisovich/iso/releases/ubuntu-9.04-dvd-i386.iso:
  #------------------- 5.6% 0.0 kBps aborted

  failed to retrieve from ubuntu-9.04-dvd-i386.iso
  Aborting, download available in ubuntu-9.04-dvd-i386.iso.part
  verifying download...ubuntu@karmic-desk-ia32:~$

I am able to subsequently re-run the command and have it successfully complete. When it does so, it downloads an amount just under 2^^32 bytes, specifically 1760689 bytes less:

  Read ubuntu-9.04-dvd-i386.iso.part. Target 5.6% complete.
  downloading from http://denisovich/iso/releases/ubuntu-9.04-dvd-i386.iso:
  #################### 100.0% 26897.1 kBps DONE

  verifying download...checksum matches OK
  used 256016384 local, fetched 4293206607

Testing with a different, larger dvd image behaved similarly, and aborted with the exact same amount remaining to download, based on the second run's output:

  [ubuntu-8.10-dvd-i386.iso]
  used 305467392 local, fetched 4293206607

So it seems likely that there's at least another unsigned 32bit int that needs to be expanded to 64bits somewhere.

That said, after the second run, sha256sums did match, and I saw no change in behavior on 64bits, soitcouldpossibly be considered an improvement.

I have completed successfully a dvd download:

verifying download...checksum matches OK
used 0 local, fetched 4135698776

On Fri, Nov 06, 2009 at 12:23:44PM -0000, Ara Pulido wrote:
> I have completed successfully a dvd download:
>
> verifying download...checksum matches OK
> used 0 local, fetched 4135698776

Ara, in the case above, the dvd you downloaded was less than 4GB in
size. Does it not abort for you if you download a dvd larger than 4GB
(4294967296 bytes)?

--
Steve Beattie
<email address hidden>
http://NxNW.org/~steve/

I tested the fix under Ubuntu 9.10 i386 and fix looks OK. I was able to download these DVDs (from LAN source):
ubuntu-9.04-dvd-i386.iso
ubuntu-9.10-dvd-amd64.iso
ubuntu-9.10-dvd-i386.iso

I tried full or partial download, all my test passed.

After that I tried to download a huge file, (isomix.bin, 5.5GB) and found no problem; I tried full and partial download.
isomix.bin file was created in this way:

$ cat u*server*iso u*alter*iso > isomix.bin
$ zsyncmake isomix.bin

$ ls -ltr isomix.iso*
-rw-r--r-- 1 root root 5599563776 2009-11-07 03:09 isomix.bin
-rw-r--r-- 1 root root 12303881 2009-11-07 03:22 isomix.bin.zsync

$ ls -l u*server*iso u*alter*iso
-rw-r--r-- 1 root root 730136576 2009-10-29 20:16 ubuntu-9.10-alternate-amd64.iso
-rw-r--r-- 1 root root 723068928 2009-10-29 20:09 ubuntu-9.10-alternate-i386.iso
-rw-r--r-- 1 root root 686305280 2009-10-29 20:02 ubuntu-9.10-server-amd64.iso
-rw-r--r-- 1 root root 671686656 2009-10-29 19:56 ubuntu-9.10-server-i386.iso
-rw-r--r-- 1 root root 718606336 2009-10-23 12:45 ubuntu-9.10-rc-alternate-amd64.iso
-rw-r--r-- 1 root root 711739392 2009-10-23 12:06 ubuntu-9.10-rc-alternate-i386.iso
-rw-r--r-- 1 root root 686188544 2009-10-23 10:53 ubuntu-9.10-rc-server-amd64.iso
-rw-r--r-- 1 root root 671832064 2009-10-23 10:04 ubuntu-9.10-rc-server-i386.iso

$ zsync -V
zsync v0.6 (compiled Nov 6 2009 05:52:12)
By Colin Phipps <email address hidden>
Published under the Artistic License v2, see the COPYING file for details.

I can replicate the problem (abort at 5.6%) reported by Steve Beattie during download of ubuntu-9.04-dvd-i386.iso at U9.10 i386.
I can download ubuntu-9.04-dvd-i386.iso at U9.04 amd64 without problem.

And I can download ubuntu-9.10-dvd-i386.iso and ubuntu-9.10-dvd-amd64.iso at U9.10 i386 without problem too.
Only ubuntu-9.04-dvd-i386.iso has that strange abort at 5.6% and it is 100% repeatable.

I can repeat the abort with my big file isomix.bin. I missed this before. Download of file is aborted at 23.2% (100% is 5599563776)

$ zsync http://server/zsync/isomix.bin.zsync
#################### 100.0% 6681.9 kBps DONE

No relevent local data found - I will be downloading the whole file. If that's not what you want, CTRL-C out. You should specify the local file is the old version of the file to download with -i (you might have to decompress it with gzip -d first). Or perhaps you just have no data that helps download the file
downloading from http://server/zsync/isomix.bin:
####---------------- 23.2% 11631.3 kBps

downloading from http://server/zsync/isomix.bin:
####---------------- 23.3% 0.0 kBps aborted

failed to retrieve from isomix.bin
Aborting, download available in isomix.bin.part
verifying download...

$ ls -ltr
total 1275272
-rw------- 1 root root 5599563776 2009-11-07 20:37 isomix.bin.part

The abort is 100% repeatable at U9.10 i386. I can finish file download after restart. U9.10 amd64 downloads the isomix.bin file without problem.

$ ls -ltr isomix.iso*
-rw-r--r-- 1 root root 5599563776 2009-11-07 03:09 isomix.bin
-rw-r--r-- 1 root root 12303881 2009-11-07 03:22 isomix.bin.zsync

Yes, this error is reproducible with any image > 4GB in size. Have reproduced here and am working on a fix.

Sorry to hear verification failed; Please resubscribe MOTU-SRU once a new patch is ready for review.

Take two, fixing the size_t limit on the amount of data that can be grabbed with an individual download run.

Incremental debdiff attached; uploaded to the queue, resubscribing motu-sru.

ACK again from motu-sru!

Accepted into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

This bug was fixed in the package zsync - 0.6-1ubuntu4

---------------
zsync (0.6-1ubuntu4) karmic-proposed; urgency=low

  * http.c: range_fetch->block_left must also be off_t, not size_t.

zsync (0.6-1ubuntu3) karmic-proposed; urgency=low

  * debian/rules: build with $(shell getconf LFS_CFLAGS), so that we can
    handle files over 2GB (such as DVD images) on 32-bit systems.
    LP: #412413.
  * http.c:
    - drop local delta, not needed if we use LFS_CFLAGS
    - fix ETA calculation on the progress bar
  * libzsync/zsync.c:
    - Use atoll() instead of atol(), to ensure support for > 32bit file
      sizes.
    - Cast to off_t before multiplying, otherwise we get random overflowage
      on the byte range.
 -- Steve Langasek <email address hidden> Sat, 07 Nov 2009 12:59:40 +0000

copied to lucid.

I can confirm that the latest version of zsync in karmic-proposed, 0.6-1ubuntu4, downloads images that are larger than 4GB correctly, both from an initial download and while updating an incomplete image. I tested on both i386 and amd64, and saw no sign of regression.

Thanks, Steve, for making this work!

This bug was fixed in the package zsync - 0.6-1ubuntu4

---------------
zsync (0.6-1ubuntu4) karmic-proposed; urgency=low

  * http.c: range_fetch->block_left must also be off_t, not size_t.

zsync (0.6-1ubuntu3) karmic-proposed; urgency=low

  * debian/rules: build with $(shell getconf LFS_CFLAGS), so that we can
    handle files over 2GB (such as DVD images) on 32-bit systems.
    LP: #412413.
  * http.c:
    - drop local delta, not needed if we use LFS_CFLAGS
    - fix ETA calculation on the progress bar
  * libzsync/zsync.c:
    - Use atoll() instead of atol(), to ensure support for > 32bit file
      sizes.
    - Cast to off_t before multiplying, otherwise we get random overflowage
      on the byte range.
 -- Steve Langasek <email address hidden> Sat, 07 Nov 2009 12:59:40 +0000

I am still facing the same bug in karmic and even Lucid 64-bit system.
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu lucid (development branch)
Release: 10.04
Codename: lucid

# uname -a
Linux ubuntu 2.6.32-8-generic #12-Ubuntu SMP Sat Dec 12 12:54:44 UTC 2009 x86_64 GNU/Linux

$ zsync http://cdimage.ubuntu.com/kubuntu/dvd/current/lucid-dvd-amd64.iso.zsync
#################### 100.0% 50.9 kBps DONE

reading seed file lucid-dvd-amd64.iso: ***********************************************************************************************************************************************************************************************************************************Read lucid-dvd-amd64.iso. Target 97.6% complete. ******************************************************************************************
downloading from http://cdimage.ubuntu.com/kubuntu/dvd/current/lucid-dvd-amd64.iso:
###################- 97.6% 5.8 kBps aborted 8 ETA

failed to retrieve from lucid-dvd-amd64.iso
Aborting, download available in lucid-dvd-amd64.iso.part
verifying download...

I want to change the status to NEW because it's not fixed here.

You were using version 0.6-1ubuntu4 of zsync from karmic-updates?

Yes, sure
$ apt-cache policy zsync
zsync:
  Installed: 0.6-1ubuntu4
  Candidate: 0.6-1ubuntu4

I cannot repeat the issue you reported. I downloaded 4 DVD of lucid without any problem at 32-bit system with U9.10. But I have one explanation for your trouble. These DVD images are snapshosts and are changed often.I guess your lucid-dvd-amd64.iso changed at server before you finished download. As a result, zsync metadata you get at the start of download process doesn't match iso image at server when you was finishing the downloadand zsync client give up with "abort" message.

II think that you can start your zsync process again; in this case, zsync client will use iso image you already downloaded as a seed and will update it with changes. Can you confirm this? Yust run again the same command in the directory with your unfinished download:

$ zsync http://cdimage.ubuntu.com/kubuntu/dvd/current/lucid-dvd-amd64.iso.zsync

Thanks psl, though I have rerun the zsync many times before I report the bug, trying it now, the problem is resolved. I guess it's the zsync file I tried is corrupt and the new file generated resolved that. Sorry for reopening a fixed bug.

Hi,
This bug also affect Ubuntu Hardy !