Ubuntu
pax package

Unable to create files larger than 2GB

Bug #56764 reported by andrei.tanasescu
6
Affects Status Importance Assigned to Milestone
pax (Debian)
Fix Released
Unknown
pax (Ubuntu)
Invalid
Wishlist
Unassigned

Bug Description

Binary package hint: pax

Try to create any archive with the following command, which exceeds 2GB on the default ext3 filesystem.

pax -x cpio -w -f <output> <input>

The command will stop with :

pax: End of archive volume 1 reached

ATTENTION! pax archive volume change required.
Ready for archive volume: 2
Input archive name or "." to quit pax.
Archive name > .
Quitting pax!
pax: Unable to write cpio header for (null)

See original description
Revision history for this message
In , Bdale Garbee (bdale) wrote : Re: Bug#317466: pax cannot access files >2gb

On Fri, 2005-07-08 at 22:02 +0200, Marc Lehmann wrote:

> pax cannot cope with large files:

The problem is that upstream for pax is OpenBSD, and the last time I
tried pulling a newer version it had dependencies on some libraries that
weren't present in Debian that I wasn't motivated to deal with. There
was a 'paxutils' package being worked on by FSF folks for a while that I
assumed would eventually be the better choice for Debian, but the last
update to that was in 1999, I think.

I think this translates roughly to "I'll be happy to accept a patch, but
don't hold your breath waiting for a solution"...

Bdale

Revision history for this message
In , Marc Lehmann (schmorp) wrote :

On Sun, Jul 10, 2005 at 02:30:58PM +0300, Bdale Garbee <email address hidden> wrote:
> On Fri, 2005-07-08 at 22:02 +0200, Marc Lehmann wrote:
>
> > pax cannot cope with large files:
>
> The problem is that upstream for pax is OpenBSD, and the last time I
> tried pulling a newer version it had dependencies on some libraries that
> weren't present in Debian that I wasn't motivated to deal with. There
> was a 'paxutils' package being worked on by FSF folks for a while that I
> assumed would eventually be the better choice for Debian, but the last
> update to that was in 1999, I think.
>
> I think this translates roughly to "I'll be happy to accept a patch, but
> don't hold your breath waiting for a solution"...

Well, how about just reocmpiling it with largefile support and seeing if
it breaks down with large files (e.g. due to on-disk changes).

In any case, I'll have a look into the issue then (but I am busy, as I
guess you are, too, so don't expect anything :)

Thanks for caring, though, and your good work in bringing the openbsd pax
to debian :)

--
                The choice of a
      -----==- _GNU_
      ----==-- _ generation Marc Lehmann
      ---==---(_)__ __ ____ __ <email address hidden>
      --==---/ / _ \/ // /\ \/ / http://schmorp.de/
      -=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE

Revision history for this message
andrei.tanasescu (ftanasescu) wrote :

Binary package hint: pax

Try to create any archive with the following command, which exceeds 2GB on the default ext3 filesystem.

pax -x cpio -w -f <output> <input>

The command will stop with :

pax: End of archive volume 1 reached

ATTENTION! pax archive volume change required.
Ready for archive volume: 2
Input archive name or "." to quit pax.
Archive name > .
Quitting pax!
pax: Unable to write cpio header for (null)

Revision history for this message
Mark Reitblatt (mark-reitblatt) wrote :

For reasons unknown to myself, pax does not support archives > 2GB in size. Something about standards compliance, though I don't know what standards (POSIX?).

Changed in pax:
importance: Untriaged → Wishlist
Revision history for this message
andrei.tanasescu (ftanasescu) wrote :

That is incorrect. Pax supports archives larger than 2gb on both my
Mac and openbsd system.

I ran a search in the debian archive for the pax package and what
happened is the following :

- someone took the pax package from openbsd
- when they ported the package, they wanted to save time and omitted a
certain library
- pax then was patched by the debian team with the changes never
making it into the main pax tree at openbsd.org
- as a result of the patching and/or incomplete porting and lack of
interest from the pax team, pax on debian and now ubuntu, does not
support archives larger than 2gb.

What needs to be done is either
- port pax from the openbsd archive again, and lose changes
- hack pax, as we have it.

On 8/18/06, Mark Reitblatt <email address hidden> wrote:
> For reasons unknown to myself, pax does not support archives > 2GB in
> size. Something about standards compliance, though I don't know what
> standards (POSIX?).
>
> ** Changed in: pax (Ubuntu)
> Importance: Untriaged => Wishlist
>
> --
> Unable to create files larger than 2GB
> https://launchpad.net/bugs/56764
>

--
Andrei Faust Tanasescu

Revision history for this message
Mark Reitblatt (mark-reitblatt) wrote :

My mistake. I was thinking of the 8 GB limit on ustar archives. It looks like this bug has been stuck upstream at Debian for a year now, due to library incompatibilities with the OpenBSD version.

Bug Watch Updater (bug-watch-updater)
Changed in pax:
status: Unknown → Unconfirmed
Revision history for this message
RJ Clay (rjclay) wrote :

Thank you for your bug report. This bug has been reported to the developers of the software. You can track it and make comments here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=317466

Changed in pax:
status: New → Confirmed
Revision history for this message
Andreas Gnau (rondom) wrote :

Is this a regression? If I remember it correctly Pax was working as usual for me with hardy. Gonna test later.
I don't think the Importance should be set to Wishlist, btw. Nowadays backups are nearly always >2GB, so this should be fixed asap.

Revision history for this message
andrei.tanasescu (ftanasescu) wrote : Re: [Bug 56764] Re: Unable to create files larger than 2GB

Pax works fine, until the 2gb mark. Thank you,

On Mon, Sep 15, 2008 at 1:55 PM, Andreas Gnau <email address hidden> wrote:

> Is this a regression? If I remember it correctly Pax was working as usual
> for me with hardy. Gonna test later.
> I don't think the Importance should be set to Wishlist, btw. Nowadays
> backups are nearly always >2GB, so this should be fixed asap.
>
> --
> Unable to create files larger than 2GB
> https://bugs.launchpad.net/bugs/56764
> You received this bug notification because you are a direct subscriber
> of the bug.
>

--
Andrei Tanasescu

Revision history for this message
In , Norman Ramsey (nr-cs) wrote : pax will write a file that it will not read

Package: pax
Version: 1:1.5-16
Followup-For: Bug #317466

Rather depressing that pax will cheerfully write a 3GB file, which it
then cannot read!

: nr@homedog 8213 ; ls -lh ../yorkie-rips.pax
-rw-rw-r-- 1 nr nr 3.8G Jun 24 20:36 ../yorkie-rips.pax
: nr@homedog 8214 ; pax -rv -f ../yorkie-rips.pax
pax: Failed open to read on ../yorkie-rips.pax <File too large>

ATTENTION! pax archive volume change required.
Ready for archive volume: 1
Input archive name or "." to quit pax.
Archive name > .
Quitting pax!
: nr@homedog 8215 ;

-- System Information:
Debian Release: 5.0.1
  APT prefers stable
  APT policy: (500, 'stable'), (50, 'unstable')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-686 (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/bash

Versions of packages pax depends on:
ii libc6 2.7-18 GNU C Library: Shared libraries

pax recommends no packages.

pax suggests no packages.

-- no debconf information

Revision history for this message
In , Norman Ramsey (nr-cs) wrote : pax cannot be compiled with transparent Large File Support

I tried rebuilding pax with the transparent Large File Support,
but it relies on /usr/include/fts.h, which says

# error "<fts.h> cannot be used with -D_FILE_OFFSET_BITS==64"

I've filed a bug report against libc6-dev.

Norman

Revision history for this message
In , Norman Ramsey (nr-cs) wrote : Re: Bug#534521: libc6-dev: /usr/include/fts.h cannot be compiled with -D_FILE_OFFSET_BITS=64

 > Please note I don't maintain libc.

Right.

 > > /usr/include/fts.h, which is part of glibc, does not work with
 > > transparent Large File Support...
 >
 > The problem there seems to be struct stat *fts_statp in FTSENT.
 > There is one variant of struct stat for 32-bit file offsets and
 > another for 64-bit. fts_read and fts_children would have to know
 > which variant the caller expects, but they don't get that
 > information.

Bletch. OK, this problem isn't likely to be fixed, then.

 > The fts functions are not documented in the glibc 2.8 manual.

I think they're part of 'gnulib' rather than 'glibc'.
I'm a bit fuzzy on the distinction.

 > If a program needs large-file support for recursive directory
 > operations now, it can perhaps be ported from <fts.h> to <ftw.h>,
 > which already supports large files but doesn't provide as many
 > features. In particular, it doesn't seem to detect hard links.

Probably a problem for pax. I think the pax bug is likely to stay open.

Thanks for your porting suggestions; I've solved my immediate problems
by using cpio, which unlike pax or tar did not choke on my files.

Norman

Bug Watch Updater (bug-watch-updater)
Changed in pax (Debian):
status: New → Won't Fix
Revision history for this message
Thorsten Glaser (mirabilos) wrote :

https://bugs.launchpad.net/ubuntu/+source/pax/+bug/56764/comments/11

↑ got it right. The real problem is that it’s impossible to enable GNU’s so-called Large File Support, and pax (correctly) uses off_t, which is broken on GNU/Linux and GNU/Hurd (but not GNU/kFreeBSD) without LFS enabled.

So this is not a bug in pax.

Changed in pax (Ubuntu):
status: Confirmed → Invalid
Bug Watch Updater (bug-watch-updater)
Changed in pax (Debian):
status: Won't Fix → Fix Released
Revision history for this message
Randy Bass (randybass) wrote :

So it looks as if pax is dead in the water for serious backup use, and there is no intention of fixing it. This 2 GB limitation should be posted in the pax man page, at least that it applies to Debian distros of Linux, if not others.

Revision history for this message
Thorsten Glaser (mirabilos) wrote :

It’s documented in the package description, as this is a packaging issue, not a bug in the program:

 .
 Note that ACLs and Extended Attributes are not supported.
 Also, on Debian GNU/Hurd and Debian GNU/Linux (but not
 Debian GNU/kFreeBSD), size of archive members is limited
 to the width of the "long" type, that is, 2 GiB on platforms
 that do not have a 64-bit "long" type, due to a bug in the
 GNU C library (Debian #317466).

Other limitations are documented in paxcpio(1):

CAVEATS
     Different file formats have different maximum file sizes. It is recom‐
     mended that a format such as cpio or ustar be used for larger files.

           File format Maximum file size
           ar 10 Gigabytes - 1 Byte
           bcpio 4 Gibibytes
           sv4cpio 4 Gibibytes
           cpio 8 Gibibytes
           tar 8 Gibibytes
           ustar 8 Gibibytes

These are inherent to the archive format and cannot be changed at all.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.