"attempt to access beyond end of device" when creating LVM snapshot

(CC'd agk2 as he seems to have a handle on this error in other bugs)

You need to improve your bug reporting skills:-)

Exactly what userspace commands did you run with exactly what options? What is the architecture? (Arch & chunksize interact.)

You also need to say what userspace versions of lvm2 & device-mapper you are using - it's not necessarily a kernel bug.

There have been a lot of fixes to kernel device-mapper, including snapshots, since 2.6.15. So test against the latest upstream kernel and userspace packages if you can and see whether or not the problem has already been fixed there.

Also give details if you have any layered devices - for example if you have md underneath dm. (There was a md+dm readahead bug fixed recently, for example.)

Good point, well made. My apologies, was hoping you'd recognise it just from the error because of it being reported numerous other places. :)

dmsetup/libdevmapper1.02: 1.02.05-1ubuntu1
lvm2: 2.02.02-1ubuntu1.5

The machine is a P4 - running the Ubuntu "server" kernel (which seems to be compiled CONFIG_M686=y).

A command line that causes the error:
lvcreate -L2G -s -n test /dev/Ubuntu/storage

(-L500M works, -L550M (rounded up to PE at 552MB) doesn't.

There are no layered devices - the LVM array is on an Areca IDE raid array, which appears as /dev/sda6.

I tried to put a newer kernel on the machine to test but had a bit of a problem with it, and as it's a production machine, couldn't really do much more there.

However, I've built another machine to test, and can report on a clean Dapper install (2.6.15) the problem occurs (with both xfs and ext3 filesystems), and when upgraded to Feisty (2.6.19) it doesn't.

lvm2: 2.02.06-2ubuntu8
dmsetup/libdevmapper1.0.2: 1.02.08-1ubuntu5

I'm trying to boot into 2.6.15 to see if it's the problem, but an update to udev means update-initramfs won't run and I can't boot the old kernel. Trying to find a way around that now..

Hello! I don't know if this helps, but I'm experiencing the same problem, with ext3 filesystems.

I'm running the latest version of Dapper. That is, if I run `apt-get update` and then `apt-get upgrade`, I do not see any packages listed.

`uname -a` returns "Linux npbbz 2.6.15-29-686 #1 SMP PREEMPT Mon Sep 24 17:24:47 UTC 2007 i686 GNU/Linux".
The file at '/proc/version' contains "Linux version 2.6.15-29-686 (buildd@terranova) (gcc version 4.0.3 (Ubuntu 4.0.3-1ubuntu5)) #1 SMP PREEMPT Mon Sep 24 17:24:47 UTC 2007".
The file at '/proc/cmdline' contains "root=/dev/mapper/VolGroup00-root ro".

The file at '/proc/cpuinfo' contains:
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) MP CPU 2.70GHz
stepping : 8
cpu MHz : 2694.761
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 5403.05

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) MP CPU 2.70GHz
stepping : 8
cpu MHz : 2694.761
cache size : 2048 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 5392.65

Package versions are:
  - linux-image-2.6.15-29-686: 2.6.15-29.60
  - libc6: 2.3.6-0ubuntu20.5
  - libdevmapper1.02: 1.02.05-1ubuntu1
  - dmsetup: 1.02.05-1ubuntu1
  - lvm-common: 1.5.20ubuntu4
  - lvm2: 2.02.02-1ubuntu1.5

I'll post my dmesg output soon!

Here's a control example, showing the output of lvcreate when there are no problems reported:

kornelak@npbbz:/home/bugzilla/stage$ dmesg|tail
[17180786.720000] EXT3-fs: mounted filesystem with ordered data mode.
[17180799.652000] kjournald starting. Commit interval 5 seconds
[17180799.656000] EXT3 FS on dm-41, internal journal
[17180799.656000] EXT3-fs: recovery complete.
[17180799.656000] EXT3-fs: mounted filesystem with ordered data mode.
[17203415.252000] ibm_acpi: ec object not found
[17808026.704000] ibm_acpi: ec object not found
[18412815.972000] ibm_acpi: ec object not found
[19017364.116000] ibm_acpi: ec object not found
[19622027.484000] ibm_acpi: ec object not found
kornelak@npbbz:/home/bugzilla/stage$ cd /var/lib/mysql/main
kornelak@npbbz:/var/lib/mysql/main$ sudo lvs
  LV VG Attr LSize Origin Snap% Move Log Copy%
  Database_Logs VolGroup00 -wi-ao 10.00G
  Database_Main VolGroup00 -wi-ao 5.00G
  Database_Stage_Orig VolGroup00 -wi-ao 35.00G
  Database_Transition VolGroup00 -wi-ao 80.00G
  Home VolGroup00 -wi-a- 3.00G
  Live VolGroup00 -wi-ao 94.00G
  Root VolGroup00 -wi-a- 10.00G
  Stage_Orig VolGroup00 -wi-ao 76.00G
  Swap VolGroup00 -wi-ao 2.00G
  root VolGroup00 -wi-ao 15.00G
kornelak@npbbz:/var/lib/mysql/main$ sudo lvcreate --verbose --snapshot --size 5G --name Stage_31743 VolGroup00/Stage_Orig
    Setting chunksize to 16 sectors.
    Finding volume group "VolGroup00"
    Archiving volume group "VolGroup00" metadata (seqno 473).
    Creating logical volume Stage_31743
    Creating volume group backup "/etc/lvm/backup/VolGroup00" (seqno 474).
    Found volume group "VolGroup00"
    Creating VolGroup00-Stage_31743
    Loading VolGroup00-Stage_31743 table
    Resuming VolGroup00-Stage_31743 (254:4)
    Zeroing start of logical volume "Stage_31743"
    Found volume group "VolGroup00"
    Removing VolGroup00-Stage_31743 (254:4)
    Found volume group "VolGroup00"
    Found volume group "VolGroup00"
    Found volume group "VolGroup00"
    Creating VolGroup00-Stage_Orig-real
    Loading VolGroup00-Stage_Orig-real table
    Resuming (254:4)
    Loading VolGroup00-Stage_Orig table
    Creating VolGroup00-Stage_31743-cow
    Loading VolGroup00-Stage_31743-cow table
    Resuming (254:8)
    Creating VolGroup00-Stage_31743
    Loading VolGroup00-Stage_31743 table
    Suspending VolGroup00-Stage_Orig (254:5)
    Suspending VolGroup00-Stage_Orig-real (254:4)
    Found volume group "VolGroup00"
    Loading VolGroup00-Stage_Orig-real table
    Suppressed VolGroup00-Stage_Orig-real identical table reload.
    Resuming VolGroup00-Stage_Orig-real (254:4)
    Loading VolGroup00-Stage_31743-cow table
    Suppressed VolGroup00-Stage_31743-cow identical table reload.
    Resuming VolGroup00-Stage_31743 (254:30)
    Resuming VolGroup00-Stage_Orig (254:5)
    Creating volume group backup "/etc/lvm/backup/VolGroup00" (seqno 475).
  Logical volume "Stage_31743" created
kornelak@npbbz:/var/lib/mysql/main$ dmesg|tail
[17180786.720000] EXT3-fs: mounted filesystem with ordered data mode.
[17180799.652000] kjournald starting. Com...

And, finally, an example of the problem:

kornelak@npbbz:/var/lib/mysql/main$ dmesg|tail
[17180786.720000] EXT3-fs: mounted filesystem with ordered data mode.
[17180799.652000] kjournald starting. Commit interval 5 seconds
[17180799.656000] EXT3 FS on dm-41, internal journal
[17180799.656000] EXT3-fs: recovery complete.
[17180799.656000] EXT3-fs: mounted filesystem with ordered data mode.
[17203415.252000] ibm_acpi: ec object not found
[17808026.704000] ibm_acpi: ec object not found
[18412815.972000] ibm_acpi: ec object not found
[19017364.116000] ibm_acpi: ec object not found
[19622027.484000] ibm_acpi: ec object not found
kornelak@npbbz:/var/lib/mysql/main$ sudo lvs
   LV VG Attr LSize Origin Snap% Move Log Copy%
  Database_Logs VolGroup00 -wi-ao 10.00G
  Database_Main VolGroup00 -wi-ao 5.00G
  Database_Stage_31743 VolGroup00 swi-a- 5.00G Database_Stage_Orig 0.00
  Database_Stage_32781 VolGroup00 swi-a- 5.00G Database_Stage_Orig 0.00
  Database_Stage_34352 VolGroup00 swi-a- 5.00G Database_Stage_Orig 0.00
  Database_Stage_Orig VolGroup00 owi-ao 35.00G
  Database_Transition VolGroup00 -wi-ao 80.00G
  Home VolGroup00 -wi-a- 3.00G
  Live VolGroup00 -wi-ao 94.00G
  Root VolGroup00 -wi-a- 10.00G
  Stage_31743 VolGroup00 swi-a- 5.00G Stage_Orig 0.00
  Stage_32781 VolGroup00 swi-a- 5.00G Stage_Orig 0.00
  Stage_34352 VolGroup00 swi-a- 5.00G Stage_Orig 0.00
  Stage_Orig VolGroup00 owi-ao 76.00G
  Swap VolGroup00 -wi-ao 2.00G
  root VolGroup00 -wi-ao 15.00G
kornelak@npbbz:/var/lib/mysql/main$ sudo lvcreate --verbose --snapshot --size 5G --name Database_Stage_35780 VolGroup00/Database_Stage_Orig
    Setting chunksize to 16 sectors.
    Finding volume group "VolGroup00"
    Archiving volume group "VolGroup00" metadata (seqno 485).
    Creating logical volume Database_Stage_35780
    Creating volume group backup "/etc/lvm/backup/VolGroup00" (seqno 486).
    Found volume group "VolGroup00"
    Creating VolGroup00-Database_Stage_35780
    Loading VolGroup00-Database_Stage_35780 table
    Resuming VolGroup00-Database_Stage_35780 (254:42)
    Zeroing start of logical volume "Database_Stage_35780"
    Found volume group "VolGroup00"
    Removing VolGroup00-Database_Stage_35780 (254:42)
    Found volume group "VolGroup00"
    Found volume group "VolGroup00"
    Found volume group "VolGroup00"
    Loading VolGroup00-Database_Stage_Orig-real table
    Suppressed VolGroup00-Database_Stage_Orig-real identical table reload.
    Loading VolGroup00-Database_Stage_Orig table
    Suppressed VolGroup00-Database_Stage_Orig identical table reload.
    Loading VolGroup00-Database_Stage_31743-cow table
    Suppressed VolGroup00-Database_Stage_31743-cow identical table reload.
    Loading VolGroup00-Database_Stage_31743 table
    Suppressed VolGroup00-Database_Stage_31743 identical table reload.
    Loading VolGroup00-Database_Stage_32781-cow table
    Suppressed VolGroup00-Database_Stage_32781-cow identical table reload.
    Loading Vol...

Can you please test this in Hardy Heron to see if it still a probem?

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this linux-source-2.6.15 kernel bug to the new "linux" package. We appreciate your patience and understanding as we make this transition. Also, if you would be interested in testing the upcoming Intrepid Ibex 8.10 release, it is available at http://www.ubuntu.com/testing . Please let us know your results. Thanks!

Marking this "Incomplete" until we get further feedback regarding testing against Hardy.

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

Hi,

Something similar is also happening to me on Dapper LTS 6.06.2 with Kernel 2.6.15-52-amd64-server.

I do not have a way of reproducing this at will.

In summary, I have the primary volume (u01) of 450GB sitting in a VG using a PV on /dev/cciss_c0d1p1, so no mdraid or other layering. Creating a 15GB snapshot of of the volume works as expected.

# lvcreate -s -n test1 -L 15G vg01/u01
  Logical volume "test1" created
#

I can then mount and use the snapshot.

Creating a second snapshot of the volume then resulted in:
# lvcreate -s -n test2 -L 20G vg01/u01
  Logical volume "test2" created
#

However, the volume is not usable:
# lvdisplay vg01
  --- Logical volume ---
  LV Name /dev/vg01/u01
  VG Name vg01
  LV UUID Be69s5-zvtC-OKp3-IJDR-MQFe-YWPe-A7Bm1S
  LV Write Access read/write
  LV snapshot status source of
                         /dev/vg01/test1 [active]
                         /dev/vg01/test2 [INACTIVE]
  LV Status available
  # open 1
  LV Size 450.00 GB
  Current LE 14400
  Segments 1
  Allocation inherit
  Read ahead sectors 0
  Block device 254:3

  --- Logical volume ---
  LV Name /dev/vg01/test1
  VG Name vg01
  LV UUID PYhpfX-5Wl3-x487-h4x6-Gb2y-7N1j-x4XB2c
  LV Write Access read/write
  LV snapshot status active destination for /dev/vg01/u01
  LV Status available
  # open 1
  LV Size 450.00 GB
  Current LE 14400
  COW-table size 15.00 GB
  COW-table LE 480
  Allocated to snapshot 0.40%
  Snapshot chunk size 8.00 KB
  Segments 1
  Allocation inherit
  Read ahead sectors 0
  Block device 254:12

  --- Logical volume ---
  LV Name /dev/vg01/test2
  VG Name vg01
  LV UUID HKyqse-NSu3-5b2y-So54-xZsr-oiNK-ndExdP
  LV Write Access read/write
  LV snapshot status INACTIVE destination for /dev/vg01/u01
  LV Status available
  # open 0
  LV Size 450.00 GB
  Current LE 14400
  COW-table size 20.00 GB
  COW-table LE 640
  Snapshot chunk size 8.00 KB
  Segments 1
  Allocation inherit
  Read ahead sectors 0
  Block device 254:14

#

And I see these errors in dmesg:
[792276.624583] attempt to access beyond end of device
[792276.624591] dm-13: rw=17, want=569894688, limit=41943040
[792276.624596] device-mapper: Error reading/writing snapshot
[792276.665813] attempt to access beyond end of device
[792276.665819] dm-13: rw=17, want=569902896, limit=41943040
[792276.923219] attempt to access beyond end of device
[792276.923225] dm-13: rw=17, want=569894688, limit=41943040
[792276.965357] attempt to access beyond end of device
[792276.965362] dm-13: rw=17, want=569902896, limit=41943040
[792277.253731] attempt to access beyond end of device
[792277.253737] dm-13: rw=17, want=569894688, limit=41943040
[792277.805186] device-mapper: snapshot is marked in...

Dapper is no longer supported. This bug report is being closed. Please reopen if this is still an issue in the current Ubuntu release, Jaunty Jackalope 9.04. To reopen the bug, click on the current status, under the Status column, and change the status back to "New". Thanks.