kernel set the wrong max_sectors_kb for 4K disks(sd.c)

Bug #1418339 reported by bugproxy
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Chris J Arges

Bug Description

[Impact]
Problem Description
===========================
We get following error messages on the console and IO fails.

[ 63.322503] sd 0:2:2:0: [sdc]
[ 63.322603] Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[ 63.322661] sd 0:2:2:0: [sdc]
[ 63.322697] Sense Key : Illegal Request [current]
[ 63.322757] sd 0:2:2:0: [sdc]
[ 63.322792] Add. Sense: Invalid field in cdb
[ 63.322839] sd 0:2:2:0: [sdc] CDB:
[ 63.322874] Write(10): 2a 00 03 02 17 a0 00 00 80 00
[ 63.323038] blk_update_request: critical target error, dev sdc, sector 403750144
[ 63.323117] sd 0:2:2:0: [sdc]
[ 63.323152] Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[ 63.323210] sd 0:2:2:0: [sdc]
[ 63.323245] Sense Key : Illegal Request [current]
[ 63.323303] sd 0:2:2:0: [sdc]
[ 63.323338] Add. Sense: Invalid field in cdb
[ 63.323385] sd 0:2:2:0: [sdc] CDB:
[ 63.323420] Write(10): 2a 00 03 02 17 20 00 00 80 00
[ 63.323580] blk_update_request: critical target error, dev sdc, sector 403749120

[ 138.410612] EXT4-fs warning (device dm-2): ext4_end_bio:317: I/O error -121 writing to inode 265776 (offset 0 size 393216 starting block 33920)
[ 138.410727] Buffer I/O error on device dm-2, logical block 33920
[ 138.410785] Buffer I/O error on device dm-2, logical block 33921
[ 138.410842] Buffer I/O error on device dm-2, logical block 33922
[ 138.410900] Buffer I/O error on device dm-2, logical block 33923
[ 138.410957] Buffer I/O error on device dm-2, logical block 33924

---uname output---
Ubuntu 15.04

Machine Type = P8(Alpine)

[Test Case]
Steps to Reproduce
=====================
 read/write to a 4K array.

[Fix]
commit 3a9794d32984b67a6d8992226918618f0e51e5d5 upstream

Revision history for this message
bugproxy (bugproxy) wrote : the patch is against 3.18

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-121259 severity-high targetmilestone-inin---
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1418339/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
affects: ubuntu → linux (Ubuntu)
tags: added: kernel-da-key
Revision history for this message
Chris J Arges (arges) wrote :
Changed in linux (Ubuntu Trusty):
importance: Undecided → Critical
Changed in linux (Ubuntu):
importance: Undecided → High
Changed in linux (Ubuntu Trusty):
importance: Critical → High
Changed in linux (Ubuntu Utopic):
importance: Undecided → High
Changed in linux (Ubuntu):
status: New → Confirmed
Changed in linux (Ubuntu Trusty):
status: New → Confirmed
Changed in linux (Ubuntu Utopic):
status: New → Confirmed
Changed in linux (Ubuntu):
assignee: nobody → Chris J Arges (arges)
Changed in linux (Ubuntu Trusty):
assignee: nobody → Chris J Arges (arges)
Changed in linux (Ubuntu Utopic):
assignee: nobody → Chris J Arges (arges)
Revision history for this message
Chris J Arges (arges) wrote :

Nevermind... should have git pulled this morning. : )
commit 3a9794d32984b67a6d8992226918618f0e51e5d5

description: updated
Revision history for this message
Chris J Arges (arges) wrote :

Is it safe to assume that this patch is:
Fixes: bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec ?

Thanks,

Changed in linux (Ubuntu Trusty):
assignee: Chris J Arges (arges) → nobody
Changed in linux (Ubuntu Utopic):
assignee: Chris J Arges (arges) → nobody
no longer affects: linux (Ubuntu Trusty)
no longer affects: linux (Ubuntu Utopic)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2015-02-06 17:15 EDT-------
(In reply to comment #9)
> Is it safe to assume that this patch is:
> Fixes: bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec ?

Yes. 3a9794d32984b67a6d8992226918618f0e51e5d5 fixes a regression introduced by bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec.

Revision history for this message
Chris J Arges (arges) wrote :

Sent to ML for 3.18 since this is high priority. Since we'll rebase on 3.19 for vivid, we'll pick it up there anyway.

Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

We picked this up in Vivid (15.04) via a rebase to 3.18.7 and as already noted this will be in 3.19 by default. Marking this Fix Committed.

Changed in linux (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
Leann Ogasawara (leannogasawara) wrote :

a v3.19 based kernel for Vivid (15.04) has been uploaded to the archive. Marking this Fix Released.

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
bugproxy (bugproxy)
tags: added: targetmilestone-inin1504
removed: targetmilestone-inin---
Revision history for this message
Mauro Casula (bodale) wrote :
Download full text (5.6 KiB)

Hello guys,
   same problem in Ubuntu 16.04 (after upgrade from 14.02) with SAN.

The server's kernel log is:

Dec 21 18:59:55 <*server_hostname*> kernel: [1932683.842008] scsi host11: iSCSI Initiator over TCP/IP
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.116519] scsi 11:0:0:0: Enclosure HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.117786] ses 11:0:0:0: Attached Enclosure device
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.117937] ses 11:0:0:0: Attached scsi generic sg4 type 13
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.126968] scsi 11:0:0:2: Direct-Access HP MSA 1040 SAN G105 PQ: 0 ANSI: 5
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.129225] sd 11:0:0:2: Attached scsi generic sg5 type 0
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.129617] sd 11:0:0:2: [sdc] 11708145664 512-byte logical blocks: (5.99 TB/5.45 TiB)
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.130833] sd 11:0:0:2: [sdc] Write Protect is off
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.130838] sd 11:0:0:2: [sdc] Mode Sense: fb 00 00 08
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.131304] sd 11:0:0:2: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.177704] sdc: sdc1
Dec 21 18:59:56 <*server_hostname*> kernel: [1932684.180951] sd 11:0:0:2: [sdc] Attached SCSI disk
Dec 21 19:03:36 <*server_hostname*> kernel: [1932904.596016] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
Dec 21 19:05:21 <*server_hostname*> kernel: [1933009.667135] sdc: sdc1
Dec 21 19:05:21 <*server_hostname*> kernel: [1933009.681393] sdc: sdc1
Dec 21 19:06:29 <*server_hostname*> kernel: [1933077.102802] sdc: sdc1
Dec 21 19:06:48 <*server_hostname*> kernel: [1933096.995685] EXT4-fs (sdc1): mounted filesystem with ordered data mode. Opts: (null)
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375691] sd 11:0:0:2: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375713] sd 11:0:0:2: [sdc] tag#1 Sense Key : Illegal Request [current]
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375720] sd 11:0:0:2: [sdc] tag#1 Add. Sense: Invalid field in cdb
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375727] sd 11:0:0:2: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 04 48 00 00 00 40 00 00 00
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375731] blk_update_request: 145 callbacks suppressed
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375735] blk_update_request: critical target error, dev sdc, sector 280576
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375745] EXT4-fs warning (device sdc1): ext4_end_bio:329: I/O error -121 writing to inode 173277192 (offset 0 size 8388608 starting block 35328)
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375753] buffer_io_error: 315894 callbacks suppressed
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375756] Buffer I/O error on device sdc1, logical block 34816
Dec 21 19:08:43 <*server_hostname*> kernel: [1933211.375766] Buf...

Read more...

Revision history for this message
Slava (dbazim) wrote :

I am also affected bu that bug
usb 2-5: new high-speed USB device number 2 using ehci-pci
[ 248.566881] usb 2-5: New USB device found, idVendor=0c76, idProduct=0005
[ 248.566885] usb 2-5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 248.566889] usb 2-5: Product: USB Flash Disk
[ 248.566891] usb 2-5: Manufacturer: SanDisk
[ 248.566894] usb 2-5: SerialNumber: 444553544200F040383D0C57
[ 248.567386] usb-storage 2-5:1.0: USB Mass Storage device detected
[ 248.570210] scsi host16: usb-storage 2-5:1.0
[ 249.571364] scsi 16:0:0:0: Direct-Access SanDisk USB Flash Disk 1.00 PQ: 0 ANSI: 2
[ 249.571994] sd 16:0:0:0: Attached scsi generic sg3 type 0
[ 249.572507] sd 16:0:0:0: [sdd] 31273440 512-byte logical blocks: (16.0 GB/14.9 GiB)
[ 249.573227] sd 16:0:0:0: [sdd] Write Protect is off
[ 249.573231] sd 16:0:0:0: [sdd] Mode Sense: 03 00 00 00
[ 249.574568] sd 16:0:0:0: [sdd] No Caching mode page found
[ 249.574575] sd 16:0:0:0: [sdd] Assuming drive cache: write through
[ 249.580859] sdd: sdd1
[ 249.583771] sd 16:0:0:0: [sdd] Attached SCSI removable disk
[ 249.608466] sd 16:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 249.608472] sd 16:0:0:0: [sdd] tag#0 Sense Key : Illegal Request [current]
[ 249.608476] sd 16:0:0:0: [sdd] tag#0 Add. Sense: Invalid command operation code
[ 249.608481] sd 16:0:0:0: [sdd] tag#0 CDB: Read(6) 08 00 08 00 08 00
[ 249.608484] blk_update_request: critical target error, dev sdd, sector 2048

Disk works ok on Win 7 but not in ubuntu 16.04
uname -a
Linux *** 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:36:54 UTC 2017 i686 i686 i686 GNU/Linux

tags: removed: kernel-da-key
Revision history for this message
Chris Puttick (cputtick) wrote :
Download full text (5.1 KiB)

We're seeing errors very similar to this on 2 16.04 VMs, both attached to different SANs. Sadly one is the live data and the other the backup, so...

kern.log snippet:

Apr 25 14:54:52 oaebackup1 kernel: [11778105.146482] EXT4-fs (sda1): previous I/O error to superblock detected
Apr 25 14:54:52 oaebackup1 kernel: [11778105.151094] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 25 14:54:52 oaebackup1 kernel: [11778105.151102] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current]
Apr 25 14:54:52 oaebackup1 kernel: [11778105.151115] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect
Apr 25 14:54:52 oaebackup1 kernel: [11778105.151120] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00
Apr 25 14:54:52 oaebackup1 kernel: [11778105.151124] blk_update_request: critical space allocation error, dev sda, sector 2048
Apr 25 14:54:52 oaebackup1 kernel: [11778105.154859] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Apr 25 14:54:52 oaebackup1 kernel: [11778105.158519] EXT4-fs error (device sda1): ext4_readdir:224: inode #11: block 11595: comm bash: path /mnt/oae-backups/lost+found: bad entry in directoryr0
Apr 25 14:54:52 oaebackup1 kernel: [11778105.165899] EXT4-fs (sda1): previous I/O error to superblock detected
Apr 25 14:54:52 oaebackup1 kernel: [11778105.170487] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 25 14:54:52 oaebackup1 kernel: [11778105.170494] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current]
Apr 25 14:54:52 oaebackup1 kernel: [11778105.170503] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect
Apr 25 14:54:52 oaebackup1 kernel: [11778105.170511] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00
Apr 25 14:54:52 oaebackup1 kernel: [11778105.170515] blk_update_request: critical space allocation error, dev sda, sector 2048
Apr 25 14:54:52 oaebackup1 kernel: [11778105.174225] Buffer I/O error on dev sda1, logical block 0, lost sync page write
Apr 25 14:55:26 oaebackup1 kernel: [11778139.432404] EXT4-fs error: 851 callbacks suppressed
Apr 25 14:55:26 oaebackup1 kernel: [11778139.432430] EXT4-fs error (device sda1): ext4_lookup:1583: inode #77009420: comm ls: deleted inode referenced: 77531011
Apr 25 14:55:26 oaebackup1 kernel: [11778139.436785] EXT4-fs: 851 callbacks suppressed
Apr 25 14:55:26 oaebackup1 kernel: [11778139.436789] EXT4-fs (sda1): previous I/O error to superblock detected
Apr 25 14:55:26 oaebackup1 kernel: [11778139.442236] scsi_io_completion: 851 callbacks suppressed
Apr 25 14:55:26 oaebackup1 kernel: [11778139.442248] sd 6:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 25 14:55:26 oaebackup1 kernel: [11778139.442255] sd 6:0:0:0: [sda] tag#0 Sense Key : Data Protect [current]
Apr 25 14:55:26 oaebackup1 kernel: [11778139.442260] sd 6:0:0:0: [sda] tag#0 Add. Sense: Space allocation failed write protect
Apr 25 14:55:26 oaebackup1 kernel: [11778139.442266] sd 6:0:0:0: [sda] tag#0 CDB: Write(16) 8a 08 00 00 00 00 00 00 08 00 00 00 00 08 00 00
Apr 25 14:55:26 oaebackup1 kernel...

Read more...

Revision history for this message
Chris Puttick (cputtick) wrote :

Additional: a 3rd SAN/VM pairing, part of the same wave of installation and in most respects an identical set up is not experiencing this issue - however this one was completed earlier in cycle so has a different kernel:
# uname -a
Linux oandata1 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

The other difference is the volume is larger 13T v. 5.8T on the 2 experiencing issues.

Revision history for this message
Chris Puttick (cputtick) wrote :

After sourcing a new backup store and backing up the live data, we've updated the server and rebooted. On mount errors were reported with the filesystem so e2fsck used. Some errors were found and fixed, remounted, no errors. First attempted write to the volume (a simple save of new text file) returns an error, with the following logged (where the entries below are from the point the disk was remounted after e2fsck):

Apr 27 17:52:05 oaedata1 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Apr 27 17:52:48 oaedata1 kernel: scsi_io_completion: 20 callbacks suppressed
Apr 27 17:52:48 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 27 17:52:48 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 Sense Key : Medium Error [current]
Apr 27 17:52:48 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 Add. Sense: Peripheral device write fault
Apr 27 17:52:48 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 CDB: Write(16) 8a 00 00 00 00 00 16 3e 48 00 00 00 00 08 00 00
Apr 27 17:52:48 oaedata1 kernel: blk_update_request: 20 callbacks suppressed
Apr 27 17:52:48 oaedata1 kernel: blk_update_request: I/O error, dev sda, sector 373180416
Apr 27 17:52:48 oaedata1 kernel: EXT4-fs warning (device sda1): ext4_end_bio:329: I/O error -5 writing to inode 5571545 (offset 0 size 4096 starting block 46647553)
Apr 27 17:52:48 oaedata1 kernel: Buffer I/O error on device sda1, logical block 46647296
Apr 27 17:52:52 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 27 17:52:52 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 Sense Key : Medium Error [current]
Apr 27 17:52:52 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 Add. Sense: Peripheral device write fault
Apr 27 17:52:52 oaedata1 kernel: sd 2:0:0:0: [sda] tag#0 CDB: Write(16) 8a 00 00 00 00 00 16 3e 48 08 00 00 00 18 00 00
Apr 27 17:52:52 oaedata1 kernel: blk_update_request: I/O error, dev sda, sector 373180424
Apr 27 17:52:52 oaedata1 kernel: EXT4-fs warning (device sda1): ext4_end_bio:329: I/O error -5 writing to inode 5571544 (offset 0 size 12288 starting block 46647556)
Apr 27 17:52:52 oaedata1 kernel: Buffer I/O error on device sda1, logical block 46647297
Apr 27 17:52:52 oaedata1 kernel: Buffer I/O error on device sda1, logical block 46647298
Apr 27 17:52:52 oaedata1 kernel: Buffer I/O error on device sda1, logical block 46647299
lines 741-793/793 (END)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.