[GM45E] i915 graphics corruption and hang

Binary package hint: xserver-xorg-video-intel

While testing for bug #429199 I triggered graphics corruption shortly followed by a GPU hang. Following a clean login I suspend/resume'd four times. On the fourth resume I had some graphics corruption, specifically window backgrounds were damaged with random speckles and lines. Also the window manager decorations (top, bottom, side bars) we invisible, they _were_ functional simply completly see through. After a short time (approx 1 min) the whole thing locked up apparently a GPU hang (which may well be the same issue as 429199).

GPU confirmed hung as below:

    apw@dm$ cat i915_gem_seqno
    Current sequence: 1807468
    Waiter sequence: 1807469
    IRQ sequence: 1807447
    apw@dm$ cat i915_gem_seqno
    Current sequence: 1807468
    Waiter sequence: 1807469
    IRQ sequence: 1807447
    apw@dm$ cat i915_gem_seqno
    Current sequence: 1807468
    Waiter sequence: 1807469
    IRQ sequence: 1807447
    apw@dm$

I have a GPU dump and some photos of the display corruption which I will attach.

ProblemType: Bug
Architecture: amd64
Date: Mon Sep 14 09:58:58 2009
DistroRelease: Ubuntu 9.10
MachineType: Dell Inc. Studio 1537
Package: xserver-xorg-video-intel 2:2.8.1-1ubuntu1
PccardctlIdent:

PccardctlStatus:

ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.31-10-generic root=UUID=cf503727-25f2-4ecd-b0f3-2b894523bcba ro crashkernel=384M-2G:64M,2G-:128M quiet splash
ProcEnviron:
 PATH=(custom, user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-10.32-generic
RelatedPackageVersions:
 xserver-xorg 1:7.4+3ubuntu5
 libgl1-mesa-glx 7.6.0~git20090817.7c422387-0ubuntu3
 libdrm2 2.4.13-1ubuntu1
 xserver-xorg-video-intel 2:2.8.1-1ubuntu1
 xserver-xorg-video-ati 1:6.12.99+git20090825.fc74e119-0ubuntu1
SourcePackage: xserver-xorg-video-intel
Uname: Linux 2.6.31-10-generic x86_64
dmi.bios.date: 09/22/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A03
dmi.board.vendor: Dell Inc.
dmi.board.version: A03
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: A03
dmi.modalias: dmi:bvnDellInc.:bvrA03:bd09/22/2008:svnDellInc.:pnStudio1537:pvrA03:rvnDellInc.:rn:rvrA03:cvnDellInc.:ct8:cvrA03:
dmi.product.name: Studio 1537
dmi.product.version: A03
dmi.sys.vendor: Dell Inc.
fglrx: Not loaded
system:
 distro: Ubuntu
 architecture: x86_64kernel: 2.6.31-10-generic

GPU dump during the GPU hang.

An example of the corruption. Note the mangled window menu line on pidgin. The corruption was worse initially but was cleaned up somewhat by the natural update of pidgin as people were found and populated into the list. The menu title only appeared when clicked.

An example of the missing window decorations from the window manager. In this shot I am actually holding the window by the 'top bar' and moving it.

Have had a second occurance of this, corruption leading to a GPU hang:

  apw@dm$ cat i915_gem_interrupt
  Interrupt enable: 00028053
  Interrupt identity: 00000000
  Interrupt mask: fffc5fac
  Pipe A stat: 00000000
  Pipe B stat: 00400306
  Interrupts received: 94979
  Current sequence: 572828
  Waiter sequence: 572829
  IRQ sequence: 572765

Both of those traces look similar, and superficially seem correct. It would be useful to tally the indirect buffers (the textures and the vertex buffers) with the contents of /sys/kernel/debug/dri/0/i915_gem_active to double check the addresses are valid.

@Chris -- will get that information too next time round...

It has also been suggested that the patch below might be pertinent (from drm-intel). Can you tell if we might have a split command from the GPU dumps?

commit 0ef82af7253c1929a3995f271b8b0db462d1a0c3
Author: Chris Wilson <email address hidden>
Date: Sat Sep 5 18:07:06 2009 +0100

    drm/i915: Pad ringbuffer with NOOPs before wrapping

    According to the docs, the ringbuffer is not allowed to wrap in the middle
    of an instruction.

    G45 PRM, Vol 1b, p101:
      While the “free space” wrap may allow commands to be wrapped around the
      end of the Ring Buffer, the wrap should only occur between commands.
      Padding (with NOP) may be required to follow this restriction.

    Do as commanded.

    [Having seen bug reports where there is evidence of split commands, but
    apparently the GPU has continued on merrily before a bizarre and untimely
    death, this may or may not fix a few random hangs.]

@Chris -- I had a quick look at the bug the patch above relates to (below) where you indicate the command is split by the buffer wrap in the first trace:

    https://bugs.freedesktop.org/show_bug.cgi?id=22336

Looking at the first trace on that bug the ring buffer looks as below:

    ringbuffer at 0x00000000:
    0x00000000: 0x02000004: MI_FLUSH
    [...]
   0x0001fff4: 0x01000000: MI_USER_INTERRUPT
   0x0001fff8: 0x18800180: MI_BATCH_BUFFER_START
   0x0001fffc: 0x0fa15000: dword 1

If we look at both of the dumps I have:

    ringbuffer at 0x00000000:
    0x00000000: 0x18800180: MI_BATCH_BUFFER_START
    0x00000004: 0x08000000: dword 1
    0x00000008: 0x02000004: MI_FLUSH
    [...]
    0x0001fff0: 0x10800001: MI_STORE_DATA_INDEX
    0x0001fff4: 0x00000080: dword 1
    0x0001fff8: 0x001b8c80: dword 2
    0x0001fffc: 0x01000000: MI_USER_INTERRUPT

and:

    ringbuffer at 0x00000000:
    0x00000000: 0x02000004: MI_FLUSH
    [...]
    0x0001fff4: 0x01000000: MI_USER_INTERRUPT
    0x0001fff8: 0x18800180: MI_BATCH_BUFFER_START
    0x0001fffc: 0x09f85000: dword 1

I am unsure about the first of these but cirtainly the second appears identicle over the buffer wrap? Could this be the same issue?

No, those 2 traces were clean of wrapped instructions in the ringbuffer. Eric spotted an issue inside ... one of the layers, I think it was drm, where he adjusted the presumed_offset by the real address and not a delta, resulting in the GPU making out-of-bounds reads.

You can try testing the patch available from:
http://lists.freedesktop.org/archives/intel-gfx/2009-September/004090.html
to see if detects this error.

I just managed to reproduce this in a very fresh instance, such that the ring buffer was not even filled once before it wedged. Therefore we can categorically rule that out. Attached is the GPU dump once more. I will also attach a tarball of the other files you requested.

Here are the other files.

mesa commit "i965: Fix relocation delta for WM surfaces" (5604b27b93), solves the same (or at least similar) issue for me on Intel GM965. It was the error "kernel: [ 184.761959] PGTBL_ER: 0x00800000" in syslog that pointed me to that commit. Both the missing window borders and the freeze are fixed now.

Andy: do you have any error messages like that PGTBL_ER or like "render error detected, EIR: 0x00000010" in your syslogs?

On Tue, Sep 15, 2009 at 01:13:35PM -0000, Albert Damen wrote:
> mesa commit "i965: Fix relocation delta for WM surfaces" (5604b27b93),
> solves the same (or at least similar) issue for me on Intel GM965. It
> was the error "kernel: [ 184.761959] PGTBL_ER: 0x00800000" in syslog
> that pointed me to that commit. Both the missing window borders and the
> freeze are fixed now.
>
> Andy: do you have any error messages like that PGTBL_ER or like "render
> error detected, EIR: 0x00000010" in your syslogs?

Yes I do have at least one from this morning:

    Sep 15 08:14:32 dm kernel: [ 7689.536305] render error detected, EIR: 0x00000010
    Sep 15 08:14:32 dm kernel: [ 7689.536307] IPEIR: 0x00000000
    Sep 15 08:14:32 dm kernel: [ 7689.536309] IPEHR: 0x01000000
    Sep 15 08:14:32 dm kernel: [ 7689.536310] INSTDONE: 0xfffffffe
    Sep 15 08:14:32 dm kernel: [ 7689.536311] INSTPS: 0x0001e000
    Sep 15 08:14:32 dm kernel: [ 7689.536312] INSTDONE1: 0xffffffff
    Sep 15 08:14:32 dm kernel: [ 7689.536314] ACTHD: 0x07c19620
    Sep 15 08:14:32 dm kernel: [ 7689.536315] page table error
    Sep 15 08:14:32 dm kernel: [ 7689.536316] PGTBL_ER: 0x00100000
    Sep 15 08:14:32 dm kernel: [ 7689.536318] [drm:i915_handle_error] *ERROR* EIR stuck: 0x00000010, masking

This would be the bug pointing to the mesa update Albert is pointing us to:

    Freedesktop bug 23254 in Drivers/DRI/i965 "Compiz doesn't survive suspend/resume cycle" [Major,Resolved: fixed] http://bugzilla.freedesktop.org/show_bug.cgi?id=23254

In parallel a couple of kernel patches have been suggested, one now eliminated, the other sounding pretty relevant to this issue and the suggested fix from Albert. I have therefore built a test kernel with these applied and uploaded it to the URL below. Those affected by this issue may well want to try this kernel to see if it fairs better. Kernels will be here shortly:

    http://people.canonical.com/~apw/lp429241-karmic/

Ok. The second patch here immediatly picks up multiple violations in the relocation offsets:

Sep 15 15:46:58 dm kernel: [ 111.176361] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216.
Sep 15 15:46:58 dm kernel: [ 111.176366] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22
Sep 15 15:46:58 dm kernel: [ 111.249110] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216.
Sep 15 15:46:58 dm kernel: [ 111.249115] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22
Sep 15 15:46:58 dm kernel: [ 111.389150] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216.
Sep 15 15:46:58 dm kernel: [ 111.389155] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22
Sep 15 15:46:58 dm kernel: [ 111.574529] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216.
Sep 15 15:46:58 dm kernel: [ 111.574535] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22
Sep 15 15:46:59 dm kernel: [ 111.615321] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj ffff88010895da80 target 2 delta 153751552 size 16777216.
Sep 15 15:46:59 dm kernel: [ 111.615326] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22

My screen is mostly unpainted, nothing compositied is visible. But it does not crash. 10 s/r without a crash, some kind of record.

The fix from upstream (Eric Anholt) is a (very straightforward) one line patch which fixes a quite obvious problem in the relocation delta:
http://cgit.freedesktop.org/mesa/mesa/commit/?id=acfea5c705f383692e661d37c5cd7da2f3db559b

The patch fixes a regression (see the error in comment 16 about the delta) the kernel complained about.

Booting the affected machine with a mesa built using that patch stops the kernel for complaining about Relocation: and returns full screen output. Multiple suspend/resume cycles without any issues, either graphics corruption or hangs.

Although the overall fix is a fix to mesa, it seems that the extra protection offered by this Relocation check is well worth having in the kernel. Will recommend this for inclusion in the karmic kernel.

the fix solved the problem on my side.
thx

Closing the -intel task, as the bug was fixed in mesa:

mesa (7.6.0~git20090817.7c422387-0ubuntu4) karmic; urgency=low

  * Add 109_fix_relocation_delta_for_wm_surfaces.patch: Fix relocation
    delta for WM surfaces. This was a regression introduced in
    0f328c90dbc893e15005f2ab441d309c1c176245 (LP: #429241).

 -- Alberto Milone <email address hidden> Tue, 15 Sep 2009 18:14:48 +0200

This bug was fixed in the package linux - 2.6.31-10.35

---------------
linux (2.6.31-10.35) karmic; urgency=low

  [ Amit Kucheria ]

  * Disable CONFIG_UEVENT_HELPER_PATH

  [ Andy Whitcroft ]

  * [Config] Enable CONFIG_USB_GADGET_DUMMY_HCD
  * remove the tlsup driver
  * remove lmpcm logitech driver support

  [ Bryan Wu ]

  * Add 3 missing files to prerm remove file list
    - LP: #345623, #415832

  [ Chris Wilson ]

  * [Upstream] drm/i915: Check that the relocation points to within the
    target
    - LP: #429241

  [ Luke Yelavich ]

  * [Config] Set CONFIG_EXT4_FS=y on ports architectures

  [ Manoj Iyer ]

  * SAUCE: Added quirk to recognize GE0301 3G modem as an interface.
    - LP: #348861

  [ Tim Gardner ]

  * Revert "[Upstream] ACPI: Add Thinkpad W500, W700, & W700ds to OSI(Linux) white-list"
  * Revert "[Upstream] ACPI: Add Thinkpad R400 & Thinkpad R500 to OSI(Linux) white-list"
  * Revert "[Upstream] ACPI: Add Thinkpad X300 & Thinkpad X301 to OSI(Linux) white-list"
  * Revert "[Upstream] ACPI: Add Thinkpad X200, X200s, X200t to OSI(Linux) white-list"
  * Revert "[Upstream] ACPI: Add Thinkpad T400 & Thinkpad T500 to OSI(Linux) white-list"
    Upstream suggests that this is not the right approach.

  * [Config] Set default I/O scheduler to DEADLINE
    CFQ seems to have some load related problems which are often exacerbated by sreadahead.
    - LP: #381300

  [ <email address hidden> ]

  * SAUCE: ipw2200: Enable LED by default
    - LP: #21367

  [ Upstream Kernel Changes ]

  * ALSA: hda - Add support for new AMD HD audio devices
    - LP: #430564

 -- Andy Whitcroft <email address hidden> Wed, 16 Sep 2009 15:37:49 +0100

I have the latest everything in karmic, and it still happens:

[ 175.073448] [drm:i915_gem_object_pin_and_relocate] *ERROR* Relocation beyond target object bounds: obj f3e43d20 target 752 delta 99729408 size 8388608.
[ 175.073456] [drm:i915_gem_execbuffer] *ERROR* Failed to pin buffers -22

... and so forth many many times.

Luckily, I can turn off desktop effects (I use Kubuntu), and if I downgrade down to 2.6.31-9 it works fine. -10 and -11 are still broken.

@kelvie
Can you attach the output of this command, please?

apt-cache policy libgl1-mesa-glx

As requested:

libgl1-mesa-glx:
  Installed: 7.6.0+git20090906.97787317-0ubuntu0tormod~jaunty
  Candidate: 7.6.0+git20090906.97787317-0ubuntu0tormod~jaunty
  Version table:
 *** 7.6.0+git20090906.97787317-0ubuntu0tormod~jaunty 0
        100 /var/lib/dpkg/status
     7.6.0~git20090817.7c422387-0ubuntu6 0
        500 http://ubuntu.mirror.rafal.ca karmic/main Packages

@kelvie
Can you install mesa from Karmic's repositories (instead of the the one in the PPA), please?
7.6.0~git20090817.7c422387-0ubuntu6

Ah, I was running the old xorg-edgers PPA when I was in Jaunty.

I didn't know it looks at the version numbers and doesn't upgrade packages that are newer, even when upgrading to Karmic.

This seems similar to what is happening on my Inspiron 4150 notebook. Suspend seems to work when I shut the lid, but on resume I end up with a black screen with random vertical red stripes. In my case, I am running a fresh installation of 9.10 from the 'alternate' install image, not an upgrade, and I have not changed anything yet (installed Pidgin. Unrelated, but I think this is all I changed from default installation). I can post more info if needed.

I have noticed some other graphics issues that make me think this could be unrelated to this bug.
-bright/dim controls don't work (didn't in last version of ubuntu either though)
-sometimes, when I adjust volume, the graphic volume indicator comes up graphically corrupt.

Happy to post more info, you just might have to tell me where to find it!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lucid' to 'verification-done-lucid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!