DRI graphics drivers causes too many wakeups

ping...

Does anybody know, whether this is normal and expected behaviour or if this is a
bug?

Anything that can be done about it?

(In reply to comment #1)
> ping...

Keep in mind that most people here are volunteers.

> Does anybody know, whether this is normal and expected behaviour or if this is a
> bug?

It's kind of expected, as the radeon driver uses bus mastering to send commands
to the graphics card when the DRI is enabled.

> Anything that can be done about it?

Hard to say without knowing exactly what kind of bus mastering activity pattern
is causing the problem. AFAIR someone else reported this before and provided
much more detail, but I don't remember if that was here or in XFree86's bugzilla.

(In reply to comment #2)
> (In reply to comment #1)

> > Does anybody know, whether this is normal and expected behaviour or if this is a
> > bug?
>
> It's kind of expected, as the radeon driver uses bus mastering to send commands
> to the graphics card when the DRI is enabled.
>

Well, if it's normal and expected, we can close or at least postpone the report.
There are certainly more critical bugs (such that lead to system freezes).

Just did not know, whether this was a bug. Somehow it's seemed like a bug, as
with or without dri modules loaded, I thought there is the same rendering
happening on the screen if I am idle (no 3D apps running).

> > Anything that can be done about it?
>
> Hard to say without knowing exactly what kind of bus mastering activity pattern
> is causing the problem. AFAIR someone else reported this before and provided
> much more detail, but I don't remember if that was here or in XFree86's bugzilla.

I searched the net, but only found a report on a suse mailing list. They said
it's a bug, but that report was never followed up on. If I find the report you
mentioned, I will put a link here, so that all information can be accessed from
one place.

> > ping...
>
> Keep in mind that most people here are volunteers.

Sure, I do. Just wanted to find out if I myself can do anything to help.

Thanks,

Michael

(In reply to comment #3)
> (In reply to comment #2)
> > (In reply to comment #1)
>
> > > Does anybody know, whether this is normal and expected behaviour or if
this is a
> > > bug?
> >
> > It's kind of expected, as the radeon driver uses bus mastering to send commands
> > to the graphics card when the DRI is enabled.
> >
>
> Well, if it's normal and expected, we can close or at least postpone the report.
> There are certainly more critical bugs (such that lead to system freezes).
>
> Just did not know, whether this was a bug. Somehow it's seemed like a bug, as
> with or without dri modules loaded, I thought there is the same rendering
> happening on the screen if I am idle (no 3D apps running).

well, when the DRI is enabled, the DDX uses the bus mastering for 2D commands as
well, so that will also cause bus master activity. I suppose we could put in
some sort of hack to shut down the CP when DPMS is triggered. I'm not sure if
that's the right fix or not.

(In reply to comment #4)
> well, when the DRI is enabled, the DDX uses the bus mastering for 2D commands as
> well, so that will also cause bus master activity. I suppose we could put in
> some sort of hack to shut down the CP when DPMS is triggered. I'm not sure if
> that's the right fix or not.
>

I'm afraid I haven't dived far enough into the technical aspects, so I don't
know teh relationship between DPMS (which is the power saving state of the
monitor, isn't it) and bus mastering, direct rendering or the CPU power saving
states...

Thus, maybe just one comment from a users perspective:

- The ideal world is (of course) to have maximum 3D power with lowest energy
consumption ;-) I guess, that is not possible, if bus mastering is needed for
"full speed graphics"

- Best compromise would be if one could choose either to enable "full speed
graphics" (using bus mastering extensively for everything) or (if possible)
"good speed graphics" with bus mastering only when necessary (i.e. for 3D apps ?
I don't know)

To my mind, I like to have a low power consumption if I'm running on battery and
have to do a lot of work. But then I also like to at least be able to start some
3D apps such as Google Earth or a nice 3D game without always modifying
xorg.conf and restarting the xserver.

It would be alright if power consumption raises when I start a 3D app. But from
a users perspective it seems unnecessary to significantly raise power
consumption without using it. (At the moment I'm running without the dri module
and for 2D apps -- such as open office, firefox, etc. -- speed is totally fine
with low power consumption -> CPU frequently in C3. But of course, no 3D
capability at all).

This is certainly not a crucial bug/feature request. There are more pressing
bugs in the driver that are of much higher priority (especially the freezes with
the dri module). So, it's fine for me if this is postponed to some later time.

Thanks for your help

Michael

(In reply to comment #5)
> (In reply to comment #4)
> > well, when the DRI is enabled, the DDX uses the bus mastering for 2D commands as
> > well, so that will also cause bus master activity. I suppose we could put in
> > some sort of hack to shut down the CP when DPMS is triggered. I'm not sure if
> > that's the right fix or not.
> >
>
> I'm afraid I haven't dived far enough into the technical aspects, so I don't
> know teh relationship between DPMS (which is the power saving state of the
> monitor, isn't it) and bus mastering, direct rendering or the CPU power saving
> states...
>

well presumably, if the the display has entered the DPMS off state, then you
aren't using the GUI, so the it should be safe to shut down the CP. I suspect a
better solution would be to use acpi hooks to turn off the CP when the kernel
tries to enter c3 (assuming the command queue is emtpy). However, as it stands
the x server doesn't have the hooks to do this at the moment as far as I know.
It's defintely a reasonable request, I'm just not sure of the best way to solve
it. I'm not sure why there would be bus master activity if there are no
commands being sent; that's what we need to figure out.

(In reply to comment #6)
>
> well presumably, if the the display has entered the DPMS off state, then you
> aren't using the GUI, so the it should be safe to shut down the CP.

Ahh, o.k. But I think this would not really make a big difference, as then CPU
power saving would only occur when the screen has turned off. However, as long
as one is actively working, there will be no benefit. E.g. at the moment I am
writing this text, I have some six to seven apps running (several open office
docs, firefox, kontact and a vmware winxp session). Nonetheless, the cpu is in
c3 about 33% of the time, thus giving me nearly 20 min more battery life (2:35
instead of 2:15).

> I'm not sure why there would be bus master activity if there are no
> commands being sent; that's what we need to figure out.
>

I feel that is the main question. I don't know what bus mastering is used for in
the driver. But it can't be something essential as I can work now without any
problems or "slowliness" without the dri module. Maybe bus mastering is
necessary for 3D apps in order to achieve high rendering performance? But right
now (with only the "office" apps running), I think, it should be perfectly o.k.,
when no bus mastering is used even with the dri module loaded.

However, if bus mastering is absolutely necessary for 3D, then with the advent
of xgl, when the whole desktop will use 3D commands, there will be no
possibility to enter c3 power saving mode, I guess. In this case, it's maybe not
worth the work to find out what's the cause for the bus mastering activity in
pure 2D apps with the dri module.

What do you think? Is it much work to find out what's the cause of the bus
mastering activity?

Cheers,

Michael

(In reply to comment #7)
> (In reply to comment #6)
> >
> > well presumably, if the the display has entered the DPMS off state, then you
> > aren't using the GUI, so the it should be safe to shut down the CP.
>
> Ahh, o.k. But I think this would not really make a big difference, as then CPU
> power saving would only occur when the screen has turned off. However, as long
> as one is actively working, there will be no benefit. E.g. at the moment I am
> writing this text, I have some six to seven apps running (several open office
> docs, firefox, kontact and a vmware winxp session). Nonetheless, the cpu is in
> c3 about 33% of the time, thus giving me nearly 20 min more battery life (2:35
> instead of 2:15).
>

Ah, I didn't realize you were trying to use c3 while you were working. I
thought you were seeing bus master activity when the computer was not being
used. If you are working, windows are updating and stuff is getting drawn (2D
or otherwise). That's what's causing the bus master activity.

>
> However, if bus mastering is absolutely necessary for 3D, then with the advent
> of xgl, when the whole desktop will use 3D commands, there will be no
> possibility to enter c3 power saving mode, I guess. In this case, it's maybe not
> worth the work to find out what's the cause for the bus mastering activity in
> pure 2D apps with the dri module.
>
> What do you think? Is it much work to find out what's the cause of the bus
> mastering activity?

In this case, I know what's causing the activity. It's the 2D drawing engine
drawing stuff on the screen while you work. As I said, when the DRI is enabled,
the 2D engines uses the CP to send commands as well (just like 3D). To get
around that you'd have to implement a way to switch the accel code between CP
and MMIO dynamically, or implement 3D using MMIO.

I have exactly the same problem.
C3 is never entered just by enabling the DRI module.
System is a Thinkpad X31 too, so this seems to be a problem with the ATI
Mobility Radeon 7000.
C3 is also not entered, when the X-server is idling.
My system gets slightly hotter with the cpu never being able to relax.

Michael

(In reply to comment #8)
>
> In this case, I know what's causing the activity. It's the 2D drawing engine
> drawing stuff on the screen while you work. As I said, when the DRI is enabled,
> the 2D engines uses the CP to send commands as well (just like 3D). To get
> around that you'd have to implement a way to switch the accel code between CP
> and MMIO dynamically, or implement 3D using MMIO.
>

O.K. then it makes perfectly sense (on a technical level).

However, implementing a way to switch between CP and MMIO acceleration would be
quite nice, as a c3 CPU state gives me about 25% more battery life (20 to 30
min) and the CPU is appr. 5 to 10 degrees Celsius colder.

Any idea, how much performance drop we would see, if the whole accel would
switch to MMIO ? Just a bit like 10% or much more like 50%?

In generell an xorg.conf option would be great saying "Accel" "MMIO" for those
who need the extra battery life and "Accel" "CP" for those that need maximum
graphics speed.

But this is certainly no longer a "real bug", rather a "feature request" now.

Thanks for the explanation and your help!

Best regards,

Michael

Switching everything DRI to MMIO would be an awful lot of work, I don't think
that's worth it. Originally, the radeon driver used to switch to MMIO on the fly
for 2D, but that caused instabilities, so it was changed to use the CP
exclusively when the DRI is enabled. This also allows accelerating more 2D
operations and yields slightly better performance for some of the others.

Even so, it might be possible to tweak the driver such that this issue can be
avoided and power saving states entered when the X server is idle. But again,
that requires knowing exactly what prevents that right now.

(In reply to comment #11)
> Switching everything DRI to MMIO would be an awful lot of work, I don't think
> that's worth it. Originally, the radeon driver used to switch to MMIO on the fly
> for 2D, but that caused instabilities, so it was changed to use the CP
> exclusively when the DRI is enabled. This also allows accelerating more 2D
> operations and yields slightly better performance for some of the others.
>

O.K. I see. Instabilities are much worse than just a higher power consumption.
So, in this case I'd rather lower battery life than any system freezes...

> Even so, it might be possible to tweak the driver such that this issue can be
> avoided and power saving states entered when the X server is idle. But again,
> that requires knowing exactly what prevents that right now.

Would be great if this could be done.

However, there might be different interpretations about "being idle". I'm not
sure what Michael Kamper means in his comment 9, but in my case "idle" means:
KDE has started, some apps are running, e.g. Open Office. Maybe no scrolling,
but there is always some drawing, e.g. the blinking cursor, the clock on the
kicker app, etc.

Of course, one might think of the DPMS modes as potential hooks to allow the CPU
to enter c3, when the screen has been blanked (as Alex suggested). But I'm not
sure that the xserver is really idle in this case, either.

If it still would be possible to tweak the driver to avoid any "unnecessary" bus
mastering activity, then I will be happy to assist whereever I can, especially
with testing some patches (I don't have much experience with patching, but I
guess, I should be able to download, patch and compile the dri driver.)

Michael

P.S.:

Looking at xgl, which may eventually become the standard in some time (or other
3D desktop implementations), if this would cause much more CP activity, then
maybe it's not worth the work at the moment to try to avoid bus mastering in 2D
apps at all. What do you think?

(In reply to comment #12)
>
> However, there might be different interpretations about "being idle". I'm not
> sure what Michael Kamper means in his comment 9, but in my case "idle" means:
> KDE has started, some apps are running, e.g. Open Office. Maybe no scrolling,
> but there is always some drawing, e.g. the blinking cursor, the clock on the
> kicker app, etc.

Even so, the X server and graphics card will be idle most of the time. This will
also be true with Xgl, I don't think the distinction between 2D and 3D really
matters either.

If you want to see what things look like when the X server never goes idle, try
running something like x11perf. :)

I'm going to attach a DRM patch that might help narrow things down.

Created an attachment (id=5904)
Fully disable CP writeback

Please rebuild the radeon kernel module with this patch, load it with no_wb=1
and see if that makes any difference. Check with

dmesg|grep writeback

that writeback is actually disabled.

Created an attachment (id=5905)
Xorg log file from the run with the patched radeon kernel driver

Sorry, that it took so long. I had to fight quite a lot of kernel oopses when
unloading the module. Actually unloading wasn't possible at all, so I had to
reboot the system several times.

Just to make sure that I am not doing something wrong, here is my workflow:

- download drm from cvs as described by dri.freedesktop.org/wiki/Download
- goto drm/shared-core
- download your patch, name it wb.pl
- apply it with "patch < wb.pl" -> no errors reported, two files patched
- goto drm/linux-core
- apply "make", no errors
- leave kde, switch to console, enter runlevel 3
- mv old modules from
/lib/modules/2.6.16.16-kanotix-up-1/kernel/drivers/char/drm/ to some other
place
- cp new radeon.ko and drm.ko
- apply "depmod -a"
- unload old modules: "rmmod radeon; rmmod drm" (or just reboot into runlevel
3)
- load new modules: "modprobe radeon no_wb=1"
- enter runlevel 5

dmesg shows:
Jun 14 12:38:33 LaptopMB kernel: [drm] Initialized drm 1.0.1 20051102
Jun 14 12:38:33 LaptopMB kernel: PCI: Unable to reserve mem region
#1:8000000@e0000000 for device 0000:01:00.0
Jun 14 12:38:33 LaptopMB kernel: ACPI: PCI Interrupt 0000:01:00.0[A] -> Link
[LNKA] -> GSI 11 (level, low) -> IRQ 11
Jun 14 12:38:33 LaptopMB kernel: [drm] Initialized radeon 1.25.0 20060524 on
minor 0:
Jun 14 12:38:33 LaptopMB kernel: [drm] Used old pci detect: framebuffer loaded
Jun 14 12:38:58 LaptopMB init: Switching to runlevel: 5
Jun 14 12:38:58 LaptopMB kernel: capifs: Rev 1.1.2.3
Jun 14 12:39:00 LaptopMB kernel: agpgart: Found an AGP 2.0 compliant device at
0000:00:00.0.
Jun 14 12:39:00 LaptopMB kernel: agpgart: Putting AGP V2 device at 0000:00:00.0
into 1x mode
Jun 14 12:39:00 LaptopMB kernel: agpgart: Putting AGP V2 device at 0000:01:00.0
into 1x mode
Jun 14 12:39:00 LaptopMB kernel: [drm] Setting GART location based on new
memory map
Jun 14 12:39:00 LaptopMB kernel: [drm] writeback test succeeded in 2 usecs
Jun 14 12:39:00 LaptopMB kernel: [drm] writeback forced off

cat /proc/acpi/processor/CPU/power:
active state: C2
max_cstate: C8
bus master activity: 32622151
states:
    C1: type[C1] promotion[C2] demotion[--] latency[000]
usage[00000010]
   *C2: type[C2] promotion[C3] demotion[C1] latency[001]
usage[00101866]
    C3: type[C3] promotion[--] demotion[C2] latency[085]
usage[00000000]

When unloading the module, I get a kernel oops:

Jun 14 12:33:31 LaptopMB kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000000
Jun 14 12:33:31 LaptopMB kernel: printing eip:
Jun 14 12:33:31 LaptopMB kernel: f938a58f
Jun 14 12:33:31 LaptopMB kernel: *pde = 00000000
Jun 14 12:33:31 LaptopMB kernel: Oops: 0000 [#1]
Jun 14 12:33:31 LaptopMB kernel: PREEMPT
Jun 14 12:33:31 LaptopMB kernel: Modules linked in: radeon drm arc4
ieee80211_crypt_wep capifs rfcomm l2cap bluetooth therm
al fan button battery ac usblp af_packet xt_tcpudp xt_state ip6table_filter
ip6_tables ipv6 iptable_filter ip_tables x_tabl
es ip_conntrack_tftp ip_conntrack_proto_sctp ip_conntrack_pptp
ip_conntrack_netlink ip_nat ip_conntrack_netbios_ns ip_connt
rack_irc ip_conntrack_ftp ip_conntrack_amanda ...

Your workflow looks good (for extra paranoia, you may want to stick a DRM_INFO
with the added RADEON_WRITEs and verify that it appears in the kernel output),
so this rules out the CP writeback as the source of the problem.

BTW, what's the unit of the latency values in /proc/acpi/processor/CPU0/power?

(In reply to comment #17)
> Your workflow looks good (for extra paranoia, you may want to stick a DRM_INFO
> with the added RADEON_WRITEs and verify that it appears in the kernel output),
> so this rules out the CP writeback as the source of the problem.
>

O.K. I also checked radeon_cp.c that it really contains

       if (radeon_no_wb == 1) {
                dev_priv->writeback_works = 0;
                DRM_INFO("writeback forced off\n");
        }

        if (!dev_priv->writeback_works) {
                /* Disable writeback to avoid unnecessary bus master transfers *
/
                RADEON_WRITE(RADEON_CP_RB_CNTL, RADEON_READ(RADEON_CP_RB_CNTL) |
 RADEON_RB_NO_UPDATE);
                RADEON_WRITE(RADEON_SCRATCH_UMSK, 0);
        }
}

after the patch and it does. Furthermore it seems to be clear that the new
module is loaded as the old one was "1.24.0 20060225". I'm just wondering
because of the kernel oops at unloading...

So as this rules out CP writebacks for increased bus mastering activity, do you
have other ideas?

> BTW, what's the unit of the latency values in /proc/acpi/processor/CPU0/power?

I think they are milliseconds, but don't take that for granted. I'm not sure...

just found this. So it's milliseconds.

----------------------------------
from acpi.sf.net:

Operation Command Sample Output
Read cat power active state: C2
default state: C1
bus master activity: 00000000
states:
   C1: promotion[C2] demotion[--] latency[000] usage[00000450]
  *C2: promotion[--] demotion[C1] latency[020] usage[00039420]
   C3: <not supported>

active state:
This is the sleep state currently used when the system is idle.

default state:
During high system loads and during booting this sleep state is used when the
system is temporarily idle.

bus mastering activity:
When bus mastering control is available (see "info" above), this shows the bus
mastering activity during the last 32 calls of the idling routine. (A bus master
is a device on your system which can access the memory without having to use the
CPU for this.) When the system is quite idle, the ACPI system may want to put
the CPU in the C3 sleep state. In this state the CPU cannot detect if a bus
master changes anything or reads from the system memory. In the case when the
CPU holds copies of this memory in its "write-back cache" or even in its
"inbound" cache, the bus master might read wrong values. This leads to data
corruption, and probably to an instable system, if not a "crash".

states:
*: this is the currently active idling state. This does not mean the system is
currently idle. It only reflects which processor sleep state is called when the
system becomes idle.
C1: the name of this state. (Note: C0 is not mentioned, it is the CPU "working"
state)
promotion[C2]: when the system load is very low, the ACPI system might decide to
switch to this C-State. Of course, this is not available for the highest
supported C-State.
demotion[--]: when the system load is higher, the ACPI system might decide to
swithc to this C-State. Of course, this is not available for the lowest C-State
latency[000]: after an Interrupt Request reaches the CPU, it takes this time in
microseconds until the Interrupt can be processed.
usage[00000450]: the number of times this sleep state has been called.

Interesting. Note that on an AGP system, there should only be AGP (as opposed to
PCI) bus master transfers, and the AGP aperture is mapped uncacheable by the
CPU, so the given reason for bus mastering to prevent C3 doesn't apply
technically. You may want to inquire with the kernel ACPI folks whether there's
a way for these circumstances to be taken into consideration.

done (http://bugzilla.kernel.org/show_bug.cgi?id=6689)

Let's see what they say.

Would be the best solution, if AGP bus mastering can be used extensively to
speed up any graphics drawing, but the CPU still enters C3. :)

Michael

I don't think the chip should do that much bust mastering activity that it would
prevent entering C3 when doing normal almost idle desktop drawing... I suspect
we might be hitting yet another issue here where the chip gets something wrong
with the memory map and thus keeps trying to hit the bus instead of vram or AGP...

(That would explain why we still have lockups every now and then on some
chipsets... random bus mastering is bad !)

Now of only somebody with a PCI analyzer could get us some traces...

(In reply to comment #22)
> I don't think the chip should do that much bust mastering activity that it would
> prevent entering C3 when doing normal almost idle desktop drawing...

Actually, my reading of comment #19 is that bus mastering might quite easily
prevent C3. But there are too many unknowns.

> I suspect we might be hitting yet another issue here where the chip gets
> something wrong with the memory map and thus keeps trying to hit the bus
> instead of vram or AGP...

Possible, but even if that's the case, I'd be surprised if that would account
for significantly more bus mastering activity than the normal CP processing.
Anyway, bug 6939 comes to mind, e.g., might be interesting to try if Option
"RenderAccel" "off" makes a difference, although I'd expect that to affect MMIO
as well.

(In reply to comment #23)
>
> Anyway, bug 6939 comes to mind, e.g., might be interesting to try if Option
> "RenderAccel" "off" makes a difference, although I'd expect that to affect MMIO
> as well.

Option "RenderAccel" "off" doesn't seem to make a difference.

Unfortunately, we don't have an answer yet from the kernel guys to whether AGP
bus mastering should prevent C3 at all.

Sorry about the phenomenal bug spam, guys. Adding xorg-team@ to the QA contact so bugs don't get lost in future.

This may have nothing to do with bus mastering activity at all. The Linux PowerTop utility blames the radeon module for causing a lot of interrupts (60 per second) on my laptop. This appears to prevent the system from entering C3 mode. Presumably this is due to vertical blanking interrupts.

Apparently the Intel graphics driver was recently fixed to enable Vblank interrupts only when somebody was interested and not all of the time. Likely the radeon module has the same problem (actually I've noticed on another machine that mga does this as well, generates 85 interrupts/sec constantly, corresponding to screen refresh rate).

As far as the bus mastering activity detection, the kernel just gets this data from the hardware, it has no control over what kind of transfers are reported as bus mastering or not. However, if the VBlank interrupt processing is resulting in bus master activity, it would explain it.

There are actually two things at play here, one is the vblank interrupts (Jesse just posted a patch on dri-devel to use the HW vblank counters) and the other is the read back of the ring pointer when processing the command ring. Once the vblank is taken care of we can look at how much it would help to adjust how often we read back the pointer.

I think this has been fixed for a while now.