Enable lowlatency settings in the generic kernel

I found this bug report by accident, while searching for something else.
I pretty much only use mainline kernels and only 1000 Hertz.
I support this proposed default Ubuntu kernel configuration change.

The tick ISR is incredibly efficient (less than 2 uSec on my test system), and I do not understand your test results as I would have expected a lot less difference. Using mainline kernel 6.8-rc1 and limiting my test system max CPU frequency so that I get similar bogo op/s as you I get:

 - CONFIG_HZ=250 : 14853.14 bogo ops/s
 - CONFIG_HZ=1000 : 14714.01 bogo ops/s (0.94% worse)
 - CONFIG_HZ=1000+nohz_full : 15100 bogo ops/s (1.6% better)

There is no power or thermal or active cores throttling on my test system.
Note: with my CPU frequency limited for this test, the tick ISR takes about 4 uSec.
The difference between the 250 and 1000 Hz kernel tests is 750 tick interrupts per second or 3 milliseconds or about 0.3%

If I do not limit my max CPU frequency I get:

 - CONFIG_HZ=250 : 38518.64 bogo ops/s
 - CONFIG_HZ=1000 : 37765.55 bogo ops/s (2.0% worse)
 - CONFIG_HZ=1000+nohz_full : 39391.32 bogo ops/s (2.3% better)

There was no power or thermal or active cores throttling for this test. However I did have to raise my processor max temperature limit from 75 to 80 degrees C for this test. Also the power is very close to the limit at 122 watts, where it will throttle at 125 watts.

Before my post yesterday, I had never used the stress-ng utility, and only did so to repeat the originally posted test case. However, there was run to run variability with the stress-ng test that I could not understand for a 100% user, 0% system, type program. I decided to retest using one my own CPU loading programs. I also did only 1 thread and forced CPU affinity.

I also tried to get a more accurate estimate of the tick ISR execution time. Trace only does time stamping to the microsecond, making it difficult. For the 1000Hz kernel and a 100 second trace, 100000 tick ISRs were captured. min 0, average 0.7, max 2 uSec.

250Hz kernel test time (less is better): 300.14 seconds (100% user 0% system).

For the 1000 Hertz kernel the extra execution time prediction is:
0.7 uSec/tick * 750 extra ticks/sec * 300.14 sec = 0.16 sec
For a predicted execution time of 300.30 seconds.

1000Hz kernel test time: 300.32 seconds.

1000Hz+nohz_full test time: 300.08 seconds.

In an attempt to get a better model the processor CPU frequency was changed from 4.8 GHz to 0.80 GHz.
Tick ISR: min 3, average 4.0, max 11 uSec.
Note: of the 100000 tick ISRs captured the 11 uSec one was a one time outlier.

250Hz kernel test time: 429.25

For the 1000 Hertz kernel the extra execution time prediction is:
4.0 uSec/tick * 750 extra ticks/sec * 429.25 sec = 1.29 sec
For a predicted execution time of 430.53 seconds.

1000Hz kernel test time: 430.38 seconds.

It may be worth trying a wider range of synthetic benchmarks to see how it affects scheduling, I/O, RCU and power consumption.

@dsmythies thank you so much for sharing the results of your tests, really useful info!

I'm planning to do more tests setting the performance governor, I've been doing my initial tests only with the default Ubuntu settings, that means with the "Balanced" power mode enabled (that I think it's using intel p-states), so that may have affected my results.

I'm also curious to measure the power consumption of 250 vs 1000, since I expect to see a little extra power consumption with HZ=1000. But then I also want to enable the lazy RCU (boot with `rcu_nocbs=all rcutree.enable_rcu_lazy=1`) and see how much power we can save vs the impact on performance.

Overall, the point that I want to prove is that, yes, we may have small regressions here and there, but with these changes in place we can provide a huge flexibility for users, that will able to tune their system for improved responsiveness, CPU throughput, or power consumption, all using the default stock kernel.

@colin-king thanks! Any suggestion in particular? I was thinking to lmbench, netperf and fio.

@Andrea, that's a good start, but it may be worth running some of the Phoronix Tests too as they are a good spread of use cases.

Looks like Michael Larabel has done some analysis for you already :-) https://www.phoronix.com/news/Ubuntu-Generic-LL-Kernel

@colin-king noticed, I just left a thank you message in the article. I'll still do the tests, but it's nice to see that someone else is contributing to this!

Another thing that I'd like to measure is to bpftrace the time spent in the tick handler before and after these changes applied, because we call the tick handler more often with HZ=1000, so it'd be nice to see how much is the distribution of such overhead.

And another cool thing that I've been told is that HZ=1000 can actually help in terms of power consumption, because apparently the CPUs have more chances to enter the idle states more quickly. This is also something that I'd like to measure.

I couldn't find this in the benchmark description on Phoronix, so I'm assuming the lowlatency kernel was booted with default parameters. Which means CONFIG_NO_HZ_FULL basically had no effect. This is probably fair for most users who won't specify `nohz_full` kernel parameter and will observe the performance difference between generic and lowlatency as shown by Phoronix tests.

But the claim in this ticket is that `nohz_full` can potentially win back some performance losses caused by CONFIG_HZ=1000. It would be useful if testing could confirm or disprove that. Ideally, Phoronix Test Suite would need to be run on generic, lowlatency without any extra kernel parameters and lowlatency with `nohz_full` parameter.

@andysem correct, without `nohz_full` specified at boot CONFIG_NO_HZ_FULL has no effect, except for the little extra overhead that it's adding to the tick handler (there is still some overhead with this option enabled, even if it's not used). That's why I'd like to measure the time spent in some of the tick callbacks (using bpftrace) with generic vs lowlatency, to have a better understanding of this extra overhead.

Then, yes, repeating the tests disabling the ticks on some CPUs (booting with nohz_full) would be another interesting metric.

For the Stress-NG 0.16.04: pts/stress-ng-1.11.0 [Test: Socket Activity] I get:

6633.31 Bogo Ops/s on a 1000Hz kernel. 0.9% improvement.
6572.92 Bogo Ops/s on a 250 Hz kernel.

I did this in a hurry, and will re-test tonight or tomorrow.

While testing locally, I have found this problem: https://bugs.launchpad.net/ubuntu/+source/linux-signed-lowlatency-hwe-6.5/+bug/2051733

If this is indeed a valid bug and not something weirdly specific to my system, I'd say `nohz_full` is non-functional.

The generic Ubuntu kernel has dynamic preempt enabled. This allows the preemption model to be changed at runtime between: none, voluntary (the default), and full.

It would be super interesting to test what impact this has for latency sensitive workloads, and whether this can help you make the lowlatency kernel build redundant.

Related discussion on preemption model for Fedora: https://pagure.io/fedora-workstation/issue/228

I redid the Phoronix Stress-NG 0.16.04: Socket Activity test, the one that showed such a dramatic difference in their test. I increased the number of test runs from 3 to 10 and time per run from 30 to 60 seconds.

I got:
250Hz kernel (generic): 6608.74 Bogo Op/s, Deviation 0.56%
1000Hz kernel (lowlatency): 6591.27 BOgo Op/s, Deviation 0.39%, 0.3% performance degradation
My kernel was mainline 6.8-rc1 using Ubuntu kernel configurations.

I posted about this on the Phoronix thread.

By the way, for readers that do not have the Phoronix stuff installed, I think the equivalent direct stress-ng command is:

stress-ng --sock -1 --no-rand-seed --sock-zerocopy --timeout 2m --metrics-brief

@arighi: Did you consider applying lowlatency settings to generic kernel but keeping CONFIG_HZ=250?

I suspect, the majority of desktop usage issues (i.e. responsiveness) would be solved by `preempt=full rcu_nocbs=all` being the default (or whatever the equivalent boot options are). While keeping CONFIG_HZ=250 would mitigate the potential throughput loss.

@andysem yes that is also another possibility. The idea is to do a lot of tests and if we find that there's a remote possibility to introduce significant performance regressions in certain cases we can still keep HZ=250, but definitely go with the other options.

More tests and results available here: https://discourse.ubuntu.com/t/enable-low-latency-features-in-the-generic-ubuntu-kernel-for-24-04/42255

I haven’t spotted any tests that show a workload with a significant improvement shown here, on discourse, or Phoronix. Am I incorrect? Please keep in mind that I’m not suggesting that this initiative not move forward as there appears to be plenty of evidence being built up suggesting that there is no significant regressions.

@turner basically any type of I/O workload can benefit from the HZ=1000 change, I haven't posted any test result, because in the scope of this proposal I wanted to focus mainly at the regression potential for CPU-intensive workloads and the benefits in terms of power consumption (this one as a positive side effect).

I will also post some numbers to better highlight the benefits of the low latency features.

This bug was fixed in the package linux - 6.8.0-20.20

---------------
linux (6.8.0-20.20) noble; urgency=medium

  * noble/linux: 6.8.0-20.20 -proposed tracker (LP: #2058221)

  * Noble update: v6.8.1 upstream stable release (LP: #2058224)
    - x86/mmio: Disable KVM mitigation when X86_FEATURE_CLEAR_CPU_BUF is set
    - Documentation/hw-vuln: Add documentation for RFDS
    - x86/rfds: Mitigate Register File Data Sampling (RFDS)
    - KVM/x86: Export RFDS_NO and RFDS_CLEAR to guests
    - Linux 6.8.1

  * Autopkgtest failures on amd64 (LP: #2048768)
    - [Packaging] update to clang-18

  * Miscellaneous Ubuntu changes
    - SAUCE: apparmor4.0.0: LSM stacking v39: fix build error with
      CONFIG_SECURITY=n
    - [Config] amd64: MITIGATION_RFDS=y

 -- Paolo Pisati <email address hidden> Mon, 18 Mar 2024 11:08:14 +0100