Support ftrace in the linux-keystone kernel

I've enabled all FTRACE function in once but the kernel hangs when booting without any message output. Now I am trying to build and test the kernel with config enabled one by one. These config are safe to enable.

CONFIG_STACK_TRACER, CONFIG_FUNCTION_TRACER, CONFIG_FTRACE

The root cause of kernel hang found. CONFIG_FUNCTION_GRAPH_TRACER shall be disabled.

With most of ftrace enabled, bonnie++ shows we have SSD I/O performance regression.
Sequential read speed from 131K/Sec down to 102K/Sec

--
Kernel version, 3.13.0-17.27 with ftrace enable

ubuntu@ubuntu:~$ bonnie++ -f -n 0
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ubuntu 13G 102888 81 63130 55 244493 85 8985 374
Latency 759ms 662ms 5933us 7319us

1.97,1.97,ubuntu,1,1415025240,13G,,,,102888,81,63130,55,,,244493,85,8985,374,,,,,,,,,,,,,,,,,,,759ms,662ms,,5933us,7319us,,,,,,

--
Kernel version, 3.13.0-17.27

ubuntu@ubuntu:~$ bonnie++ -f -n 0
Writing intelligently...done
Rewriting...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Version 1.97 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ubuntu 13G 131511 97 74773 64 247270 84 8543 373
Latency 103ms 102ms 6456us 7237us

Ubuntu-3.13.0-17.27 stock kernel (armhf native compiler)
Seq block write: 120256
Seq block read: 248465
1.97,1.97,ubuntu,1,1415173504,13G,,,,120256,88,68041,60,,,248465,83,8613,374,,,,,,,,,,,,,,,,,,,592ms,679ms,,5825us,6860us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains)
Seq block write: 125299
Seq block read: 248085
1.97,1.97,ubuntu,1,1415170358,13G,,,,125299,90,68961,59,,,248085,83,8943,368,,,,,,,,,,,,,,,,,,,615ms,650ms,,47026us,6681us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE
Seq block write: 124215
Seq block read: 247361
1.97,1.97,ubuntu,1,1415174007,13G,,,,124215,90,69289,60,,,247361,84,8646,395,,,,,,,,,,,,,,,,,,,645ms,655ms,,6983us,6955us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS
Seq block write: 124133
Seq block read: 249600
1.97,1.97,ubuntu,1,1415172874,13G,,,,124133,91,70943,60,,,249600,82,8950,366,,,,,,,,,,,,,,,,,,,695ms,657ms,,6958us,6886us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT
Seq block write: 127101
Seq block read: 249724
1.97,1.97,ubuntu,1,1415175849,13G,,,,127101,91,69083,59,,,249724,82,8559,375,,,,,,,,,,,,,,,,,,,600ms,655ms,,6435us,6542us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT + CONFIG_BLK_DEV_IO_TRACE + CONFIG_PROBE_EVENTS
Seq block write: 125434
Seq block read: 248367
1.97,1.97,ubuntu,1,1415174904,13G,,,,125434,90,67893,59,,,248367,84,8538,398,,,,,,,,,,,,,,,,,,,606ms,646ms,,7441us,6794us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT + CONFIG_BLK_DEV_IO_TRACE + CONFIG_PROBE_EVENTS + CONFIG_FUNCTION_TRACER
Seq block write: 121617
Seq block read: 244984
1.97,1.97,ubuntu,1,1415177546,13G,,,,121617,88,69699,60,,,244984,85,9117,377,,,,,,,,,,,,,,,,,,,1169ms,653ms,,6126us,6757us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT + CONFIG_BLK_DEV_IO_TRACE + CONFIG_PROBE_EVENTS + CONFIG_FUNCTION_TRACER + CONFIG_FUNCTION_PROFILER
Seq block write: 125562
Seq block read: 245324
1.97,1.97,ubuntu,1,1415177109,13G,,,,125562,91,69791,60,,,245324,84,8768,364,,,,,,,,,,,,,,,,,,,607ms,653ms,,7092us,6782us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT + CONFIG_BLK_DEV_IO_TRACE + CONFIG_PROBE_EVENTS + CONFIG_FUNCTION_TRACER + CONFIG_FUNCTION_PROFILER + CONFIG_STACK_TRACER
Seq block write: 123092
Seq block read: 242909

1.97,1.97,ubuntu,1,1415180590,13G,,,,123092,89,68307,59,,,242909,84,8904,368,,,,,,,,,,,,,,,,,,,686ms,658ms,,7699us,6824us,,,,,,

Ubuntu-3.13.0-17.27 kernel (cross toolchains) + CONFIG_FTRACE + CONFIG_FTRACE_SYSCALLS + CONFIG_SCHED_TRACER + CONFIG_TRACER_SNAPSHOT + CONFIG_BLK_DEV_IO_TRACE + CONFIG_PROBE_EVENTS + CONFIG_FUNCTION_TRACER + CONFIG_FUNCTION_PROFILER + CONFIG_STACK_TRACER + CONFIG_ATH10K_TRACING + CONFIG_IWLWIFI_DEVICE_TRACING + CONFIG_SCSI_IPR_TRACE - CONFIG_CONTEXT_TRACKING_FORCE
Seq block write: 139323
Seq block read: 281052
1.97,1.97,ubuntu,1,1415182781,13G,,,,139323,89,76252,55,,,281052,68,8966,311,,,,,,,,,,,,,,,,,,,602ms,661ms,,5823us,6525us,,,,,,

I will propose this patch for next SRU cycle

I will propose this patch for next SRU cycle

ubuntu@ubuntu:~$ grep TRACER /boot/config-3.13.0-18-keystone
# CONFIG_ATH5K_TRACER is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
# CONFIG_FUNCTION_GRAPH_TRACER is not set
CONFIG_FUNCTION_TRACER=y
CONFIG_GENERIC_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_NOP_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set

This bug was fixed in the package linux-keystone - 3.13.0-18.29

---------------
linux-keystone (3.13.0-18.29) trusty; urgency=low

  [ Ike Panhc ]

  * Release Tracking Bug
    - LP: #1392687
  * Rebase to Ubuntu-3.13.0-40.69

  [ Ubuntu: 3.13.0-40.69 ]

  * Release Tracking Bug
    - re-used previous tracking bug
  * regmap: fix kernel hang on regmap_bulk_write with zero val_count.
 -- Ike Panhc <email address hidden> Fri, 14 Nov 2014 20:28:11 +0800