StarPU Handbook
 All Data Structures Files Functions Variables Typedefs Enumerations Enumerator Groups Pages
Performance Feedback

Using The Temanejo Task Debugger

StarPU can connect to Temanejo >= 1.0rc2 (see http://www.hlrs.de/temanejo), to permit nice visual task debugging. To do so, build Temanejo's libayudame.so, install Ayudame.h to e.g. /usr/local/include, apply the tools/patch-ayudame to it to fix C build, re-./configure, make sure that it found it, rebuild StarPU. Run the Temanejo GUI, give it the path to your application, any options you want to pass it, the path to libayudame.so.

Make sure to specify at least the same number of CPUs in the dialog box as your machine has, otherwise an error will happen during execution. Future versions of Temanejo should be able to tell StarPU the number of CPUs to use.

Tag numbers have to be below 4000000000000000000ULL to be usable for Temanejo (so as to distinguish them from tasks).

On-line Performance Feedback

Enabling On-line Performance Monitoring

In order to enable online performance monitoring, the application can call starpu_profiling_status_set() with the parameter STARPU_PROFILING_ENABLE. It is possible to detect whether monitoring is already enabled or not by calling starpu_profiling_status_get(). Enabling monitoring also reinitialize all previously collected feedback. The environment variable STARPU_PROFILING can also be set to 1 to achieve the same effect. The function starpu_profiling_init() can also be called during the execution to reinitialize performance counters and to start the profiling if the environment variable STARPU_PROFILING is set to 1.

Likewise, performance monitoring is stopped by calling starpu_profiling_status_set() with the parameter STARPU_PROFILING_DISABLE. Note that this does not reset the performance counters so that the application may consult them later on.

More details about the performance monitoring API are available in Profiling.

Per-task Feedback

If profiling is enabled, a pointer to a structure starpu_profiling_task_info is put in the field starpu_task::profiling_info when a task terminates. This structure is automatically destroyed when the task structure is destroyed, either automatically or by calling starpu_task_destroy().

The structure starpu_profiling_task_info indicates the date when the task was submitted (starpu_profiling_task_info::submit_time), started (starpu_profiling_task_info::start_time), and terminated (starpu_profiling_task_info::end_time), relative to the initialization of StarPU with starpu_init(). It also specifies the identifier of the worker that has executed the task (starpu_profiling_task_info::workerid). These date are stored as timespec structures which the user may convert into micro-seconds using the helper function starpu_timing_timespec_to_us().

It it worth noting that the application may directly access this structure from the callback executed at the end of the task. The structure starpu_task associated to the callback currently being executed is indeed accessible with the function starpu_task_get_current().

Per-codelet Feedback

The field starpu_codelet::per_worker_stats is an array of counters. The i-th entry of the array is incremented every time a task implementing the codelet is executed on the i-th worker. This array is not reinitialized when profiling is enabled or disabled.

Per-worker Feedback

The second argument returned by the function starpu_profiling_worker_get_info() is a structure starpu_profiling_worker_info that gives statistics about the specified worker. This structure specifies when StarPU started collecting profiling information for that worker (starpu_profiling_worker_info::start_time), the duration of the profiling measurement interval (starpu_profiling_worker_info::total_time), the time spent executing kernels (starpu_profiling_worker_info::executing_time), the time spent sleeping because there is no task to execute at all (starpu_profiling_worker_info::sleeping_time), and the number of tasks that were executed while profiling was enabled. These values give an estimation of the proportion of time spent do real work, and the time spent either sleeping because there are not enough executable tasks or simply wasted in pure StarPU overhead.

Calling starpu_profiling_worker_get_info() resets the profiling information associated to a worker.

When an FxT trace is generated (see Generating Traces With FxT), it is also possible to use the tool starpu_workers_activity (see Monitoring Activity) to generate a graphic showing the evolution of these values during the time, for the different workers.

Bus-related Feedback

TODO: ajouter STARPU_BUS_STATS

The bus speed measured by StarPU can be displayed by using the tool starpu_machine_display, for instance:

StarPU has found:
        3 CUDA devices
                CUDA 0 (Tesla C2050 02:00.0)
                CUDA 1 (Tesla C2050 03:00.0)
                CUDA 2 (Tesla C2050 84:00.0)
from    to RAM          to CUDA 0       to CUDA 1       to CUDA 2
RAM     0.000000        5176.530428     5176.492994     5191.710722
CUDA 0  4523.732446     0.000000        2414.074751     2417.379201
CUDA 1  4523.718152     2414.078822     0.000000        2417.375119
CUDA 2  4534.229519     2417.069025     2417.060863     0.000000

StarPU-Top Interface

StarPU-Top is an interface which remotely displays the on-line state of a StarPU application and permits the user to change parameters on the fly.

Variables to be monitored can be registered by calling the functions starpu_top_add_data_boolean(), starpu_top_add_data_integer(), starpu_top_add_data_float(), e.g.:

starpu_top_data *data = starpu_top_add_data_integer("mynum", 0, 100, 1);

The application should then call starpu_top_init_and_wait() to give its name and wait for StarPU-Top to get a start request from the user. The name is used by StarPU-Top to quickly reload a previously-saved layout of parameter display.

starpu_top_init_and_wait("the application");

The new values can then be provided thanks to starpu_top_update_data_boolean(), starpu_top_update_data_integer(), starpu_top_update_data_float(), e.g.:

Updateable parameters can be registered thanks to starpu_top_register_parameter_boolean(), starpu_top_register_parameter_integer(), starpu_top_register_parameter_float(), e.g.:

float alpha;
starpu_top_register_parameter_float("alpha", &alpha, 0, 10, modif_hook);

modif_hook is a function which will be called when the parameter is being modified, it can for instance print the new value:

void modif_hook(struct starpu_top_param *d) {
fprintf(stderr,"%s has been modified: %f\n", d->name, alpha);
}

Task schedulers should notify StarPU-Top when it has decided when a task will be scheduled, so that it can show it in its Gantt chart, for instance:

starpu_top_task_prevision(task, workerid, begin, end);

Starting StarPU-Top (StarPU-Top is started via the binary starpu_top.) and the application can be done two ways:

Off-line Performance Feedback

Generating Traces With FxT

StarPU can use the FxT library (see https://savannah.nongnu.org/projects/fkt/) to generate traces with a limited runtime overhead.

You can either get a tarball:

$ wget http://download.savannah.gnu.org/releases/fkt/fxt-0.2.11.tar.gz

or use the FxT library from CVS (autotools are required):

$ cvs -d :pserver:anonymous\@cvs.sv.gnu.org:/sources/fkt co FxT
$ ./bootstrap

Compiling and installing the FxT library in the $FXTDIR path is done following the standard procedure:

$ ./configure --prefix=$FXTDIR
$ make
$ make install

In order to have StarPU to generate traces, StarPU should be configured with the option --with-fxt :

$ ./configure --with-fxt=$FXTDIR

Or you can simply point the PKG_CONFIG_PATH to $FXTDIR/lib/pkgconfig and pass --with-fxt to ./configure

When FxT is enabled, a trace is generated when StarPU is terminated by calling starpu_shutdown(). The trace is a binary file whose name has the form prof_file_XXX_YYY where XXX is the user name, and YYY is the pid of the process that used StarPU. This file is saved in the /tmp/ directory by default, or by the directory specified by the environment variable STARPU_FXT_PREFIX.

Creating a Gantt Diagram

When the FxT trace file filename has been generated, it is possible to generate a trace in the Paje format by calling:

$ starpu_fxt_tool -i filename

Or alternatively, setting the environment variable STARPU_GENERATE_TRACE to 1 before application execution will make StarPU do it automatically at application shutdown.

This will create a file paje.trace in the current directory that can be inspected with the ViTE trace visualizing open-source tool. It is possible to open the file paje.trace with ViTE by using the following command:

$ vite paje.trace

To get names of tasks instead of "unknown", fill the optional starpu_codelet::name, or use a performance model for them.

In the MPI execution case, collect the trace files from the MPI nodes, and specify them all on the command starpu_fxt_tool, for instance:

$ starpu_fxt_tool -i filename1 -i filename2

By default, all tasks are displayed using a green color. To display tasks with varying colors, pass option -c to starpu_fxt_tool.

Traces can also be inspected by hand by using the tool fxt_print, for instance:

$ fxt_print -o -f filename

Timings are in nanoseconds (while timings as seen in vite are in milliseconds).

Creating a DAG With Graphviz

When the FxT trace file filename has been generated, it is possible to generate a task graph in the DOT format by calling:

$ starpu_fxt_tool -i filename

This will create a dag.dot file in the current directory. This file is a task graph described using the DOT language. It is possible to get a graphical output of the graph by using the graphviz library:

$ dot -Tpdf dag.dot -o output.pdf

Monitoring Activity

When the FxT trace file filename has been generated, it is possible to generate an activity trace by calling:

$ starpu_fxt_tool -i filename

This will create a file activity.data in the current directory. A profile of the application showing the activity of StarPU during the execution of the program can be generated:

$ starpu_workers_activity activity.data

This will create a file named activity.eps in the current directory. This picture is composed of two parts. The first part shows the activity of the different workers. The green sections indicate which proportion of the time was spent executed kernels on the processing unit. The red sections indicate the proportion of time spent in StartPU: an important overhead may indicate that the granularity may be too low, and that bigger tasks may be appropriate to use the processing unit more efficiently. The black sections indicate that the processing unit was blocked because there was no task to process: this may indicate a lack of parallelism which may be alleviated by creating more tasks when it is possible.

The second part of the picture activity.eps is a graph showing the evolution of the number of tasks available in the system during the execution. Ready tasks are shown in black, and tasks that are submitted but not schedulable yet are shown in grey.

Performance Of Codelets

The performance model of codelets (see Performance Model Example) can be examined by using the tool starpu_perfmodel_display:

$ starpu_perfmodel_display -l
file: <malloc_pinned.hannibal>
file: <starpu_slu_lu_model_21.hannibal>
file: <starpu_slu_lu_model_11.hannibal>
file: <starpu_slu_lu_model_22.hannibal>
file: <starpu_slu_lu_model_12.hannibal>

Here, the codelets of the example lu are available. We can examine the performance of the kernel 22 (in micro-seconds), which is history-based:

$ starpu_perfmodel_display -s starpu_slu_lu_model_22
performance model for cpu
# hash      size       mean          dev           n
57618ab0    19660800   2.851069e+05  1.829369e+04  109
performance model for cuda_0
# hash      size       mean          dev           n
57618ab0    19660800   1.164144e+04  1.556094e+01  315
performance model for cuda_1
# hash      size       mean          dev           n
57618ab0    19660800   1.164271e+04  1.330628e+01  360
performance model for cuda_2
# hash      size       mean          dev           n
57618ab0    19660800   1.166730e+04  3.390395e+02  456

We can see that for the given size, over a sample of a few hundreds of execution, the GPUs are about 20 times faster than the CPUs (numbers are in us). The standard deviation is extremely low for the GPUs, and less than 10% for CPUs.

This tool can also be used for regression-based performance models. It will then display the regression formula, and in the case of non-linear regression, the same performance log as for history-based performance models:

$ starpu_perfmodel_display -s non_linear_memset_regression_based
performance model for cpu_impl_0
	Regression : #sample = 1400
	Linear: y = alpha size ^ beta
		alpha = 1.335973e-03
		beta = 8.024020e-01
	Non-Linear: y = a size ^b + c
		a = 5.429195e-04
		b = 8.654899e-01
		c = 9.009313e-01
# hash		size		mean		stddev		n
a3d3725e	4096           	4.763200e+00   	7.650928e-01   	100
870a30aa	8192           	1.827970e+00   	2.037181e-01   	100
48e988e9	16384          	2.652800e+00   	1.876459e-01   	100
961e65d2	32768          	4.255530e+00   	3.518025e-01   	100
...

The same can also be achieved by using StarPU's library API, see Performance Model and notably the function starpu_perfmodel_load_symbol(). The source code of the tool starpu_perfmodel_display can be a useful example.

The tool starpu_perfmodel_plot can be used to draw performance models. It writes a .gp file in the current directory, to be run with the tool gnuplot, which shows the corresponding curve.

starpu_non_linear_memset_regression_based.png

When the field starpu_task::flops is set, starpu_perfmodel_plot can directly draw a GFlops curve, by simply adding the -f option:

$ starpu_perfmodel_plot -f -s chol_model_11

This will however disable displaying the regression model, for which we can not compute GFlops.

starpu_chol_model_11_type.png

When the FxT trace file filename has been generated, it is possible to get a profiling of each codelet by calling:

$ starpu_fxt_tool -i filename
$ starpu_codelet_profile distrib.data codelet_name

This will create profiling data files, and a .gp file in the current directory, which draws the distribution of codelet time over the application execution, according to data input size.

distrib_data.png

This is also available in the tool starpu_perfmodel_plot, by passing it the fxt trace:

$ starpu_perfmodel_plot -s non_linear_memset_regression_based -i /tmp/prof_file_foo_0

It will produce a .gp file which contains both the performance model curves, and the profiling measurements.

starpu_non_linear_memset_regression_based_2.png

If you have the statistical tool R installed, you can additionally use

$ starpu_codelet_histo_profile distrib.data

Which will create one .pdf file per codelet and per input size, showing a histogram of the codelet execution time distribution.

distrib_data_histo.png

Trace statistics

More than just codelet performance, it is interesting to get statistics over all kinds of StarPU states (allocations, data transfers, etc.). This is particularly useful to check what may have gone wrong in the accurracy of the simgrid simulation.

This requires the R statistical tool, with the plyr, ggplot2 and data.table packages. If your system distribution does not have packages for these, one can fetch them from CRAN:

$ R
> install.packages("plyr")
> install.packages("ggplot2")
> install.packages("data.table")

The pj_dump tool from pajeng is also needed (see https://github.com/schnorr/pajeng)

One can then get textual or .csv statistics over the trace states:

$ starpu_paje_state_stats -v native.trace simgrid.trace
"Value"         "Events_native.csv" "Duration_native.csv" "Events_simgrid.csv" "Duration_simgrid.csv"
"Callback"      220                 0.075978              220                  0
"chol_model_11" 10                  565.176               10                   572.8695
"chol_model_21" 45                  9184.828              45                   9170.719
"chol_model_22" 165                 64712.07              165                  64299.203
$ starpu_paje_state_stats native.trace simgrid.trace

And one can plot histograms of execution times, of several states for instance:

$ starpu_paje_draw_histogram -n chol_model_11,chol_model_21,chol_model_22 native.trace simgrid.trace

and see the resulting pdf file:

paje_draw_histogram.png

Theoretical Lower Bound On Execution Time

StarPU can record a trace of what tasks are needed to complete the application, and then, by using a linear system, provide a theoretical lower bound of the execution time (i.e. with an ideal scheduling).

The computed bound is not really correct when not taking into account dependencies, but for an application which have enough parallelism, it is very near to the bound computed with dependencies enabled (which takes a huge lot more time to compute), and thus provides a good-enough estimation of the ideal execution time.

Theoretical Lower Bound On Execution Time Example provides an example on how to use this.

Memory Feedback

It is possible to enable memory statistics. To do so, you need to pass the option --enable-memory-stats when running configure. It is then possible to call the function starpu_data_display_memory_stats() to display statistics about the current data handles registered within StarPU.

Moreover, statistics will be displayed at the end of the execution on data handles which have not been cleared out. This can be disabled by setting the environment variable STARPU_MEMORY_STATS to 0.

For example, if you do not unregister data at the end of the complex example, you will get something similar to:

$ STARPU_MEMORY_STATS=0 ./examples/interface/complex
Complex[0] = 45.00 + 12.00 i
Complex[0] = 78.00 + 78.00 i
Complex[0] = 45.00 + 12.00 i
Complex[0] = 45.00 + 12.00 i
$ STARPU_MEMORY_STATS=1 ./examples/interface/complex
Complex[0] = 45.00 + 12.00 i
Complex[0] = 78.00 + 78.00 i
Complex[0] = 45.00 + 12.00 i
Complex[0] = 45.00 + 12.00 i

#---------------------
Memory stats:
#-------
Data on Node #3
#-----
Data : 0x553ff40
Size : 16

#--
Data access stats
/!\ Work Underway
Node #0
	Direct access : 4
	Loaded (Owner) : 0
	Loaded (Shared) : 0
	Invalidated (was Owner) : 0

Node #3
	Direct access : 0
	Loaded (Owner) : 0
	Loaded (Shared) : 1
	Invalidated (was Owner) : 0

#-----
Data : 0x5544710
Size : 16

#--
Data access stats
/!\ Work Underway
Node #0
	Direct access : 2
	Loaded (Owner) : 0
	Loaded (Shared) : 1
	Invalidated (was Owner) : 1

Node #3
	Direct access : 0
	Loaded (Owner) : 1
	Loaded (Shared) : 0
	Invalidated (was Owner) : 0

Data Statistics

Different data statistics can be displayed at the end of the execution of the application. To enable them, you need to pass the option --enable-stats when calling configure. When calling starpu_shutdown() various statistics will be displayed, execution, MSI cache statistics, allocation cache statistics, and data transfer statistics. The display can be disabled by setting the environment variable STARPU_STATS to 0.

$ ./examples/cholesky/cholesky_tag
Computation took (in ms)
518.16
Synthetic GFlops : 44.21
#---------------------
MSI cache stats :
TOTAL MSI stats	hit 1622 (66.23 %)	miss 827 (33.77 %)
...
$ STARPU_STATS=0 ./examples/cholesky/cholesky_tag
Computation took (in ms)
518.16
Synthetic GFlops : 44.21