StarPU Handbook
|
To get an idea of what is happening, a lot of performance feedback is available, detailed in this chapter. The various information should be checked for.
You can also use the Temanejo task debugger (see Using The Temanejo Task Debugger) to visualize the task graph more easily.
StarPU can use the FxT library (see https://savannah.nongnu.org/projects/fkt/) to generate traces with a limited runtime overhead.
You can get a tarball from http://download.savannah.gnu.org/releases/fkt/?C=M
Compiling and installing the FxT library in the $FXTDIR
path is done following the standard procedure:
$ ./configure --prefix=$FXTDIR $ make $ make install
In order to have StarPU to generate traces, StarPU needs to be configured again after installing FxT, and configuration show:
FxT trace enabled: yes
If configure
does not find FxT automatically, it can be specified by hand with the option --with-fxt :
$ ./configure --with-fxt=$FXTDIR
Or you can simply point the PKG_CONFIG_PATH
environment variable to $FXTDIR/lib/pkgconfig
When STARPU_FXT_TRACE is set to 1, a trace is generated when StarPU is terminated by calling starpu_shutdown(). The trace is a binary file whose name has the form prof_file_XXX_YYY
where XXX
is the username, and YYY
is the MPI id of the process that used StarPU (or 0 when running a sequential program). One can change the name of the file by setting the environment variable STARPU_FXT_SUFFIX, its contents will be used instead of prof_file_XXX
. This file is saved in the /tmp/
directory by default, or by the directory specified by the environment variable STARPU_FXT_PREFIX.
The additional configure
option --enable-fxt-lock can be used to generate trace events which describes the lock's behavior during the execution. It is however very heavy and should not be used unless debugging StarPU's internal locking.
When the FxT trace file prof_file_something
has been generated, it is possible to generate different trace formats by calling:
$ starpu_fxt_tool -i /tmp/prof_file_something
Or alternatively, setting the environment variable STARPU_GENERATE_TRACE to 1
before application execution will make StarPU automatically generate all traces at application shutdown. Note that if the environment variable STARPU_FXT_PREFIX is set, files will be generated in the given directory.
One can also set the environment variable STARPU_GENERATE_TRACE_OPTIONS to specify options, see starpu_fxt_tool –help
, for example:
$ export STARPU_GENERATE_TRACE=1 $ export STARPU_GENERATE_TRACE_OPTIONS="-no-acquire"
When running an MPI application, STARPU_GENERATE_TRACE will not work as expected (each node will try to generate trace files, thus mixing outputs...), you have to collect the trace files from the MPI nodes, and specify them all on the command starpu_fxt_tool
, for instance:
$ starpu_fxt_tool -i /tmp/prof_file_something*
By default, the generated trace contains all information. To reduce the trace size, various -no-foo
options can be passed to starpu_fxt_tool
, see starpu_fxt_tool –help
.
One of the generated files is a trace in the Paje format. The file, located in the current directory, is named paje.trace
. It can be viewed with ViTE (https://solverstack.gitlabpages.inria.fr/vite/) a trace visualizing open-source tool. To open the file paje.trace
with ViTE, use the following command:
$ vite paje.trace
Once the file is opened in ViTE interface, we will see the figure as shown below:
We can then click the "No arrows" button in task bar of ViTE interface, to better observe the Gantt diagram that illustrates the start and end dates of the different tasks or activities of a program.
In the Gantt diagram, the bar types such as devices (CPU or GPU) are displayed on the left side. Each task is represented by a horizontal rectangle that spans the duration of the task. The rectangles are arranged along a timeline axis, which is shown at the top of the Gantt diagram and represents the overall duration of the program in milliseconds. The position of the bar along the timeline shows when the task begins and ends. We can see some long red bars at the beginning and end of the entire timeline, which represent that the unit is idle. There are no tasks at these moments, and workers are waiting or in a sleeping state.
Then as shown in the following figure, press and hold the left mouse button to select the area you want to zoom in on. Release the button to view the selected area, and we can repeat the zoom action multiple times.
This zoom result is:
Right-clicking anywhere on the Gantt diagram restores the previous zoom view.
One can press and hold the left mouse button inside the top blue bar to select horizontally, which will horizontally zoom in on all Gantt diagrams within the selected time range.
This zoom result is:
After zooming in, we can observe numerous blocks of varying colors, each block representing a task. Blocks of diverse colors signify different types of tasks. When we double-click on any block, a pop-up window will show related status about that task, such as its type and which worker (CPU/GPU) it belongs to, etc.
The state information displayed in the pop-up window can be:
unknown
) by filling the optional starpu_codelet::name, or assigning it a performance model. The name can also be set with the field starpu_task::name or by using STARPU_NAME when calling starpu_task_insert() TASK_VERBOSE_EXTRA
, otherwise it will be -1 -c
to starpu_fxt_tool
or in STARPU_GENERATE_TRACE_OPTIONS. Tasks can also be given a specific color by setting the field starpu_codelet::color or the starpu_task::color. When we call starpu_task_insert(), we can use STARPU_TASK_COLOR to set the color. Colors are expressed with the following format 0xRRGGBB
(e.g. 0xFF0000
for red). See basic_examples/task_insert_color
for examples on how to assign colorsIn the shown figure, the set of color as following:
To modify the colors in Vite interface, select "Preferences" then "Settings" in the options bar, and then choose the "States" tab in the newly opened window to select different colors for different operations, as shown in the figure below. One has to click the reload button at the top left to reload the trace with the new colors.
We can see that there is a curve below task blocks, which represents the corresponding GFlop/s
. Double-clicking near the curve will display the current GFlop/s
information in a pop-up window (as shown in the figure). If we only click on the curve, a vertical red line shows up, and we can read on it the GFlop/s
values of all the curves at the same time.
For GPUs, there are three additional curves above the task blocks that can be double-clicked to open a pop-up window to view information. Let's zoom in on the three curves during the entire execution process as illustrated in the figure:
As shown in the figure below, the top curve represents the amount of GPU-managed memory in MBytes, while the bottom two curves represent the data transfer between tasks on the CPU and GPU, and between tasks on different GPUs. They respectively indicate the incoming and outgoing data transfer bandwidth. By looking at the memory curve, we can observe that the memory usage kept increasing at first, but due to the reutilization of the allocations by StarPU, the curve gradually became stable later on.
Above these three curves, we can see some blocks which represent driver copy (see the top of the figure below), i.e. a memory copy. The light green blocks represent the actual copies, the dark green blocks represent asynchronous copy submissions, and the burgundy blocks represent allocating and freeing. Double-clicking on a block allows us to view relevant information in the pop-up window.
Here, a couple of issues may show up:
Below the GPU task blocks and GFlops
curve (see the bottom of the figure above), we can see some other blocks that represent the CPU waiting for the GPU to complete the task. During time, CPU can do variable actions which are represented by blocks of different colors, such as:
progressing
, it keeps polling for task or data transfer completion scheduling
submitting task
executing
, it is executing the application codelet function. Here it is very short because the codelet just submits a kernel asynchronously. callback
overhead
. This state is not supposed to be long, as it represents everything that we did not classify as an operation that is supposed to be long like the operations mentioned above. If you find situations where some overhead is long, this is a bug worth reporting so we can fix it.and we can always double-click on the block to view relevant information in the pop-up window.
We can horizontally zoom in on a section of the Gantt diagram, and deselect the "No arrows" option. This will allow us to see a complete process of data transfer, as shown in the following figure:
In the above figure, we can see a long segment of magenta color in CUDA2_0 task blocks. At the same time, we can see that there are numerous transfers between other workers during this time period. This indicates that CUDA2_0 is waiting for the completion of the data transfers needed by the task it wants to execute.
At the top of the entire Gantt diagram, there are three curves that represent the information of the scheduler. Let's zoom in on the three curves during the entire execution process as illustrated in the figure below:
As shown in the figure below, from top to bottom, they respectively indicate the number of submitted uncompleted tasks, the number of ready tasks, and the total GFlop/s
for this moment. By double-clicking on the curves, we can view relevant information in the pop-up window.
At the very bottom of the entire Gantt diagram, we will see a red bar, which represents the main thread waiting for tasks. In front of the red bar (see the figure below), there are some dark red bars, which represent the main thread submitting tasks.
Below these red bars, we can see some white vertical lines with small circles on top, which represent events. The default events can be either task push or task pop or task wait for all. The application can inject its own events at any desired moment with the function starpu_fxt_trace_user_event() or starpu_fxt_trace_user_event_string(). Similarly, double-clicking on the white bars allows you to see relevant information in the pop-up window.
To get statistics on the time spent in runtime overhead, we can use the statistics plugin of ViTE. In the Preferences menu, select Plugins. In "States Type", select "Worker State". Then click on "Reload" to update the histogram. The red "Idle" percentages are due to lack of parallelism, the "FetchingInput" percentages are due to waiting for data transfers. The brown "Overhead" and "Scheduling" percentages are due to the overhead of the runtime and of the scheduler.
Another generated trace file is a task graph described using the DOT language. The file, created in the current directory, is named dag.dot
file in the current directory. It is possible to get a graphical output of the graph by using the graphviz
library:
$ dot -Tpdf dag.dot -o output.pdf
Another generated trace file gives details on the executed tasks. The file, created in the current directory, is named tasks.rec
. This file is in the recutils
format, i.e. Field: value
lines, and empty lines are used to separate each task. This can be used as a convenient input for various ad-hoc analysis tools. By default, it only contains information about the actual execution. Performance models can be obtained by running starpu_tasks_rec_complete
on it:
$ starpu_tasks_rec_complete tasks.rec tasks2.rec
which will add EstimatedTime
lines which contain the performance model-estimated time (in µs) for each worker starting from 0. Since it needs the performance models, it needs to be run the same way as the application execution, or at least with STARPU_HOSTNAME
set to the hostname of the machine used for execution, to get the performance models of that machine.
Another possibility is to obtain the performance models as an auxiliary perfmodel.rec
file, by using the starpu_perfmodel_recdump
utility:
$ starpu_perfmodel_recdump tasks.rec -o perfmodel.rec
One can also simply call starpu_task_get_name() to get the name of a task.
The file, sched_tasks.rec
, created in the current directory, in the recutils
format, gives information about the tasks scheduling, and lists the push and pop actions of the scheduler. For each action, it gives the timestamp, the job priority and the job id. Each action is separated from the next one by empty lines. The job id associated with the task can be retrieved by calling starpu_task_get_job_id().
Another generated trace file is an activity trace. The file, created in the current directory, is named activity.data
. A profile of the application showing the activity of StarPU during the execution of the program can be generated:
$ starpu_workers_activity activity.data
This will create a file named activity.eps
in the current directory. This picture is composed of two parts. The first part shows the activity of the different workers. The green sections indicate which proportion of the time was spent executed kernels on the processing unit. The red sections indicate the proportion of time spent in StarPU: an important overhead may indicate that the granularity may be too low, and that bigger tasks may be appropriate to use the processing unit more efficiently. The black sections indicate that the processing unit was blocked because there was no task to process: this may indicate a lack of parallelism, which may be alleviated by creating more tasks when it is possible.
The second part of the picture activity.eps
is a graph showing the evolution of the number of tasks available in the system during the execution. Ready tasks are shown in black, and tasks that are submitted but not schedulable yet are shown in grey.
When using modular schedulers (i.e. schedulers which use a modular architecture, and whose name start with "modular-"), the call to starpu_fxt_tool
will also produce a trace.html
file which can be viewed in a javascript-enabled web browser. It shows the flow of tasks between the components of the modular scheduler.
starpu_fxt_tool
produces a file called comms.rec
which describes all MPI communications. The script starpu_send_recv_data_use.py
uses this file and tasks.rec
in order to produce two graphs: the first one shows durations between the reception of data and their usage by a task and the second one plots the same graph but with elapsed time between send and usage of a data by the sender.
When launched with the option -number-events
, starpu_fxt_tool
will produce a file named number_events.data
. This file contains the number of events for each event type. Events are represented with their key. To convert event keys to event names, you can use the starpu_fxt_number_events_to_names.py
script:
$ starpu_fxt_number_events_to_names.py number_events.data
The number of recorded events (and thus the performance overhead introduced by tracing) can be reduced by setting which categories of events to record with the environment variable STARPU_FXT_EVENTS.
For computing statistics, it is useful to limit the trace to a given portion of the time of the whole execution. This can be achieved by calling
before calling starpu_init(), to prevent tracing from starting immediately. Then
and
can be used around the portion of code to be traced. This will show up as marks in the trace, and states of workers will only show up for that portion.
After calibrating performance models of codelets (see Performance Model Example and Performance Model Calibration), they can be examined by using the tool starpu_perfmodel_display
:
$ starpu_perfmodel_display -l file: <malloc_pinned.hannibal> file: <starpu_slu_lu_model_trsm_ru.hannibal> file: <starpu_slu_lu_model_getrf.hannibal> file: <starpu_slu_lu_model_gemm.hannibal> file: <starpu_slu_lu_model_trsm_ll.hannibal>
Here, the codelets of the example lu
are available. We can examine the performance of the kernel 22
(in micro-seconds), which is history-based:
$ starpu_perfmodel_display -s starpu_slu_lu_model_gemm performance model for cpu # hash size mean dev n 57618ab0 19660800 2.851069e+05 1.829369e+04 109 performance model for cuda_0 # hash size mean dev n 57618ab0 19660800 1.164144e+04 1.556094e+01 315 performance model for cuda_1 # hash size mean dev n 57618ab0 19660800 1.164271e+04 1.330628e+01 360 performance model for cuda_2 # hash size mean dev n 57618ab0 19660800 1.166730e+04 3.390395e+02 456
We can see that for the given size, over a sample of a few hundreds of execution, the GPUs are about 20 times faster than the CPUs (numbers are in us). The standard deviation is extremely low for the GPUs, and less than 10% for CPUs.
This tool can also be used for regression-based performance models. It will then display the regression formula, and in the case of non-linear regression, the same performance log as for history-based performance models:
$ starpu_perfmodel_display -s non_linear_memset_regression_based performance model for cpu_impl_0 Regression : #sample = 1400 Linear: y = alpha size ^ beta alpha = 1.335973e-03 beta = 8.024020e-01 Non-Linear: y = a size ^b + c a = 5.429195e-04 b = 8.654899e-01 c = 9.009313e-01 # hash size mean stddev n a3d3725e 4096 4.763200e+00 7.650928e-01 100 870a30aa 8192 1.827970e+00 2.037181e-01 100 48e988e9 16384 2.652800e+00 1.876459e-01 100 961e65d2 32768 4.255530e+00 3.518025e-01 100 ...
The same can also be achieved by using StarPU's library API, see Performance Model and notably the function starpu_perfmodel_load_symbol(). The source code of the tool starpu_perfmodel_display
can be a useful example.
An XML output can also be printed by using the -x
option:
$ tools/starpu_perfmodel_display -x -s non_linear_memset_regression_based <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE StarPUPerfmodel SYSTEM "starpu-perfmodel.dtd"> <!-- symbol non_linear_memset_regression_based --> <!-- All times in us --> <perfmodel version="45"> <combination> <device type="CPU" id="0" ncores="1"/> <implementation id="0"> <!-- cpu0_impl0 (Comb0) --> <!-- time = a size ^b + c --> <nl_regression a="5.429195e-04" b="8.654899e-01" c="9.009313e-01"/> <entry footprint="a3d3725e" size="4096" flops="0.000000e+00" mean="4.763200e+00" deviation="7.650928e-01" nsample="100"/> <entry footprint="870a30aa" size="8192" flops="0.000000e+00" mean="1.827970e+00" deviation="2.037181e-01" nsample="100"/> <entry footprint="48e988e9" size="16384" flops="0.000000e+00" mean="2.652800e+00" deviation="1.876459e-01" nsample="100"/> <entry footprint="961e65d2" size="32768" flops="0.000000e+00" mean="4.255530e+00" deviation="3.518025e-01" nsample="100"/> </implementation> </combination> </perfmodel>
The tool starpu_perfmodel_plot
can be used to draw performance models. It writes a .gp
file in the current directory, to be run with the tool gnuplot
, which shows the corresponding curve.
$ tools/starpu_perfmodel_plot -s non_linear_memset_regression_based $ gnuplot starpu_non_linear_memset_regression_based.gp $ gv starpu_non_linear_memset_regression_based.png
When the field starpu_task::flops is set (or STARPU_FLOPS is passed to starpu_task_insert()), starpu_perfmodel_plot
can directly draw a GFlops/s curve, by simply adding the -f
option:
$ starpu_perfmodel_plot -f -s chol_model_potrf
This will however disable displaying the regression model, for which we can not compute GFlops/s.
When the FxT trace file prof_file_something
has been generated, it is possible to get a profiling of each codelet by calling:
$ starpu_fxt_tool -i /tmp/prof_file_something $ starpu_codelet_profile distrib.data codelet_name
This will create profiling data files, and a distrib.data.gp
file in the current directory, which draws the distribution of codelet time over the application execution, according to data input size.
This is also available in the tool starpu_perfmodel_plot
, by passing it the fxt trace:
$ starpu_perfmodel_plot -s non_linear_memset_regression_based -i /tmp/prof_file_foo_0
It will produce a .gp
file which contains both the performance model curves, and the profiling measurements.
If you have the statistical tool R
installed, you can additionally use
$ starpu_codelet_histo_profile distrib.data
Which will create one .pdf
file per codelet and per input size, showing a histogram of the codelet execution time distribution.
A performance model of the energy of codelets can also be recorded thanks to the starpu_codelet::energy_model field of the starpu_codelet structure. StarPU usually cannot record this automatically, since the energy measurement probes are usually not fine-grain enough. It is however possible to measure it by writing a program that submits batches of tasks, let StarPU measure the energy requirement of the batch, and compute an average, see Measuring energy and power with StarPU .
The energy performance model can then be displayed in Joules with starpu_perfmodel_display
just like the time performance model. The starpu_perfmodel_plot
needs an extra -e
option to display the proper unit in the graph:
$ tools/starpu_perfmodel_plot -e -s non_linear_memset_regression_based_energy $ gnuplot starpu_non_linear_memset_regression_based_energy.gp $ gv starpu_non_linear_memset_regression_based_energy.png
The -f
option can also be used to display the performance in terms of GFlops/s/W, i.e. the efficiency:
$ tools/starpu_perfmodel_plot -f -e -s non_linear_memset_regression_based_energy $ gnuplot starpu_gflops_non_linear_memset_regression_based_energy.gp $ gv starpu_gflops_non_linear_memset_regression_based_energy.png
We clearly see here that it is much more energy-efficient to stay in the L3 cache.
One can combine the two time and energy performance models to draw Watts:
$ tools/starpu_perfmodel_plot -se non_linear_memset_regression_based non_linear_memset_regression_based_energy $ gnuplot starpu_power_non_linear_memset_regression_based.gp $ gv starpu_power_non_linear_memset_regression_based.eps
It is possible to get statistics about tasks length and data size by using :
$ starpu_fxt_data_trace filename [codelet1 codelet2 ... codeletn]
Where filename is the FxT trace file and codeletX the names of the codelets you want to profile (if no names are specified, starpu_fxt_data_trace
will profile them all). This will create a file, data_trace.gp
which can be executed to get a .eps
image of these results. On the image, each point represents a task, and each color corresponds to a codelet.
More than just codelet performance, it is interesting to get statistics over all kinds of StarPU states (allocations, data transfers, etc.). This is particularly useful to check what may have gone wrong in the accuracy of the SimGrid simulation.
This requires the R
statistical tool, with the plyr
, ggplot2
and data.table
packages. If your system distribution does not have packages for these, one can fetch them from CRAN
:
$ R > install.packages("plyr") > install.packages("ggplot2") > install.packages("data.table") > install.packages("knitr")
The pj_dump
tool from pajeng
is also needed (see https://github.com/schnorr/pajeng)
One can then get textual or .csv
statistics over the trace states:
$ starpu_paje_state_stats -v native.trace simgrid.trace "Value" "Events_native.csv" "Duration_native.csv" "Events_simgrid.csv" "Duration_simgrid.csv" "Callback" 220 0.075978 220 0 "chol_model_potrf" 10 565.176 10 572.8695 "chol_model_trsm" 45 9184.828 45 9170.719 "chol_model_gemm" 165 64712.07 165 64299.203 $ starpu_paje_state_stats native.trace simgrid.trace
An other way to get statistics of StarPU states (without installing R
and pj_dump
) is to use the starpu_trace_state_stats.py
script, which parses the generated trace.rec
file instead of the paje.trace
file. The output is similar to the previous script, but it doesn't need any dependencies.
The different prefixes used in trace.rec
are:
E: Event type N: Event name C: Event category W: Worker ID T: Thread ID S: Start time
Here's an example on how to use it:
$ starpu_trace_state_stats.py trace.rec | column -t -s "," "Name" "Count" "Type" "Duration" "Callback" 220 Runtime 0.075978 "chol_model_potrf" 10 Task 565.176 "chol_model_trsm" 45 Task 9184.828 "chol_model_gemm" 165 Task 64712.07
starpu_trace_state_stats.py
can also be used to compute the different efficiencies. Refer to the usage description to show some examples.
And one can plot histograms of execution times, of several states, for instance:
$ starpu_paje_draw_histogram -n chol_model_potrf,chol_model_trsm,chol_model_gemm native.trace simgrid.trace
and see the resulting pdf file:
A quick statistical report can be generated by using:
$ starpu_paje_summary native.trace simgrid.trace
it includes gantt charts, execution summaries, as well as state duration charts and time distribution histograms.
Other external Paje analysis tools can be used on these traces, one just needs to sort the traces by timestamp order (which not guaranteed to make recording more efficient):
$ starpu_paje_sort paje.trace
Performance counter values could be obtained from the PAPI framework if ./configure
detected the libpapi.
In Debian, the libpapi-dev
package provides the required files. Additionally, the papi-tools
package contains a set of useful tools, for example papi_avail
to see which counters are available.
To be able to use Papi counters, one may need to reduce the level of the kernel parameter kernel.perf_event_paranoid
to 2 or below. See https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html for the security impact of this parameter.
Then one has to set the STARPU_PROFILING environment variable to 1 and specify which events to record with the STARPU_PROF_PAPI_EVENTS environment variable. For instance:
export STARPU_PROFILING=1 STARPU_PROF_PAPI_EVENTS="PAPI_TOT_INS PAPI_TOT_CYC"
The comma can also be used to separate events to monitor.
In the current simple implementation, only CPU tasks have their events measured and require CPUs that support the PAPI events. It is important to note that not all events are available on all systems, and general PAPI recommendations should be followed.
The counter values can be accessed using the profiling interface:
Also, it can be accessed and/or saved with tracing when using STARPU_FXT_TRACE. With the use of starpu_fxt_tool
the file papi.rec
is generated containing the following triple:
Task Id Event Id Value
External tools like rec2csv
can be used to convert this rec file to a csv
file, where each line represents a value for an event for a task.
StarPU can record a trace of what tasks are needed to complete the application, and then, by using a linear system, provide a theoretical lower bound of the execution time (i.e. with an ideal scheduling).
The computed bound is not really correct when not taking into account dependencies, but for an application which have enough parallelism, it is very near to the bound computed with dependencies enabled (which takes a huge lot more time to compute), and thus provides a good-enough estimation of the ideal execution time.
Then there is an example to show how to use this.
For kernels with history-based performance models (and provided that they are completely calibrated), StarPU can very easily provide a theoretical lower bound for the execution time of a whole set of tasks. See for instance examples/lu/lu_example.c
: before submitting tasks, call the function starpu_bound_start(), and after complete execution, call starpu_bound_stop(). starpu_bound_print_lp() or starpu_bound_print_mps() can then be used to output a Linear Programming problem corresponding to the schedule of your tasks. Or starpu_bound_print_dot() can be used to print a task dependency graph in the DOT format. Run it through lp_solve
or any other linear programming solver, and that will give you a lower bound for the total execution time of your tasks. If StarPU was compiled with the library glpk
installed, starpu_bound_compute() can be used to solve it immediately and get the optimized minimum, in ms. Its parameter integer
allows deciding whether integer resolution should be computed and returned. Besides to solve it immediately and get the optimized minimum starpu_bound_print() can also print the statistics of actual execution and theoretical upper bound.
The deps
parameter tells StarPU whether to take tasks, implicit data, and tag dependencies into account. Tags released in a callback or similar are not taken into account, only tags associated with a task are. It must be understood that the linear programming problem size is quadratic with the number of tasks and thus the time to solve it will be very long, it could be minutes for just a few dozen tasks. You should probably use lp_solve -timeout 1 test.pl -wmps test.mps
to convert the problem to MPS format and then use a better solver, glpsol
might be better than lp_solve
for instance (the –pcost
option may be useful), but sometimes doesn't manage to converge. cbc
might look slower, but it is parallel. For lp_solve
, be sure to try at least all the -B
options. For instance, we often just use lp_solve -cc -B1 -Bb -Bg -Bp -Bf -Br -BG -Bd -Bs -BB -Bo -Bc -Bi
, and the -gr
option can also be quite useful. The resulting schedule can be observed by using the tool starpu_lp2paje
, which converts it into the Paje format.
Data transfer time can only be taken into account when deps
is set. Only data transfers inferred from implicit data dependencies between tasks are taken into account. Other data transfers are assumed to be completely overlapped.
Setting deps
to 0 will only take into account the actual computations on processing units. However, it still properly takes into account the varying performances of kernels and processing units, which is quite more accurate than just comparing StarPU performances with the fastest of the kernels being used.
The prio
parameter tells StarPU whether to simulate taking into account the priorities as the StarPU scheduler would, i.e. schedule prioritized tasks before less prioritized tasks, to check to which extend this results to a less optimal solution. This increases even more computation time.
Creating views with StarVZ (see: https://github.com/schnorr/starvz) is made up of two steps. The initial stage consists of a pre-processing of the traces generated by the application, while the second one consists of the analysis itself and is carried out with R packages' aid. StarVZ is available at CRAN (https://cran.r-project.org/package=starvz) and depends on pj_dump
(from pajeng
) and rec2csv
(from recutils
).
To download and install StarVZ, it is necessary to have R
, pajeng
, and recutils:
# For pj_dump and rec2csv apt install -y pajeng recutils # For R apt install -y r-base libxml2-dev libssl-dev libcurl4-openssl-dev libgit2-dev libboost-dev
To install the StarVZ, the following command can be used:
echo "install.packages('starvz', repos = 'https://cloud.r-project.org')" | R --vanilla
To generate traces from an application, it is necessary to set STARPU_GENERATE_TRACE and build StarPU with FxT. Then, StarVZ can be used on a folder with StarPU FxT traces to produce a default view:
export PATH=$(Rscript -e 'cat(system.file("tools/", package = "starvz"), sep="\n")'):$PATH starvz /foo/path-to-fxt-files
An example of default view:
One can also use existing trace files (paje.trace
, tasks.rec
, data.rec
, papi.rec
and dag.dot
) skipping the StarVZ internal call to starpu_fxt_tool with:
starvz --use-paje-trace /foo/path-to-trace-files
Alternatively, each StarVZ step can be executed separately. Step 1 can be used on a folder with:
starvz -1 /foo/path-to-fxt-files
Then the second step can be executed directly in R. StarVZ enables a set of different plots that can be configured on a .yaml file. A default file is provided (default.yaml
); also, the options can be changed directly in R.
library(starvz) library(dplyr) dtrace <- starvz_read("./", selective = FALSE) # show idleness ratio dtrace$config$st$idleness = TRUE # show ABE bound dtrace$config$st$abe$active = TRUE # find the last task with dplyr dtrace$config$st$tasks$list = dtrace$Application %>% filter(End == max(End)) %>% .$JobId # show last task dependencies dtrace$config$st$tasks$active = TRUE dtrace$config$st$tasks$levels = 50 plot <- starvz_plot(dtrace)
An example of visualization follows:
The StarPU Eclipse Plugin provides the ability to generate the different traces directly from the Eclipse IDE.
Download the Eclipse installer from https://www.eclipse.org/downloads/packages/installer. When you run the installer, click on Eclipse IDE for Java Developers to start the installation process.
To be able to develop C/C++ applications, you need to install the CDT plugin. To do so, go to the Help dropdown menu at the top of the Eclipse window, choose Install New Software .... In the new window, enter the URL http://download.eclipse.org/tools/cdt/releases/9.10 into the box Work with and press the return key.
You need then to select CDT Main Features, then click the button Next twice, accept the terms of the license, and click the button Finish. Eclipse will ask you to restart.
To be able to compile the plugin, you need to install the plugin development environment (PDE). To do so, go to the menu Help, choose Eclipse Marketplace.... In the new window, enter PDE into the box Find and press the return key.
You can then click on the button Install of the Eclipse PDE latest. You may need to confirm the installation, then accept the terms of the license, and finally restart the Eclipse IDE.
The installation is now done.
StarPU can now be compiled and installed with its Eclipse plugin. To do so, you first need to configure StarPU with the option --enable-eclipse-plugin. The Eclipse IDE executable eclipse
must be in your PATH
.
export PATH=$HOME/usr/local/eclipse/java-2021-03/eclipse:$PATH mkdir build cd build ../configure --prefix=$HOME/usr/local/starpu --enable-eclipse-plugin make make install
The StarPU Eclipse plugin is installed in the directory dropins
.
$ ls $HOME/usr/local/eclipse/java-2021-03/eclipse/dropins StarPU_1.0.0.202105272056.jar
In the next section, we will show you how to use the plugin.
Once StarPU has been configured and installed with its Eclipse plugin, you first need to set up your environment for StarPU.
cd $HOME/usr/local/starpu source ./bin/starpu_env
To generate traces from the application, it is necessary to set STARPU_FXT_TRACE to 1.
export STARPU_FXT_TRACE=1
The eclipse workspace together with an example is available in lib/starpu/eclipse-plugin
.
cd ./lib/starpu/eclipse-plugin eclipse -data workspace
You can then open the file hello/hello.c
, and build the application by pressing Ctrl-B
.
The application can now be executed.
After executing the C/C++ StarPU application, one can use the StarPU plugin to generate and visualise the task graph of the application. The StarPU plugin eclipse is either available through the icons in the upper toolbar, or from the dropdown menu StarPU
.
To start, one first need to run the StarPU FxT tool, either through the FxT
icon of the toolbar, or from the menu StarPU
/ StarPU FxT Tool
. This will call the tool starpu_fxt_tool
to generate traces for your application execution.
A message dialog box is displayed to confirm the generation of the different traces.
One of the generated files is a Paje trace which can be viewed with ViTE, a trace explorer. To open and visualise the file paje.trace
with ViTE, one can select the second command of the StarPU menu, which is named Generate Paje Trace
, or click on the second icon named Trace
in the toolbar.
Another generated trace file is a task graph described using the DOT language. It is possible to get a graphical output of the graph by calling the graphviz library
. To do this, one can click on the third command of StarPU menu. A task graph of the application in the png
format is then generated.
In StarPU eclipse plugin, one can display the graph task directly from eclipse, or through a web browser. To do this, there is another command named Generate SVG graph
in the StarPU menu or HGraph in the toolbar of eclipse.
From the HTML file, you can see the graph task, and by clicking on a task name, it will open the C file in which the task submission was called (if you have an editor which understands the syntax href="file.c#123"
).
It is possible to enable memory statistics. To do so, you need to pass the option --enable-memory-stats when running configure
. It is then possible to call the function starpu_data_display_memory_stats() to display statistics about the current data handles registered within StarPU.
Moreover, statistics will be displayed at the end of the execution on data handles which have not been cleared out. This can be disabled by setting the environment variable STARPU_MEMORY_STATS to 0
.
For example, by adding a call to the function starpu_data_display_memory_stats() in the fblock example before unpartitioning the data, one will get something similar to:
$ STARPU_MEMORY_STATS=1 ./examples/filters/fblock ... #--------------------- Memory stats : #------- Data on Node #2 #----- Data : 0x5562074e8670 Size : 144 #-- Data access stats /!\ Work Underway Node #0 Direct access : 0 Loaded (Owner) : 0 Loaded (Shared) : 0 Invalidated (was Owner) : 1 Node #2 Direct access : 0 Loaded (Owner) : 1 Loaded (Shared) : 0 Invalidated (was Owner) : 0 #------- Data on Node #3 #----- Data : 0x5562074e9338 Size : 96 #-- Data access stats /!\ Work Underway Node #0 Direct access : 0 Loaded (Owner) : 0 Loaded (Shared) : 0 Invalidated (was Owner) : 1 Node #3 Direct access : 0 Loaded (Owner) : 1 Loaded (Shared) : 0 Invalidated (was Owner) : 0 #--------------------- ...
Different data statistics can be displayed at the end of the execution of the application. To enable them, you need to define the environment variable STARPU_ENABLE_STATS. When calling starpu_shutdown() various statistics will be displayed, execution, MSI cache statistics, allocation cache statistics, and data transfer statistics. The display can be disabled by setting the environment variable STARPU_STATS to 0
. If the environment variable STARPU_BUS_STATS is defined, you can call starpu_profiling_bus_helper_display_summary() to display statistics about the bus. If the environment variable STARPU_WORKER_STATS is defined, you can call starpu_profiling_worker_helper_display_summary() to display statistics about the workers. You can also call starpu_display_stats() which call both starpu_profiling_bus_helper_display_summary() and starpu_profiling_worker_helper_display_summary() at the same time.
$ ./examples/cholesky/cholesky_tag Computation took (in ms) 518.16 Synthetic GFlops : 44.21 #--------------------- MSI cache stats : TOTAL MSI stats hit 1622 (66.23 %) miss 827 (33.77 %) ...
$ STARPU_STATS=0 ./examples/cholesky/cholesky_tag Computation took (in ms) 518.16 Synthetic GFlop/s : 44.21
When an MPI execution is traced, especially if the execution is on several nodes, clock synchronization issues can appear. One may notice them mainly on communications (they are received before they are sent, for instance).
Each processor can call the function starpu_profiling_set_id() to set the ID used for the profiling trace filename. This function can be useful when executing an MPI program on several nodes, as it enables each processor to set a unique ID that helps to differentiate its trace file from the files generated by other processors. By doing this, it becomes easier to analyze and compare the profiling results of each processor separately, which is particularly helpful for large-scale parallel applications.
By default, StarPU does two MPI barriers with all MPI processes: one at the beginning of the application execution and one at the end. Then, starpu_fxt_tool
considers all processes leave the barriers at the exact same time, which makes two points for time synchronization between MPI processes.
However, a simple MPI barrier can be not precise enough, because the assumption all processes leave the barriers at the exact same time is in reality false. To have a more precise barrier, one may use the mpi_sync_clocks library (automatically provided when StarPU is built with NewMadeleine, but it can also be used with other MPI libraries). It provides a synchronized barrier, which aims at actually releasing all processes at the exact same time. Unfortunately, the gained precision costs some time (several seconds per barrier), that is why one can disable this precise synchronization with the environment variable STARPU_MPI_TRACE_SYNC_CLOCKS set to 0
, and use the faster MPI barrier instead.
Traces can also be inspected by hand by using the tool fxt_print
, for instance:
$ fxt_print -o -f /tmp/prof_file_something
Timings are in nanoseconds (while timings as seen in ViTE are in milliseconds).