StarPU Handbook
|
StarPU provides a pkg-config
executable to facilitate the retrieval of necessary compiler and linker flags. This is useful when compiling and linking an application with StarPU, as certain flags or libraries (such as CUDA
or libspe2
) may be required.
If StarPU is not installed in a standard location, the path of StarPU's library must be specified in the environment variable PKG_CONFIG_PATH
to allow pkg-config
to find it. For example, if StarPU is installed in $STARPU_PATH
, you can set the variable PKG_CONFIG_PATH
like this:
$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$STARPU_PATH/lib/pkgconfig
The flags required to compile or link against StarPU are then accessible with the following commands:
$ pkg-config --cflags starpu-1.4 # options for the compiler $ pkg-config --libs starpu-1.4 # options for the linker
Please note that it is still possible to use the API provided in StarPU version 1.0 by calling pkg-config
with the starpu-1.0
package. Similar packages are provided for starpumpi-1.0
and starpufft-1.0
. For the API provided in StarPU version 0.9, you can use pkg-config
with the libstarpu
package. Similar packages are provided for libstarpumpi
and libstarpufft
.
Make sure that pkg-config –libs starpu-1.4
produces valid output before going further. To achieve this, make sure that your PKG_CONFIG_PATH
is correctly set to the location where starpu-1.4.pc
was installed during the make install
process.
Furthermore, if you intend to link your application statically, remember to include the –static
option during the linking process.
Additionally, for runtime execution, it is necessary to set the LD_LIBRARY_PATH
environment variable. This ensures that dynamic libraries are located and loaded correctly during runtime.
$ export LD_LIBRARY_PATH=$STARPU_PATH/lib:$LD_LIBRARY_PATH
And finally you should set the PATH
variable to get access to various StarPU tools:
$ export PATH=$PATH:$STARPU_PATH/bin
Run the following command to ensure that StarPU is executing properly and successfully detecting your hardware. If any issues arise, examine the output of lstopo
from the hwloc
project and report any problems either to the hwloc project or to us.
$ starpu_machine_display
A tool is provided to help set all the environment variables needed by StarPU. Once StarPU is installed in a specific directory, calling the script bin/starpu_env
will set in your current environment the variables STARPU_PATH
, LD_LIBRARY_PATH
, PKG_CONFIG_PATH
, PATH
and MANPATH
.
$ source $STARPU_PATH/bin/starpu_env
When using a Makefile, the following lines can be added to set the options for the compiler and the linker:
CFLAGS += $$(pkg-config --cflags starpu-1.4) LDLIBS += $$(pkg-config --libs starpu-1.4)
If you have a test-starpu.c
file containing for instance:
You can build it with make test-starpu
and run it with ./test-starpu
This section shows a minimal example integrating StarPU in an existing application's CMake build system.
Let's assume we want to build an executable from the following source code using CMake:
The CMakeLists.txt
file below uses the Pkg-Config support from CMake to autodetect the StarPU installation and library dependences (such as libhwloc
) provided that the PKG_CONFIG_PATH
variable is set, and is sufficient to build a statically-linked executable. This example has been successfully tested with CMake 3.2, though it may work with earlier CMake 3.x versions.
The following CMakeLists.txt
implements an alternative, more complex strategy, still relying on Pkg-Config, but also taking into account additional flags. While more complete, this approach makes CMake's build types (Debug, Release, ...) unavailable because of the direct affectation to variable CMAKE_C_FLAGS
. If both the full flags support and the build types support are needed, the CMakeLists.txt
below may be altered to work with CMAKE_C_FLAGS_RELEASE
, CMAKE_C_FLAGS_DEBUG
, and others as needed. This example has been successfully tested with CMake 3.2, though it may work with earlier CMake 3.x versions.
Basic examples using StarPU are built in the directory examples/basic_examples/
(and installed in $STARPU_PATH/lib/starpu/examples/
). You can for example run the example vector_scal
.
$ ./examples/basic_examples/vector_scal BEFORE: First element was 1.000000 AFTER: First element is 3.140000
When StarPU is used for the first time, the directory $STARPU_HOME/.starpu/
is created, performance models will be stored in this directory (STARPU_HOME).
Please note that buses are benchmarked when StarPU is launched for the first time. This may take a few minutes, or less if libhwloc
is installed. This step is done only once per user and per machine.
Batch files are provided to run StarPU applications under Microsoft Visual C. They are installed in $STARPU_PATH/bin/msvc
.
To execute a StarPU application, you first need to set the environment variable STARPU_PATH.
c:\....> cd c:\cygwin\home\ci\starpu\ c:\....> set STARPU_PATH=c:\cygwin\home\ci\starpu\ c:\....> cd bin\msvc c:\....> starpu_open.bat starpu_simple.c
The batch script will run Microsoft Visual C with a basic project file to run the given application.
The batch script starpu_clean.bat
can be used to delete all compilation generated files.
The batch script starpu_exec.bat
can be used to compile and execute a StarPU application from the command prompt.
c:\....> cd c:\cygwin\home\ci\starpu\ c:\....> set STARPU_PATH=c:\cygwin\home\ci\starpu\ c:\....> cd bin\msvc c:\....> starpu_exec.bat ..\..\..\..\examples\basic_examples\hello_world.c
MSVC StarPU Execution ... /out:hello_world.exe ... Hello world (params = {1, 2.00000}) Callback function got argument 0000042 c:\....>
StarPU automatically binds one thread per CPU core. It does not use SMT/hyperthreading because kernels are usually already optimized for using a full core, and using hyperthreading would make kernel calibration rather random.
Since driving GPUs is a CPU-consuming task, StarPU dedicates one core per GPU.
While StarPU tasks are executing, the application is not supposed to do computations in the threads it starts itself, tasks should be used instead.
If the application needs to reserve some cores for its own computations, it can do so with the field starpu_conf::reserve_ncpus, get the core IDs with starpu_get_next_bindid(), and bind to them with starpu_bind_thread_on().
Another option is for the application to pause StarPU by calling starpu_pause(), then to perform its own computations, and then to resume StarPU by calling starpu_resume() so that StarPU can execute tasks.
If a computation library used by the application actually creates its own thread, it may be useful to call starpu_bind_thread_on_worker() before e.g. initializing the library, so that the library records which binding it is supposed to use. And then call starpu_bind_thread_on_main() again, or starpu_bind_thread_on_cpu() if a core was reserved with starpu_get_next_bindid().
In case that computation library wants to bind threads itself, and uses physical numbering instead of logical numbering (as defined by hwloc), starpu_cpu_os_index() can be used to convert from StarPU cpuid to OS cpu index.
When both CUDA and OpenCL drivers are enabled, StarPU will launch an OpenCL worker for NVIDIA GPUs only if CUDA is not already running on them. This design choice was necessary as OpenCL and CUDA can not run at the same time on the same NVIDIA GPU, as there is currently no interoperability between them.
To enable OpenCL, you need either to disable CUDA when configuring StarPU:
$ ./configure --disable-cuda
or when running applications:
$ STARPU_NCUDA=0 ./application
OpenCL will automatically be started on any device not yet used by CUDA. So on a machine running 4 GPUS, it is therefore possible to enable CUDA on 2 devices, and OpenCL on the other 2 devices by calling:
$ STARPU_NCUDA=2 ./application
StarPU stores performance model files for bus benchmarking and codelet profiles in different directories.
By default, all files are stored in $STARPU_HOME/.starpu/sampling
.
If the environment variable STARPU_HOME is not defined, its default value is $HOME
on Unix environments, and $USERPROFILE
on Windows environments.
Environment variables STARPU_PERF_MODEL_DIR and STARPU_PERF_MODEL_PATH can also be used to specify other directories in which to store performance files (Simulated Benchmarks).
The configure option --with-perf-model-dir can also be used to define a performance model directory.
When looking for performance files either for bus benchmarking or for codelet performances, StarPU
$prefix/share/starpu/perfmodels/sampling
If the files are not present and must be created, they will be created in the first defined directory from the list above.
rm -rf $PWD/xxx && STARPU_PERF_MODEL_DIR=$PWD/xxx ./application
will use performance model files from the directory $STARPU_HOME/.starpu/sampling
if they are available, otherwise will create these files in $STARPU_PERF_MODEL_DIR
.
To know the list of directories StarPU will search for performances files, one can use the tool starpu_perfmodel_display
$ starpu_perfmodel_display -d directory: </home/user1/.starpu/sampling/codelets/45/> directory: </usr/local/install/share/starpu/perfmodels/sampling/codelets/45/>
$ STARPU_PERF_MODEL_DIR=/tmp/xxx starpu_perfmodel_display -d directory: </tmp/xxx/codelets/45/> directory: </home/user1/.starpu/sampling/codelets/45/> directory: </usr/local/install/share/starpu/perfmodels/sampling/codelets/45/>
When using the variable STARPU_PERF_MODEL_DIR, the directory will be created if it does not exist when dumping new performance model files.
When using the variable STARPU_PERF_MODEL_PATH, only existing directories will be taken into account.
$ mkdir /tmp/yyy && STARPU_PERF_MODEL_DIR=/tmp/xxx STARPU_PERF_MODEL_PATH=/tmp/zzz:/tmp/yyy starpu_perfmodel_display -d [starpu][adrets][_perf_model_add_dir] Warning: directory </tmp/zzz> as set by variable STARPU_PERF_MODEL_PATH does not exist directory: </tmp/xxx/codelets/45/> directory: </home/user1/.starpu/sampling/codelets/45/> directory: </tmp/yyy/codelets/45/> directory: </usr/local/install/share/starpu/perfmodels/sampling/codelets/45/>
Once your application has created the performance files in a given directory, it is thus possible to move these files in another location and keep using them.
./application # files are created in $HOME/.starpu/sampling mv $HOME/.starpu/sampling /usr/local/starpu/sampling STARPU_PERF_MODEL_DIR=/usr/local/starpu/sampling ./application