Skip to content

Alternative compilers - simple example

GCC is the most supported compiler

The ITSR Applications Team have compiled the majority of applications available via modules on Apocrita using GCC. Whilst other compilers are available on Apocrita as outlined below, you will receive the best support from us should you choose to use GCC as well.

In all of the examples so far in this tutorial, we have used the GCC compiler. Using other compilers is also possible, but it requires a different workflow.

For the purposes of the examples below, we are going to try compile and run a very simple C++ MPI binary, first with our recommended GCC and Open MPI stack, and then using Intel and Intel MPI.

Comparative GCC and Open MPI example

For the purposes of comparison, a spack.yaml template for compiling and running this example using our recommended GCC and Open MPI setup can be found in the spack-config-templates repository inside the environment-templates sub-directory for the version of Spack you are using - called helloworld-gcc-ompi.yaml:

Contents of spack.yaml for Hello World GCC/Open MPI (click to expand) 
spack:
  concretizer:
    unify: true
  config:
    install_tree:
      root: /gpfs/scratch/${USER}/spack/apps
      projections:
        ^mpi: '{architecture}/{compiler.name}-{compiler.version}/{name}/{version}-{^mpi.name}-{^mpi.version}-{hash:7}'
        all: '{architecture}/{compiler.name}-{compiler.version}/{name}/{version}-{hash:7}'
    license_dir: /gpfs/scratch/${USER}/spack/licenses
    source_cache: /gpfs/scratch/${USER}/spack/cache
  modules:
    prefix_inspections:
      ./share/aclocal:
      - ACLOCAL_PATH
      ./lib:
      - LD_LIBRARY_PATH
      - LIBRARY_PATH
      ./lib64:
      - LD_LIBRARY_PATH
      - LIBRARY_PATH
      ./include:
      - C_INCLUDE_PATH
      - CPLUS_INCLUDE_PATH
  packages:
    all:
      target: [x86_64_v4]
    gcc:
      externals:
      - spec: gcc@11.4.1 languages='c,c++,fortran'
        prefix: /usr
        extra_attributes:
          compilers:
            c: /usr/bin/gcc
            cxx: /usr/bin/g++
            fortran: /usr/bin/gfortran
      - spec: gcc@12.2.0 languages='c,c++,fortran'
        prefix: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg
        extra_attributes:
          compilers:
            c: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gcc
            cxx: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/g++
            fortran: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg/bin/gfortran
          flags: {}
          environment: {}
          extra_rpaths: []
    openmpi:
      buildable: false
      prefer:
      - '@5.0.3 +gpfs'
      externals:
      - spec: "openmpi@5.0.3 +gpfs %gcc@12.2.0"
        prefix: /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3
      - spec: "openmpi@5.0.3-nogpfs ~gpfs %gcc@12.2.0"
        prefix: /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3-nogpfs
    ucx:
      buildable: false
      externals:
      - spec: "ucx@1.16.0%gcc@12.2.0"
        prefix: /share/apps/rocky9/general/libs/ucx/gcc/12.2.0/1.16.0
  repos:
    builtin:
      destination: /share/apps/rocky9/general/apps/spack-packages/2026.03.0
  specs:
    - gcc@12.2.0
    - openmpi
  upstreams:
    apocrita:
      install_tree: /share/apps/rocky9/spack/apps
  view:
    default:
      root: .spack-env/view

If we save this as /gpfs/scratch/${USER}/spack-environments/helloworld-gcc-ompi/spack.yaml and then activate, spec and install the environment:

Output of activate and spec Spack commands (click to expand) 
$ spack env activate -p /gpfs/scratch/${USER}/spack-environments/helloworld-gcc-ompi
[helloworld-gcc-ompi] $ spack spec
[e]  gcc@12.2.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky9 target=x86_64
[e]  openmpi@5.0.3+atomics~cuda~debug+fortran+gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm+romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none platform=linux os=rocky9 target=x86_64 %c,cxx,fortran=gcc@12.2.0
Output of spack install (click to expand) 
[helloworld-gcc-ompi] $ spack install -j ${SLURM_NTASKS}
==> Concretized 2 specs
[e]  c6glrz7  gcc@12.2.0+binutils+bootstrap~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' platform=linux os=rocky9 target=x86_64
[e]  equm4yy  openmpi@5.0.3+atomics~cuda~debug+fortran+gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm+romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none platform=linux os=rocky9 target=x86_64 %c,cxx,fortran=gcc@12.2.0

[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-11.4.1/gcc/12.2.0-6frskzg (external gcc-12.2.0-c6glrz75exefrjwq5gurxmxiqkc34ubt)
[+] /share/apps/rocky9/general/libs/openmpi/gcc/12.2.0/5.0.3 (external openmpi-5.0.3-equm4yyfcvzhqexakbrqsbo5r5ju55zu)
==> Updating view at /gpfs/scratch/abc123/spack-environments/helloworld-gcc-ompi/.spack-env/view

To ensure that all required environment variables are activated for Open MPI, it is best to deactivate the environment and then re-activate it again to enable them:

spack env deactivate
spack env activate -p /gpfs/scratch/${USER}/spack-environments/helloworld-gcc-ompi

Create a file called hello_world_mpi.cpp in a folder of your choice, for this example we will use the path:

/gpfs/scratch/${USER}/hello_world_cpp/hello_world_mpi.cpp

Use the sample code from the previously mentioned example:

hello_world_mpi.cpp (click to expand) 
#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv){
    int process_Rank, size_Of_Cluster;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size_Of_Cluster);
    MPI_Comm_rank(MPI_COMM_WORLD, &process_Rank);

    for(int i = 0; i < size_Of_Cluster; i++){
        if(i == process_Rank){
            printf("Hello World from process %d of %d\n", process_Rank, size_Of_Cluster);
        }
        MPI_Barrier(MPI_COMM_WORLD);
    }

    MPI_Finalize();
    return 0;
}

Then, much as our previous cmake example, the Spack environment enables us to compile the simple example:

hello_world_ompi.exe compile and install 
[helloworld-gcc-ompi] $ cd /gpfs/scratch/${USER}/hello_world_cpp
[helloworld-gcc-ompi] [hello_world_cpp] $ mpic++ hello_world_mpi.cpp -o hello_world_ompi.exe

You can then run the compiled executable as instructed:

hello_world_ompi.exe output 
[helloworld-gcc-ompi] [hello_world_cpp] $ mpirun -np ${SLURM_NTASKS} ./hello_world_ompi.exe
Hello World from process 0 of 8
Hello World from process 1 of 8
Hello World from process 2 of 8
Hello World from process 3 of 8
Hello World from process 4 of 8
Hello World from process 5 of 8
Hello World from process 6 of 8
Hello World from process 7 of 8

Intel and Intel MPI example

Let's run through the same thing again, but this time we will compile using Intel and Intel MPI.

A spack.yaml template for compiling and running this example using Intel and Intel MPI can be found in the spack-config-templates repository inside the environment-templates sub-directory for the version of Spack you are using - called helloworld-intel-intelmpi.yaml:

Contents of spack.yaml for Hello World Intel/Intel MPI (click to expand) 
spack:
  concretizer:
    unify: true
  config:
    install_tree:
      root: /gpfs/scratch/${USER}/spack/apps
      projections:
        ^mpi: '{architecture}/{compiler.name}-{compiler.version}/{name}/{version}-{^mpi.name}-{^mpi.version}-{hash:7}'
        all: '{architecture}/{compiler.name}-{compiler.version}/{name}/{version}-{hash:7}'
    license_dir: /gpfs/scratch/${USER}/spack/licenses
    source_cache: /gpfs/scratch/${USER}/spack/cache
  modules:
    prefix_inspections:
      ./share/aclocal:
      - ACLOCAL_PATH
      ./lib:
      - LD_LIBRARY_PATH
      - LIBRARY_PATH
      ./lib64:
      - LD_LIBRARY_PATH
      - LIBRARY_PATH
      ./include:
      - C_INCLUDE_PATH
      - CPLUS_INCLUDE_PATH
  packages:
    all:
      target: [x86_64_v4]
    intel-oneapi-compilers:
      externals:
      - spec: intel-oneapi-compilers@2024.1.0
        prefix: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-compilers/2024.1.0-bdqsx5f
        extra_attributes:
          compilers:
            c: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-compilers/2024.1.0-bdqsx5f/compiler/2024.1/bin/icx
            cxx: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-compilers/2024.1.0-bdqsx5f/compiler/2024.1/bin/icpx
            fortran: /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-compilers/2024.1.0-bdqsx5f/compiler/2024.1/bin/ifx
          flags: {}
          environment: {}
          extra_rpaths: []
  repos:
    builtin:
      destination: /share/apps/rocky9/general/apps/spack-packages/2026.03.0
  specs:
    - intel-oneapi-compilers
    - intel-oneapi-mpi
  upstreams:
    apocrita:
      install_tree: /share/apps/rocky9/spack/apps
  view:
    default:
      root: .spack-env/view

So, what has changed?

  • The packages section just lists intel-oneapi-compilers@2024.1.0, the Intel compiler, not GCC
  • The packages section no longer lists our external Open MPI and UCX versions...
  • ...because we have replaced Open MPI with intel-oneapi-mpi in our specs
  • intel-oneapi-compilers replaces the gcc@12.2.0 we had in the first GCC/Open MPI example above
  • Remember, Intel packages commence with intel-oneapi-* - for more information refer back to the Custom Scopes tutorial

If we save this as /gpfs/scratch/${USER}/spack-environments/helloworld-intel-intelmpi/spack.yaml and then activate, spec and install the environment:

Output of activate and spec Spack commands (click to expand) 
$ spack env activate -p /gpfs/scratch/${USER}/spack-environments/helloworld-intel-intelmpi
[helloworld-intel-intelmpi] $ spack spec
[e]  intel-oneapi-compilers@2024.1.0~amd+envmods~nvidia build_system=generic platform=linux os=rocky9 target=x86_64
[^]  intel-oneapi-mpi@2021.12.1+envmods~external-libfabric~generic-names~ilp64 build_system=generic platform=linux os=rocky9 target=x86_64_v4
[^]      ^gcc-runtime@12.2.0 build_system=generic platform=linux os=rocky9 target=x86_64_v4
[e]      ^glibc@2.34 build_system=autotools platform=linux os=rocky9 target=x86_64_v4
Output of spack install (click to expand) 
[helloworld-intel-intelmpi] $ spack install -j ${SLURM_NTASKS}
==> Concretized 2 specs
[e]  ngzdq2p  intel-oneapi-compilers@2024.1.0~amd+envmods~nvidia build_system=generic platform=linux os=rocky9 target=x86_64
[^]  kmurskw  intel-oneapi-mpi@2021.12.1+envmods~external-libfabric~generic-names~ilp64 build_system=generic platform=linux os=rocky9 target=x86_64_v4
[^]  w77gg5r      ^gcc-runtime@12.2.0 build_system=generic platform=linux os=rocky9 target=x86_64_v4
[e]  xri56vc      ^glibc@2.34 build_system=autotools platform=linux os=rocky9 target=x86_64_v4

[+] /usr (external glibc-2.34-xri56vcyzs7kkvakhoku3fefc46nw25y)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-compilers/2024.1.0-bdqsx5f (external intel-oneapi-compilers-2024.1.0-ngzdq2pq6qlfghsd7jz74bvfdcyyx7yp)
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/gcc-runtime/12.2.0-w77gg5r
[+] /share/apps/rocky9/spack/apps/linux-rocky9-x86_64_v4/gcc-12.2.0/intel-oneapi-mpi/2021.12.1-kmurskw
==> Updating view at /gpfs/scratch/abc123/spack-environments/helloworld-intel-intelmpi/.spack-env/view

To ensure that all required environment variables are activated for Intel and Intel MPI, it is best to deactivate the environment and then re-activate it again to enable them:

spack env deactivate
spack env activate -p /gpfs/scratch/${USER}/spack-environments/helloworld-intel-intelmpi

We can use the same hello_world_mpi.cpp as above to again compile the simple example (this time we will use the name hello_world_intelmpi.exe):

Use mpiicpx instead of mpiicc

The linked example uses mpiicc to compile which is now deprecated. As we are using version 2024.1.0 for this tutorial, you will need to use mpiicpx instead, as detailed below.

hello_world_intelmpi.exe compile and install 
[helloworld-intel-intelmpi] $ cd /gpfs/scratch/${USER}/hello_world_cpp
[helloworld-intel-intelmpi] [hello_world_cpp] $ mpiicpx hello_world_mpi.cpp -o hello_world_intelmpi.exe

You can again run the compiled executable as instructed:

hello_world_intelmpi.exe output 
[helloworld-intel-intelmpi] [hello_world_cpp] $ mpirun -np ${SLURM_NTASKS} ./hello_world_intelmpi.exe
Hello World from process 0 of 8
Hello World from process 1 of 8
Hello World from process 2 of 8
Hello World from process 3 of 8
Hello World from process 4 of 8
Hello World from process 5 of 8
Hello World from process 6 of 8
Hello World from process 7 of 8