Difference between revisions of "FAQ"
(→Are development tools available on the clusters ?) |
(→Software) |
||
| Line 66: | Line 66: | ||
pgf95 ← PGI Fortran 95 compiler | pgf95 ← PGI Fortran 95 compiler | ||
pghpf ← PGI High-Performance Fortran compiler | pghpf ← PGI High-Performance Fortran compiler | ||
| + | |||
| + | ==== How do I set up my environment for these compilers?==== | ||
| + | On our clusters, we use environment modules to prepare the environment for applications and compilers. By loading the corresponding module with the module load command, e.g.,<br> | ||
| + | module load gcc/4.8.2 | ||
| + | the environment variables as <tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt> and so on are adapted to the compiler you were loading. | ||
=== How do I compile MPI applications ?=== | === How do I compile MPI applications ?=== | ||
| + | The compilation of parallel applications based on the Message Passing Interface (MPI) is slightly more complicated. Once you have loaded the compiler of your choice, you must also decide '''which MPI library''' you want to use. Two MPI libraries are available on Brutus: | ||
| + | |||
| + | * Open MPI (recommended) | ||
| + | * MVAPICH2 | ||
| + | |||
| + | Applications compiled with Open MPI or MVAPICH2 run on nodes connected to the InfiniBand network. | ||
| + | |||
| + | '''Open MPI is recommended for all applications.''' MVAPICH2 is provided for applications that are not compatible (or do not run well) with Open MPI. | ||
| + | |||
| + | Two series of modules — <tt>open_mpi</tt> and <tt>mvapich2</tt> — are available to configure your environment for a particular MPI library. In addition, these modules define '''wrappers''' — e.g. <tt>mpicc</tt>, <tt>mpif90</tt> — that greatly simplify the compilation of MPI applications. These wrappers are compiler-dependent and invoke whichever compiler was active (loaded) when you loaded the MPI module. For this reason, the MPI module must '''absolutely''' be loaded '''after''' the compiler module. | ||
| + | |||
| + | To summarize, the compilation of an MPI application should look somewhat like this: | ||
| + | |||
| + | module load ''compiler'' | ||
| + | module load ''MPI library'' | ||
| + | mpicc ''program'' -o ''executable'' ← C program | ||
| + | mpiCC ''program'' -o ''executable'' ← C++ program | ||
| + | mpif77 ''program'' -o ''executable'' ← Fortran 77 program | ||
| + | mpif90 ''program'' -o ''executable'' ← Fortran 90 program | ||
| + | |||
| + | ==== Can I use another implementation of MPI?==== | ||
| + | Yes. We provide MVAPICH2, but we do not compile all libraries with support for MVAPICH2. We strongly recommend to use the centrally installed OpenMPI library. | ||
| + | |||
| + | ==== What about OpenMP applications?==== | ||
| + | You can use OpenMP but do not forget to set <tt>OMP_NUM_THREADS=#threads</tt> and submit it with the <tt>bsub</tt> option <tt>-n #threads</tt>. | ||
| + | |||
===What scientific libraries are available on the clusters ?=== | ===What scientific libraries are available on the clusters ?=== | ||
| + | On our clusters, we provide a large range of scientific libraries and/or applications: | ||
| + | |||
| + | <ul> | ||
| + | <li> PGI compilers come with the AMD's Core Math Library (ACML)</li> | ||
| + | <li> Intel's Math Kernel Library (MKL)</li> | ||
| + | <li> openblas (former GOTO)</li> | ||
| + | <li> netlib blas and lapack</li> | ||
| + | <li> scalapack</li> | ||
| + | <li> FFTW</li> | ||
| + | <li> GSL</li> | ||
| + | <li> python packages: numpy, scipy</li> | ||
| + | <li> Boost</li> | ||
| + | <li> Eigen </li> | ||
| + | <li> Eigen </li> | ||
| + | </ul> | ||
===Can you please allow me to run sudo for installing my code ?=== | ===Can you please allow me to run sudo for installing my code ?=== | ||
Revision as of 09:12, 22 August 2016
Contents
- 1 General
- 2 Access
- 3 Software
- 3.1 Do you provide any software on your clusters ?
- 3.2 Why does my 32-bit executable not work on your clusters ?
- 3.3 Can I run Windows executables on the clusters ?
- 3.4 Can you please update GLIBC on the clusters ?
- 3.5 Is it necessary to recompile or can I just copy my application to a cluster ?
- 3.6 Are development tools available on the clusters ?
- 3.7 How do I compile MPI applications ?
- 3.8 What scientific libraries are available on the clusters ?
- 3.9 Can you please allow me to run sudo for installing my code ?
- 3.10 Why can't I install my application into /usr/bin and /usr/lib64 ?
- 3.11 Is there a license available for application XYZ ?
- 4 Environment modules
- 5 Submitting jobs
- 5.1 Can I run an application on the login nodes?
- 5.2 Can I access a compute node via ssh or rsh?
- 5.3 How do I execute a program on the cluster?
- 5.4 How do I submit a simple command ?
- 5.5 How do I submit a shell script ?
- 5.6 How do I submit a parallel job ?
- 5.7 What are the processor and time limits ?
- 5.8 What is the maximal amount of memory that I can use ?
- 5.9 Which queue should I choose ?
- 5.10 How many jobs can I submit ?
- 5.11 How much time should I request for my job ?
- 5.12 What happens when a job reaches its time limit ?
- 5.13 How do I submit a series of jobs (job chaining)?
- 6 Monitoring jobs
- 6.1 When does my job start ?
- 6.2 How can I check the status of my job(s) ?
- 6.3 Why is my job waiting for a long time in the queue ?
- 6.4 Where is my job's output ?
- 6.5 Can I see my job's output in real time ?
- 6.6 How do I know when my job is done ?
- 6.7 Can I see the resources used by my job(s) ?
- 6.8 How do I kill a job ?
- 7 Data management and file transfer
- 7.1 How much disk space is available on then clusters ?
- 7.2 How much space can I use ?
- 7.3 What happens when I reach my quota ?
- 7.4 What if I need more space ?
- 7.5 Why is there a limit for the number of files in my home/scratch directory ?
- 7.6 Why is storage in the cluster more expensive than cheap external USB 3 hard drives ?
- 7.7 How long can I keep files in the scratch directories ?
- 7.8 Why did you delete my files in scratch ?
- 7.9 Are my files backed up regularly ?
- 7.10 How do I restore a file from a backup ?
- 7.11 What is the fastest way to transfer files from/to the cluster ?
- 7.12 Why is file transfer very slow ?
- 8 Miscellaneous
General
Who is ID SIS HPC ?
What services are provided ID SIS HPC ?
Do I need to pay or are the services for free ?
Where can I find more information ?
Access
Who can use the services of ID SIS HPC ?
Anyone within ETH may use the services provided by ID SIS HPC. Professors and institutes who participated in the purchase of the system are guaranteed a share of the resources proportional to their investment. Other users must share the public resources allocated to the IT Services. Researchers from other Swiss and international institutions can use the services, as long as they have a collaboration with an institute of ETH Zurich.
Do I need an account ?
For most services, you can directly login with the credentials of your NETHZ account. The only exceptions are the Brutus cluster and the CLC genomics service, where you still need to apply for an account. If you are interested in using these services, then please contact cluster support.
How do I log in ?
For security reasons, you can only access our services from within the ETH network. If you are outside the ETH network, you have to establish a VPN connection first. From a Linux or a Mac OS X computer, you can login with
ssh username@hostname
or for graphical applications with
ssh -Y username@hostname
If you are using Windows, then you need to make use of a third party application, as for instance PuTTY or Cygwin, that provide a console to enter the ssh command, analogue to the Linux users. For graphical applications, Windows users require a software called an X-server, which provides X11 forwarding. Common X-servers are Cygwin/X, Xming, Exceed, XWin-32.
X11-forwarding with -X does not work, what am I doing wrong?
As described above, you have to use the -Y option for X11-forwarding. Log in with:
ssh -Y username@hostname
How can I change my password?
Since the NETHZ password is used for the log in to our services, the users can change their password at
Can I change my default shell?
Bash is the default shell for all users. The configuration of our services is complex and everything is tested extensively using bash. It is therefore the only shell that we fully support. You are free to use a different shell, but you are doing so at your own risks.
Software
Do you provide any software on your clusters ?
On our clusters, we provide a wide range of centrally installed applications and libraries. Our software stack contains commercial as well as open source software. An overview on all centrally installed applications can be found in our application tables.
Why does my 32-bit executable not work on your clusters ?
Our cluster are pure 64-bit systems. Your 32-bit executable might runs without problems in some cases, but there are certain limitations. A 32-bit executable can only use up to 3 GB of virtual memory. If you try to use more, this might results in a segmentation fault or an out of memory error message. The solution for this problem is to recompile your application for 64-bit.
Can I run Windows executables on the clusters ?
Windows executables do not run under Linux. In order to be able to run your application on our clusters, you need to make sure that it is a 64-bit binary for Linux.
Can you please update GLIBC on the clusters ?
The libc is part of the operating system. Updating libc is equivalent to updating the operating system on the cluster. Therefore we can not just update libc. If your executable requires a newer version of libc (GLIBC), then please consider recompiling the source code directly on the cluster, where you would like to run the executable.
Is it necessary to recompile or can I just copy my application to a cluster ?
Statically linked, single-processor executables built on standard x86 Linux platforms should run without any problem on our clusters. Recompliling may improves the performance, though. Dynamically linked executables will not run if the required shared libraries are either not available or not compatible (e.g. 32-bit executable and 64-bit library). Recompiling is recommended.
Are development tools available on the clusters ?
On our clusters we provide different versions of the standard compilers from gcc, intel and pgi. To identify the actual versions that are installed on the cluster, please use the module available command:
module available gcc module available intel module available pgi
Executables corresponding to the compilers:
gcc ← GNU C compiler g++ ← GNU C++ compiler gfortran ← GNU Fortran 90/95 compiler
icc ← Intel C compiler icpc ← Intel C++ compiler ifort ← Intel Fortran 90/95 compiler
pgcc ← PGI C compiler pgCC ← PGI C++ compiler pgf77 ← PGI Fortran 77 compiler pgf90 ← PGI Fortran 90 compiler pgf95 ← PGI Fortran 95 compiler pghpf ← PGI High-Performance Fortran compiler
How do I set up my environment for these compilers?
On our clusters, we use environment modules to prepare the environment for applications and compilers. By loading the corresponding module with the module load command, e.g.,
module load gcc/4.8.2
the environment variables as PATH, LD_LIBRARY_PATH and so on are adapted to the compiler you were loading.
How do I compile MPI applications ?
The compilation of parallel applications based on the Message Passing Interface (MPI) is slightly more complicated. Once you have loaded the compiler of your choice, you must also decide which MPI library you want to use. Two MPI libraries are available on Brutus:
- Open MPI (recommended)
- MVAPICH2
Applications compiled with Open MPI or MVAPICH2 run on nodes connected to the InfiniBand network.
Open MPI is recommended for all applications. MVAPICH2 is provided for applications that are not compatible (or do not run well) with Open MPI.
Two series of modules — open_mpi and mvapich2 — are available to configure your environment for a particular MPI library. In addition, these modules define wrappers — e.g. mpicc, mpif90 — that greatly simplify the compilation of MPI applications. These wrappers are compiler-dependent and invoke whichever compiler was active (loaded) when you loaded the MPI module. For this reason, the MPI module must absolutely be loaded after the compiler module.
To summarize, the compilation of an MPI application should look somewhat like this:
module load compiler module load MPI library mpicc program -o executable ← C program mpiCC program -o executable ← C++ program mpif77 program -o executable ← Fortran 77 program mpif90 program -o executable ← Fortran 90 program
Can I use another implementation of MPI?
Yes. We provide MVAPICH2, but we do not compile all libraries with support for MVAPICH2. We strongly recommend to use the centrally installed OpenMPI library.
What about OpenMP applications?
You can use OpenMP but do not forget to set OMP_NUM_THREADS=#threads and submit it with the bsub option -n #threads.
What scientific libraries are available on the clusters ?
On our clusters, we provide a large range of scientific libraries and/or applications:
- PGI compilers come with the AMD's Core Math Library (ACML)
- Intel's Math Kernel Library (MKL)
- openblas (former GOTO)
- netlib blas and lapack
- scalapack
- FFTW
- GSL
- python packages: numpy, scipy
- Boost
- Eigen
- Eigen
Can you please allow me to run sudo for installing my code ?
Due to security reasons, we can not allow users to run sudo for installing their application of choice. The clusters are shared by more than 2000 people, and if we would allow them to use sudo, this could cause a lot of problem, which would affect all other cluster users. We recommend that you install software in your home directory, such that you do not need to run sudo for the installation step.
Why can't I install my application into /usr/bin and /usr/lib64 ?
The directories /usr/bin and /usr/lib64 are primarily used by the operating system for installing packages through the packet manager and only our system administrators have write access to them. The centrally installed applications and libraries are located in /cluster/apps and user software should be installed in the home directory.
Is there a license available for application XYZ ?
The ID SIS HPC team operates and maintains the HPC clusters and provides some more services, but we do not provide any software license at all. Licenses for commercial applications are either provided by the central license administration of ETH or directly by a research group or an institute/department.
