Difference between revisions of "Storage and data transfer"

Once you can log in to the cluster, you can start setting up your calculation job and you need your data. Therefore, two questions arise:

1. Where to store data?
2. How to transfer data?

Here, we explain the storage system on the cluster and give examples how to transfer data between your local computer and the cluster.

Quick examples

Upload a directory from your local computer to /cluster/scratch/username ($SCRATCH) on Euler

$ scp -r dummy_dir username@euler.ethz.ch:/cluster/scratch/username/

Log in to the cluster and check your disk space quota

$ lquota

Global Scratch

$ cd $SCRATCH
$ pwd
/cluster/scratch/username

• $SCRATCH is a fast, short-term storage for computations running on the cluster. It is created automatically upon first access (cd $SCRATCH) and visible (mounted) only when accessed.

• It has strict usage rules (see $SCRATCH/__USAGE_RULES__ for details) and has no backup.

Local Scratch

/scratch on each compute node ($TMPDIR)

• The local scratch is intended for serial, I/O-intensive applications. Therefore, it has a very short life span. Data are deleted automatically when the job ends.
• Scratch space must be requested by the job and has no backup.

See how to use local scratch

Project

$ cd /cluster/project/groupname

Similar to $HOME, but for groups, it is a safe, long-term storage for critical data.

Work

$ cd /cluster/work/groupname

Similar to global scratch, but without purge, it is a fast, short-or medium-term storage for large computations.

The folder is visible only when accessed.

Central NAS

Groups who have purchased storage on the central NAS of ETH provided ID Systemdienste can access it on our clusters.

Other NAS

Groups who are operating their own NAS can export a shared file system via NFS to Euler. The user and group ID's on the NAS needs to be consistent with ETH user names and groups.

The NAS share needs to be mountable via NFSv3 (shares that only support CIFS cannot be mounted on the HPC clusters), and exported to the subnet of our HPC clusters. The NAS is then mounted automatically on our clusters under

/nfs/servername/sharename

Using scp command

Upload dummy_file from your workstation to your home directory on Euler

$ scp dummy_file username@euler.ethz.ch:

Download dummy_file from Euler to the current directory on your workstation

$ scp username@euler.ethz.ch:dummy_file .

Copy a directory to Euler

$ scp -r dummy_dir username@euler.ethz.ch:

Example: upload a directory with rsync

Create two files in the dummy directory and use rsync to transfer the folder

$ mkdir dummy_dir
$ touch dummy_dir/dummy_file1 dummy_dir/dummy_file2
$ rsync -av dummy_dir username@euler.ethz.ch:dummy_dir