Storage and data transfer
Once you can log in to the cluster, you can start setting up your calculation job and you need your data. Therefore, two questions arise:
1. Where to store data?
Here, we explain the storage system on the cluster and give examples how to transfer data between your local computer and the cluster.
Upload a directory from your local computer to /cluster/scratch/username ($SCRATCH) on Euler
$ scp -r dummy_dir email@example.com:/cluster/scratch/username/
Log in to the cluster and check your disk space quota
Personal storage for all users
$ cd $HOME $ pwd /cluster/home/username
- $HOME is a safe, long-term storage for critical data (program source, scripts, etc.) and is accessible only by the user (owner). This means other people cannot read its contents.
- There is a disk quota of 16/20 GB and a maximum of 80’000/100’000 files (soft/hard quota). You can check the quota with the command lquota.
- Its content is saved every hour/day/week using snapshot, which is stored in the hidden .snapshot directory.
$ cd $SCRATCH $ pwd /cluster/scratch/username
/scratch on each compute node ($TMPDIR)
Shareholders can buy the space on Project and Work as much as they need, and manage access rights. Quota can be checked with lquota. The content is backed up multiple times per week.
$ cd /cluster/project/groupname
Similar to $HOME, but for groups, it is a safe, long-term storage for critical data.
$ cd /cluster/work/groupname
Similar to global scratch, but without purge, it is a fast, short-or medium-term storage for large computations.
The folder is visible only when accessed.
Groups who have purchased storage on the central NAS of ETH provided ID Systemdienste can access it on our clusters.
Groups who are operating their own NAS can export a shared file system via NFS to Euler. The user and group ID's on the NAS needs to be consistent with ETH user names and groups.
The NAS share needs to be mountable via NFSv3 (shares that only support CIFS cannot be mounted on the HPC clusters), and exported to the subnet of our HPC clusters. The NAS is then mounted automatically on our clusters under
File system comparison
|File system||Life span||Max size||Snapshots||Backup||Small files||Large files|
|$SCRATCH||2 weeks||2.5 TB||-||-||o||✓✓|
|Local /scratch||duration of job||800 GB||-||-||✓✓||o|
- Snapshots: up to 3 weeks
- Backup: up to 90 days
Data transfer with command line tools
Using scp command
Upload dummy_file from your workstation to your home directory on Euler
$ scp dummy_file firstname.lastname@example.org:
Download dummy_file from Euler to the current directory on your workstation
$ scp email@example.com:dummy_file .
Copy a directory to Euler
$ scp -r dummy_dir firstname.lastname@example.org:
Example: upload a directory with rsync
Create two files in the dummy directory and use rsync to transfer the folder
$ mkdir dummy_dir $ touch dummy_dir/dummy_file1 dummy_dir/dummy_file2 $ rsync -av dummy_dir email@example.com:dummy_dir
Data transfer with graphical tools
- User guide: Storage systems
- Too much space is used by your output files
- Best practices guide for Lustre file system