New scratch filesystem for Euler
Contents
Introduction
Up to now, the personal scratch directories (/cluster/scratch/$USER) and the work storage shares (/cluster/work/) were hosted on the same storage system in Euler. By adding a new DDN ES400NV Lustre filesystem for the personal scratch directories, these two types of storage are now separated and independent.
The new Lustre file system is based on very fast NVMe drives and can deliver an impressive sequential read performance of up to 48GB/s and a sequential write performance of up to 37 GB/s for all Euler partitions. Besides the bandwidth and thanks to its flash technology the system can sustain a very high number of IO operations in parallel.
Migration of users personal scratch directories
The migration of the users scratch directory is transparent.
- New scratch directories will be created on the new filesystem
- Existing directories will be migrated
The new file system will have the same paths as the old one, therefore no action needs to be taken on the users side. Note that during the migration process some users might encounter:
- Inability to login into Euler
- Jobs not actively running
This is due to the need to avoid race conditions during the migration of the directories. The above is not expected to last more than a few minutes.
Troubleshooting
Performance issues
The new storage system is expected to deliver at least the same performance as the previous storage system. Some jobs might however benefit of faster bandwidth or storage IOPS. In case you encounter performance issues with the new scratch file system, please contact cluster-support@id.ethz.ch
