New scratch filesystem for Euler

From ScientificComputing
Revision as of 08:45, 6 April 2021 by Sfux (talk | contribs)

Jump to: navigation, search

Introduction

Up to now, the personal scratch directories

 /cluster/scratch/$USER

and the work storage shares

/cluster/work/

were hosted on the same storage system in Euler. By adding a new DDN ES400NV Lustre filesystem for the personal scratch directories, these two types of storage are now separated and independent.

New Lustre filesystem

The new Lustre file system is based on very fast NVMe SSDs and can deliver an impressive sequential read performance of up to 48GB/s and a sequential write performance of up to 37 GB/s for all Euler partitions.

Migration of users personal scratch directories

The migration of the users scratch directory is transparent.

  • New scratch directories will be created on the new filesystem
  • Existing directories will be migrated

The new file system will have the same paths as the old one, therefore no action needs to be taken on the users side. Note that during the migration process some users might encounter:

  • Inability to login into Euler
  • Jobs not actively running

This is due to the need to avoid race conditions during the migration of the directories. The above is not expected to last more than a few minutes.

Troubleshooting

Performance issues

The new storage system is expected to deliver at least the same performance as the previous storage system. Some jobs might however benefit of faster bandwidth or storage IOPS. In case you encounter performance issues with the new scratch file system, please contact cluster-support@id.ethz.ch