Difference between revisions of "New scratch filesystem for Euler"
| Line 1: | Line 1: | ||
==Introduction== | ==Introduction== | ||
| − | Up to now, the personal scratch directories (<tt>/cluster/scratch/$USER</tt>) and the work storage shares (<tt>/cluster/work/</tt>) were hosted on the same | + | Up to now, the personal scratch directories (<tt>/cluster/scratch/$USER</tt>) and the work storage shares (<tt>/cluster/work/</tt>) were hosted on the same storage system in Euler. By adding a new DDN SFA400NVX Lustre filesystem for the personal scratch directories, these two types of storage are now separated and independent. The new Lustre file system is based on very fast NVMe SSDs and can deliver an impressive sequential read performance of up to 48GB/s and a sequential write performance of up to 37 GB/s for all Euler partitions. |
==Migration of users personal scratch directories== | ==Migration of users personal scratch directories== | ||
| Line 8: | Line 8: | ||
* Existing directories will be migrated | * Existing directories will be migrated | ||
| − | The new file system will have the same paths as the old one, therefore no action needs to be taken on the users side. | + | The new file system will have the same paths as the old one, therefore no action needs to be taken on the users side. Note that during the migration process some users might encounter: |
| + | |||
| + | * Inability to login into Euler | ||
| + | * Jobs not actively running | ||
| + | |||
| + | This is due to the need to avoid race conditions during the migration of the directories. The above is not expected to last more than a few minutes. | ||
==Troubleshooting== | ==Troubleshooting== | ||
===Performance issues=== | ===Performance issues=== | ||
| − | In case you encounter performance issues with the new scratch file system, please contact cluster-support@id.ethz.ch | + | The new storage system is expected to deliver at least the same performance as the previous storage system. Some jobs might however benefit of faster bandwidth or storage IOPS. In case you encounter performance issues with the new scratch file system, please contact cluster-support@id.ethz.ch |
Revision as of 09:25, 6 April 2021
Contents
Introduction
Up to now, the personal scratch directories (/cluster/scratch/$USER) and the work storage shares (/cluster/work/) were hosted on the same storage system in Euler. By adding a new DDN SFA400NVX Lustre filesystem for the personal scratch directories, these two types of storage are now separated and independent. The new Lustre file system is based on very fast NVMe SSDs and can deliver an impressive sequential read performance of up to 48GB/s and a sequential write performance of up to 37 GB/s for all Euler partitions.
Migration of users personal scratch directories
The migration of the users scratch directory is transparent.
- New scratch directories will be created on the new filesystem
- Existing directories will be migrated
The new file system will have the same paths as the old one, therefore no action needs to be taken on the users side. Note that during the migration process some users might encounter:
- Inability to login into Euler
- Jobs not actively running
This is due to the need to avoid race conditions during the migration of the directories. The above is not expected to last more than a few minutes.
Troubleshooting
Performance issues
The new storage system is expected to deliver at least the same performance as the previous storage system. Some jobs might however benefit of faster bandwidth or storage IOPS. In case you encounter performance issues with the new scratch file system, please contact cluster-support@id.ethz.ch
