Difference between revisions of "Storage systems"

From ScientificComputing
Jump to: navigation, search
Line 11: Line 11:
 
|   || home || /cluster/home || NFS (ZFS) || permanent || 16 GB || yes || never || important files, medium and large files
 
|   || home || /cluster/home || NFS (ZFS) || permanent || 16 GB || yes || never || important files, medium and large files
 
|-
 
|-
|   || work || /cluster/work || Panasas || 4 years || 100 GB || no || files older than 15 days || parallel I/O, medium and large files (> 10MB)
+
|   || work || /cluster/work || Panasas || 4 years || || no || files older than 15 days || parallel I/O, medium and large files (> 10MB)
 
|-
 
|-
 
|   || public scratch || /cluster/scratch_xp || Lustre  || 4 years ||  || no || files older than 15 days || parallel I/O, large files (> 50MB)
 
|   || public scratch || /cluster/scratch_xp || Lustre  || 4 years ||  || no || files older than 15 days || parallel I/O, large files (> 50MB)
Line 17: Line 17:
 
|   || shareholder scratch || /cluster/scratch_xl || Lustre || 4 years || || no || no || parallel I/O, large files (> 50MB)
 
|   || shareholder scratch || /cluster/scratch_xl || Lustre || 4 years || || no || no || parallel I/O, large files (> 50MB)
 
|-
 
|-
|   || local scratch || /scratch || ext3 || job || 10GB - 1TB || no || after your job finishes || single node I/O, small files OK
+
|   || local scratch || /scratch || ext3 || job || 10GB - 1TB || no || after job finishes || single node I/O, small files OK
 
|-
 
|-
 
|   || central NAS || /nfs || || flexible ||  flexible || optional || no || data
 
|   || central NAS || /nfs || || flexible ||  flexible || optional || no || data
Line 25: Line 25:
 
|   || home || /cluster/home ||  || permanent || 16 GB || yes || never || important files, medium and large files
 
|   || home || /cluster/home ||  || permanent || 16 GB || yes || never || important files, medium and large files
 
|-
 
|-
|   || personal scratch || /cluster/scratch || || 15 days ||  || no || 15 days || parallel I/O, medium and large files (> 10MB)
+
|   || personal scratch || /cluster/scratch || || 15 days ||  || no || files older than 15 days || parallel I/O, medium and large files (> 10MB)
 
|-
 
|-
|   || project || /cluster/project ||  || 4 years || flexible || ||  ||
+
|   || project || /cluster/project ||  || 4 years || flexible || no ||  ||
 
|-
 
|-
|   || work || /cluster/work ||  || 4 years || flexible ||  ||  ||
+
|   || work || /cluster/work ||  || 4 years || flexible || no ||  ||
 
|-
 
|-
|   || local scratch || /scratch ||  || job || 800 GB || ||  ||  
+
|   || local scratch || /scratch ||  || job || 800 GB || end of job ||  ||  
 
|-
 
|-
|   || central NAS || /nfs ||  || flexible || flexible || No || ||  
+
|   || central NAS || /nfs ||  || flexible || flexible || no || ||  
 
|}
 
|}
  

Revision as of 11:37, 30 August 2016

Introduction

On our cluster, we provide multiple storage systems, which are optimized for different purposes. On this page, we try to give you an overview on the different storage systems and provide you all the information that is needed to choose the most optimal file system for your purpose. Some of the storage systems are optimized for large files and high performance, whereas others can also deal with small files. Some are for permanent storage of data (homes) whereas the scratch file systems are only for temporary storage of data. Since the available storage space on our clusters is limited and shared between all users, we set quotas in order to prevent single users from filling up an entire storage system with their data. A summary of general questions about file systems and storage can be found in our FAQ. If you have questions or encounter problems with the storage systems provided on our clusters, then please contact cluster support.

Available storage systems

First of all, we would like to start with an overview on the available storage systems in Brutus. The description of the storage systems contains link that redirect you to more detailed wiki pages on these systems.

Brutus Name Mountpoint Filesystem type Life span Max. size Backup Purged Should be used for
  home /cluster/home NFS (ZFS) permanent 16 GB yes never important files, medium and large files
  work /cluster/work Panasas 4 years no files older than 15 days parallel I/O, medium and large files (> 10MB)
  public scratch /cluster/scratch_xp Lustre 4 years no files older than 15 days parallel I/O, large files (> 50MB)
  shareholder scratch /cluster/scratch_xl Lustre 4 years no no parallel I/O, large files (> 50MB)
  local scratch /scratch ext3 job 10GB - 1TB no after job finishes single node I/O, small files OK
  central NAS /nfs flexible flexible optional no data
Euler Name Mountpoint Filesystem type Life span Max. size Backup Purged Should be used for
  home /cluster/home permanent 16 GB yes never important files, medium and large files
  personal scratch /cluster/scratch 15 days no files older than 15 days parallel I/O, medium and large files (> 10MB)
  project /cluster/project 4 years flexible no
  work /cluster/work 4 years flexible no
  local scratch /scratch job 800 GB end of job
  central NAS /nfs flexible flexible no

Central storage from ID storage

You can buy central NAS shares from the ID storage group. These shares are also accessible on our clusters. When you access your central NAS share on one of our cluster, then it is mounted automatically. Please make sure that the permissions on the NAS share and the NFS export are done properly.

User storage systems

If you have your own NAS storage, managed by yourself, then you can also mount it on our clusters, but we do not provide any support for it. We only support NAS shares that are managed by the ID storage group.

Choosing the optimal storage system

File transfer

Quotas