Difference between revisions of "Storage systems"

From ScientificComputing
Jump to: navigation, search
Line 51: Line 51:
 
<noinclude>==External storage==</noinclude><includeonly>===External storage===</includeonly>
 
<noinclude>==External storage==</noinclude><includeonly>===External storage===</includeonly>
 
<noinclude>===Central NAS===</noinclude><includeonly>====Central NAS====</includeonly>
 
<noinclude>===Central NAS===</noinclude><includeonly>====Central NAS====</includeonly>
* Mountable via NFS
+
Groups who have purchased storage on the central NAS of ETH can ask the storage group of IT services to export it to our HPC clusters. There are certain requirements that need to be fullfilled in order to use central NAS shares on our HPC clusters.
* Export to Euler subnet
+
 
* Contact Systemdienste
+
* The NAS share needs to be mountable via NFS (shares that only support CIFS cannot be mounted on the HPC clusters).
 +
* The NAS share needs to be exported to the subnet of our HPC clusters (please contact ''ID Systemdienste'', and ask them for an NFS export of your NAS share)
 +
 
 +
NAS shares are then mounted automatically when you access them. The mount-point of such a NAS share is
 +
 
 +
/nfs
 +
 
 +
When a NAS share is mounted on our HPC clusters, then it is accessible from all the compute nodes in the cluster.
 +
 
 
<noinclude>===Local NAS===</noinclude><includeonly>====Local NAS====</includeonly>
 
<noinclude>===Local NAS===</noinclude><includeonly>====Local NAS====</includeonly>
 +
Groups who are operating their own NAS, can export a shared file system via NSF to our HPC clusters.
 +
 
* NFSv3
 
* NFSv3
 
* NETHZ user names and groups
 
* NETHZ user names and groups
 
* Export to Euler subnet (put example for export file), check if load balancer is part of the Euler subnet.
 
* Export to Euler subnet (put example for export file), check if load balancer is part of the Euler subnet.
 +
 
<noinclude>==Comparison==</noinclude><includeonly>===Comparison===</includeonly>
 
<noinclude>==Comparison==</noinclude><includeonly>===Comparison===</includeonly>
 
Put table from presentation here
 
Put table from presentation here
<!--
 
{| class="wikitable"
 
|-
 
! rowspan="7" | Brutus !! Name !! Mountpoint !! Filesystem type !! Life span !! Max. size !! Backup !! Purged !! Should be used for
 
|-
 
| home || /cluster/home || NFS (ZFS) || permanent || 16 GB || yes || never || important files, medium and large files
 
|-
 
| work || /cluster/work || Panasas || 4 years ||  || no || files older than 15 days || parallel I/O, medium and large files (> 10MB)
 
|-
 
| public scratch || /cluster/scratch_xp || Lustre  || 4 years ||  || no || files older than 15 days || parallel I/O, large files (> 50MB)
 
|-
 
| shareholder scratch || /cluster/scratch_xl || Lustre || 4 years || || no || no || parallel I/O, large files (> 50MB)
 
|-
 
| local scratch || /scratch || ext3 || job || 10GB - 1TB || no || after job finishes || single node I/O, small files OK
 
|-
 
| central NAS || /nfs || || flexible ||  flexible || optional || no || data
 
|-
 
! rowspan="7" | Euler !! Name !! Mountpoint !! Filesystem type !! Life span !! Max. size !! Backup !! Purged !! Should be used for
 
|-
 
| home || /cluster/home ||  || permanent || 16 GB || yes || never || important files, medium and large files
 
|-
 
| personal scratch || /cluster/scratch || || 15 days ||  || no || files older than 15 days || parallel I/O, medium and large files (> 10MB)
 
|-
 
| project || /cluster/project ||  || 4 years || flexible || snapshots || no ||
 
|-
 
| work || /cluster/work ||  || 4 years || flexible || no  || no ||
 
|-
 
| local scratch || /scratch ||  || job || 800 GB || no || end of job  ||
 
|-
 
| central NAS || /nfs ||  || flexible || flexible || no || no ||
 
|}
 
 
'''Mount point''':<br>
 
A mount point is the path, to which a physical storage medium or system is mapped.
 
 
'''NAS''':<br>
 
''Network attached storage''  is file-level computer data storage connected to a computer network providing data access to a heterogeneous group of clients.
 
 
'''NFS''':<br>
 
''Network file system'' is a protocol, that allows the access of data via a network.
 
 
'''Scratch space''':<br>
 
Location for short term storage of data.
 
-->
 
  
 
<noinclude>==Choosing the optimal storage system==</noinclude><includeonly>===Choosing the optimal storage system===</includeonly>
 
<noinclude>==Choosing the optimal storage system==</noinclude><includeonly>===Choosing the optimal storage system===</includeonly>

Revision as of 07:01, 31 August 2016

Introduction

On our cluster, we provide multiple storage systems, which are optimized for different purposes. Since the available storage space on our clusters is limited and shared between all users, we set quotas in order to prevent single users from filling up an entire storage system with their data.

A summary of general questions about file systems, storage and file transfer can be found in our FAQ. If you have questions or encounter problems with the storage systems provided on our clusters or file transfer, then please contact cluster support.

Personal storage (everyone)

Home

On our clusters, we provide a home directory (folder) for every user that can be used for safe long term storage of important and critical data (program source, script, input file, etc.). It is created on your first login to the cluster and accessible through the path

/cluster/home/username

The path is also saved in the variable $HOME. The permissions are set that only you can access the data in your home directory and no other user. Your home directory is limited to 16 GB and a maximum of 100'000 files and directories (inodes). The content of your home is saved every hour and there is also a nightly backup (tape).

Scratch

We also provide a personal scratch directory (folder) for every user, that can be used for short-term storage of larger amounts of data. It is created, when you access it the first time through the path

/cluster/scratch/username

The path is also saved in the variable $SCRATCH. It is visible (mounted), only when you access it. If you try to access it with a graphical tool, then you need to specify the full path as it is might not visible in the /cluster/scratch top-level directory. Before you use your personal scratch directory, please carefully read the usage rules to avoid misunderstandings. The usage rules can also be displayed directly on the cluster with the following command.

cat $SCRATCH/__USAGE_RULES__

Your personal scratch directory has a disk quota of 2.5 TB and a maximum of 1'000'000 files and directories (inodes). There is no backup for the personal scratch directories and they are purged on a regular basis (see usage rules).

Group storage (sharholders only)

Project

Shareholder groups have the option to purchase additional storage inside the cluster. The project file system is designed for safe long-term storage of critical data (like the home directory). Shareholder groups can buy as much space as they need. The path for project storage is

/cluster/project/groupname

Access rights and restriction is managed by the shareholder group. We recommend to use NETHZ groups for this purpose. Backup (tape) is not included, but can be purchase optionally. If you are interested in getting more information and prices of the project storage, then please contact cluster support.

Work

Apart from project storage, shareholder groups also have the option to buy so-called work (high-performance) storage. It is optimized for I/O performance and can be used for short- or medium-term storage for large computations (like scratch, but without regular purge). Shareholders can buy as much space as they need. The path for work storage is

/cluster/work/groupname

Access rights and restriction is managed by the shareholder group. We recommend to use NETHZ groups for this purpose. The directory is visible (mounted), only when accessed. If you are interested in getting more information and prices of the work storage, then please contact cluster support.

Local scratch (on each compute node)

The compute nodes in our HPC clusters also have some local hard drives, which can be used for temporary storing data during a calculation. The main advantage of the local scratch is, that it is located directly inside the compute nodes and not attached via the network. This is very beneficial for serial, I/O-intensive applications. The path of the local scratch is

/scratch

You can either create a directory in local scratch yourself, as part of a batch job, or you can use a directory in local scratch, which is automatically created by the batch system. LSF creates a unique directory in local scratch for every job. At the end of the job, LSF is also taking care of cleaning up this directory. The path of the directory is stored in the environment variable

$TMPDIR

If you use $TMPDIR, then you need to request scratch space from the batch system.

External storage

Central NAS

Groups who have purchased storage on the central NAS of ETH can ask the storage group of IT services to export it to our HPC clusters. There are certain requirements that need to be fullfilled in order to use central NAS shares on our HPC clusters.

  • The NAS share needs to be mountable via NFS (shares that only support CIFS cannot be mounted on the HPC clusters).
  • The NAS share needs to be exported to the subnet of our HPC clusters (please contact ID Systemdienste, and ask them for an NFS export of your NAS share)

NAS shares are then mounted automatically when you access them. The mount-point of such a NAS share is

/nfs

When a NAS share is mounted on our HPC clusters, then it is accessible from all the compute nodes in the cluster.

Local NAS

Groups who are operating their own NAS, can export a shared file system via NSF to our HPC clusters.

  • NFSv3
  • NETHZ user names and groups
  • Export to Euler subnet (put example for export file), check if load balancer is part of the Euler subnet.

Comparison

Put table from presentation here

Choosing the optimal storage system

File transfer

Command line tools

Graphical tools

Quotas