Accessing Polybox From Euler

ETH Zurich provides to all its members an on-premise cloud storage service called Polybox. Although it is not possible to access it directly from Euler, it is possible to copy data from Euler to Polybox (and vice versa) using a tool called "rclone".

Setting up rclone

Since "rclone" is installed centrally on Euler, you do not need to install it yourself, but you need to configure it to access your own Polybox storage. You can find the rclone documentation online. On the cluster run the command rclone config. During the configuration, you will need to do the following:

• Choose to create a new remote typing in an “n”.
• You will be asked to name the new remote destination (e.g. "polybox").
• Afterwards you will need to configure the storage type. We use the protocol WebDAV, so you need to type in “webdav” or the number it has on the list.
• Now rclone needs to know where it has to connect to. Here you need to enter your polybox WebDAV path. This should be: https://polybox.ethz.ch/remote.php/dav/files/USERNAME/ (replace USERNAME by yours).
• Rclone needs to know now what software behind Polybox is. This is in our case ownCloud, so you will need to type in “owncloud” or the number of it on the list.
• Now it’s time to configure your user: Here you need to type in the user you use for Polybox, so it should be the same username than on the Euler cluster.
• Rclone now asks you for a password, we need this to be able to connect to Polybox, so you need to type in “y”.
• The configuration now asks you for your password, you will need to type it in twice. The password will not be shown on the terminal and it will be encrypted in the configuration file.
• This is basically it. For the bearer you can just press enter and skip the advanced configuration with “n”, keep the remote “polybox” with “y” and quit the config with “q”.

The config file rclone creates during the configuration process can be found in your home directory under the path .config/rclone/rclone.conf.

Using rclone

When you are done with the steps above you can test the configuration by listing your directories or files on Polybox. For this you can use:

• List your directories on Polybox: rclone lsd polybox:
• List your files on Polybox: rclone ls polybox:

If you can list your directories and files on Polybox with these two commands, your configuration was successful and you can upload/download files from/to Polybox.

To upload files from Euler to Polybox you will need to use a command looking like this: rclone copy /SOURCE/FOLDER/ON/EULER polybox:DESTINATION/FOLDER/ON/POLYBOX

Restrictions

• Although "rclone" is compatible with a multitude of cloud storage providers, Polybox is the only use case that we support on Euler
• Using "rclone" on Euler to transfer data from/to other cloud storage providers is strongly discouraged due to possible negative side-effects (not restricted to yourself) — don't complain to us if it destroys all your cloud data!
• "rclone" should only be used on the cluster's login nodes; do not use it in batch jobs!

Performance tips

We recommend compressing your files before uploading them (e.g. zip). It has 2 main impacts:

• Reduce the number of files to upload
• Reduce the total size of data to upload

You can upload multiple files at the same time with the option --transfers 8 (default is 4). Please do not use huge numbers as it will produce a large load on Euler and Polybox.