Just like on any other Linux system you have a home directory on each of our systems, which is located under
/home/<user>. This is the default directory for all your data and it becomes your current working directory when you connect to the cluster. Your home directory is the same from all login and compute nodes of the corresponding cluster.
Home directories are not backed up regularly.
On Bender, your home directory has a quota (maximum allowed size) of 100 GB. Once you exceed 90% of that threshold, you will get an error message when creating additional files. You have to get back under the 90% limit within 7 days in this case.
You can check how much disk space you are using with the command
We may grant exceptions to this rule if we deem it justified. If you absolutely cannot do your computations with the 100 GB limit, please contact email@example.com and describe your problem. We will then consider increasing your quota.
On Bonna and Marvin, only you have read and write access to your own home directory. You can change permissions of files and directories by using the
We strongly advise against giving write access to your home directory to people you do not know.
On our systems, home directories are automatically mounted whenever they are needed. This means that they might not always be visible with an
ls call, however you can still
cd into them (provided the owner has set the permissions accordingly). The owner does not need to be logged in for this.
On Bender, your home directory is the primary location for all data from your computations. However, if you need higher file I/O speeds, you may use the local SSDs on the Bender compute nodes.
For every job, a temporary directory is automatically created and you can access it at the path
$USER is your user name and
$SLURM_JOB_ID is the numerical job ID of the job (both can be retrieved within a job from environment variables with the same name). For example, a job with ID 23111 of user
kuckertz would create a dircetory named
All data residing in the temporary directory will be deleted at the end of the job!
See the Bonna wiki here.
None of our systems have enough storage space to store your research data for longer periods of time. Please refer to our colleagues who manage the research data infrastructure for long-tem storage options.
You can transfer files from and to any of our systems via any method that works via SSH.
OpenSSH comes with the command
scp (secure copy) which is used to securely copy files and folders back and forth between your local machine and a cluster through the SSH network. In simplified syntax, with the
scp command you copy
<source> data and create
<target> data with the exact same content as the original, like this:
$ scp <source> <target>
For example, to copy a file
foobar.sh from the current working directory of your local machine into your home directory on the cluster, type:
$ scp ./foobar.sh <user>@<cluster-address>:~/foobar.sh
If you instead want to copy a file
foobar.sh from your home directory on the cluster into the current working directory of your local system, just specify source and target files the other way around like this:
$ scp <user>@<cluster-address>:~/foobar.sh ./foobar.sh
Obviously, it is mandatory that you specify the exact name of the
<source> file you want to copy. However, you are able to name the
<target> file as you like, thus copying and renaming data simultaneously.
Transfering whole directories works exactly the same except that you need to add the
-r flag for recursively copying entire directories. In the following example we first copy a directory
foobar onto the cluster and then back to your local machine:
$ scp -r ./foobar <user>@<cluster-address>:~
$ scp -r <user>@<cluster-address>:~/foobar .
A more sophisticated approach to transfer data is the use of the rsync utility with its
rsync command. The simplified syntax and its use in general is the same as for the
$ rsync <source> <target>
If you would like to copy a directory recursively you could add the
-r flag as is described above. However, the advantage of the
rsync command lies in its capability for you to be able to comfortably archive and compress your data before the transfering process takes place. This method is more efficient and much more time-saving than copying irectories raw, especially when you need to transfer lots of data. For this purpose, let us assume that you want to transfer a directory
foobar from the current working directory of your local system onto your home directory of a cluster. One way using the
rsync command could be as follows:
$ rsync -azv ./foobar <user>@<cluster-address>:~
In this example, we use three options:
-a for archiving,
-z for compressing, and an additional
-v to print verbose information about the transfering process. You do not need to use
-r in this case, because an archived directory will be interpreted as a single file. After the transfering process has finished,
rsync will automatically uncompress and extract the archived directory back to its intial state.