Copying data

This page describes tools for copying/moving data between local and remote locations.

For transparently accessing remote directories locally, see Mounting remote filesystems locally.

Why we use csnhr.nhr.fau.de in the examples.

We use dialog server csnhr.nhr.fau.de as remote host in all examples, as

• our default configuration tunnels every SSH connection through csnhr.nhr.fau.de and using a different host would just create unnecessary load and network traffic,
• all filesystems like $HOME, $HPCVAULT, $WORK are mounted there.

scp

scp is a command line tool to copy data between locations that can be local or remote.

The general syntax is

scp [<options>] <source> [<source> [...]] <destination>

where <source> can appear multiple times. <source> and <destination> can be:

• a local path
• a remote path in the form of <remote-host>:[<remote path>]
  • Without <remote path> the remote $HOME directory is assumed.
  • If <remote path> is a relative path, i.e. it does not start with a /, then the remote target is $HOME/<remote path>.

Typical options (<options>) include

• -r: recursively copy a directory, mandatory for directories.
• -p: preserve modification times and access rights.

Prerequisites for using scp with our systems:

• Your SSH connection is configured and you are using SSH keys.
• You successfully can connect to dialog server csnhr.nhr.fau.de with the command ssh csnhr.nhr.fau.de.

Copy from your local machine to remote

scp -r data_dir1 ~/other_data_dir2 /mnt/data3 csnhr.nhr.fau.de:data

Copies the three directories data_dir1, ~/other_data_dir2, and /mnt/data3 into the directory $HOME/data on csnhr.nhr.fau.de. As data denotes a relative path, the remote path is prefixed with $HOME. As we copy directories, the -r option is required to copy the directories content recursively.

scp -r simulation csnhr.nhr.fau.de:/home/atuin/<group>/<user>

Copies the directory simulation to the remote $WORK directory /home/atuin/<group>/<user> on csnhr.nhr.fau.de. You have to adjust the path according to your $WORK directory. As we copy directories, the -r option is required to copy the directories content recursively.

Copy from remote to your local machine

scp -r csnhr.nhr.fau.de:results/2023 /mnt/backup/

Copies the directory $HOME/results/2023 to the directory /mnt/backup/ on your local machine. As we copy directories, the -r option is required to copy the directories content recursively.

Using wildcards in remote locations

You can specify wildcards like * or ? for remote locations, but you have to escape them with a backslash (\):

scp csnhr.nhr.fau.de:results/2023/\*.dat .

This copies all files from csnhr.nhr.fau.de in the directory $HOME/results/2023 ending in .dat to the local current directory.

rsync

rsync is like scp a command line to for transferring files between local and remote locations. rsync provides more features than scp, e.g. it resumes transfers that were interrupted.

rsync [<options>] <source> [<source> [...]] <destination>

where <source> can appear multiple times. <source> and <destination> can be:

• a local path
• a remote path in the form of <remote-host>:[<remote path>]
  • Without <remote path> the remote $HOME directory is assumed.
  • If <remote path> is a relative path, i.e. it does not start with a /, then the remote target is $HOME/<remote path>.

Typical options (<options>) include

• -a: archive mode, preserve attributes, permissions, symbol links, ...
• --append-verify: resume partially transferred files instead of retransmitting them.
• -p: preserve modification times and access rights.
• -v: verbose output.
• -z: compress data during transfer
• -zc <alg>: use compression algorithm <alg>
• -zl <num>: use compression level <num>

Using compression -z when transferring files between NHR@FAU systems might increase duration

Prerequisites for using rsync with our systems:

• Your SSH connection is configured and you are using SSH keys.
• You successfully can connect to csnhr.nhr.fau.de with the command ssh csnhr.nhr.fau.de.

Copy local directory to remote, note the trailing slash in the source location

rsync -avz samples/2023 csnhr.nhr.fau.de:samples

Copy the local directory samples/2023 to the remote directory $HOME/samples/2023. This means a file samples/2023/some-file will be copied to $HOME/samples/2023/some-file.

rsync -avz samples/2023/ csnhr.nhr.fau.de:samples

Copy the content of local directory samples/2023 to the remote directory $HOME/samples/. This means a file samples/2023/some-file will be copied to $HOME/samples/some-file.

Copy remote directory to local location

rsync -avz csnhr.nhr.fau.de:/home/atuin/<group>/<user>/results ~/simulation

Copy the remote directory /home/atuin/<group>/<user>/results to the local directory $HOME/simulation. The directory result will be located under $HOME/simulation.

rsync -avz csnhr.nhr.fau.de:/home/atuin/<group>/<user>/results/\*.dat ~/simulation

This copies all files from csnhr.nhr.fau.de in the directory /home/atuin/<group>/<user>/results ending in .dat to the local current directory ~/simulation.

WinSCP

WinSCP is graphical SFTP client for Windows where you can copy or move data between a local and remote machine by drag on drop.

Connecting

After installation on your system, open WinSCP and click on New Tab. The Login dialog opens, where you specify the connection properties.

Login dialog	Advanced Site Settings dialog

In the Login dialog:

1. For Host name enter csnhr.nhr.fau.de.
2. For User name enter your HPC user name.

3. Click on the button Advanced..., the Advanced Site Settings dialog opens.

   1. In the left tree select Authentication.

   2. Under Private key file enter the path to your private SSH key you use for connecting to our systems.

      Depending on the type of your private SSH key, WinSCP might convert it into a different format first.

   3. Exit the dialog with clicking OK.

4. Optional: by clicking Save you can save the connection settings.

5. Click Login to connect to csnhr.nhr.fau.de.

Usage

After connecting you will be presented window with two panels. Left is your local directory structure, right the remote directory structure. You can now copy or move data between the local and remote system by drag and drop.