Data Backup


This manual describes data backup options and how to use them. If you would like to use other technologies, please mention this in the End Of Run Summary by accessing it through the User Portal and your suggestions will be taken into consideration.

Remote Network File Transfer

Dedicated Backup Computer

  • smbcopy.slac.stanford.edu:

All data transfers should be carried using this dedicated computer for backup.

Warning: Do not initiate data transfers on any other computer as It will adversely affect data collection or data processing.

File Compression

Compressing files will reduce the overall time for file transfer. Data analysis files compress well, as do image files from ADSC and mar CCD detectors.

Files from the Pilatus and Eiger detectors in CBF format and files from the MAR345 image plate scanner are by default already compressed.

Commonly used compression programs are:

  • gzip
  • bzip2

bzip2 compress a little bit better at the expense of longer execution time. To compress all image files in your current directory type:

gzip  *.img

Data Transfer

Data files at SSRL can be transferred to a remote computer (i.e a laptop on the visitor network or a computer at your home lab) with any file transfer program that uses a secure transport protocol, such as:

  • scp             (Linux command line)
  • sftp             (lLnux command line)
  • rsync          (Linux command line)
  • gftp             (Linux GUI)
  • WinSCP      (Windows GUI)
  • FilZilla.        (Windows GUI)

Warning: Make sure that you have enough disk space to store all your data at the target location before you start the file transfer. To get the size of a directory in kilobytes use the command:

du  -sk  image_directory

Some example commands that can be used to transfer datasets:

scp  lyso_*.img  username@my_home_computer:my_home_directory/

Recursive transfer with compression:

scp  -r  -C  image_directory  username@my_home_computer:my_home_directory/

rsync compares the directory content at two different locations and only copies what is needed to "synchronize" the two.

rsync  -auvP  image_directory  username@my_home_computer: my_home_directory/

The first time the command is issued, everything in image_directory will be copied to my_home_directory. If you repeat the command after collecting additional data, only the new or modified files will be transferred, keeping the directories "synchronized".

On-Site Laptop or Hard Drive

Backing up directly to a FireWire  (IEEE1394) or USB connected laptop or hard disk is a very fast and convenient way to store data.

Supported File Systems

The standard Linux file systems ext2, ext3, ext4 or xfs as well as the Windows FAT32 file system are supported

Note: Windows NTFS and Mac HFS are not supported.

Warning: File names on FAT32 are case insensitive. File or directory names that only differ by "case" will be overwritten when copying from a Linux computer to a FAT32 file system.

Connecting the Drive

The Linux workstations at each beamline are configured to allow connections to external hard disks via Fire Wire or USB. A cable labeled "Fire Wire" comes out of the console station next to the monitor. The tool board has a spare Fire Wire cable and a USB cable that can be used with the workstations at the station.

Power up the disk and connect the cable, wait ~10 s to let the operating system detect the disk. The Window Manager will create an icon on the desktop and in the File Manager for each file system (or volume) on the disk (Fig. 1).

Figure 1. Examples of a FAT32 volume, "FAT32_VOL", and an unlabeled volume, "103G Volume".

The name of the icon is same as the volume label if one exists, "FAT32_VOL" (Fig. 1).  An unlabeled volume will get an icon name that relates to the size of the volume, for example "103G Volume" (Fig. 1).

To mount the volume, right click on the icon and select "Mount Volume" from the drop down menu.

Once a labeled volume is mounted, directories can be accessed on /media:

/media/FAT32_VOL/

Unlabeled volume use the following pattern, depending on the number of unlabeled volumes:

/media/disk/
/media/disk2/
/media/disk3/
...