School Storage Arrangements - where to keep your files

We have a number of servers with large disc arrays in various parts of the School which are used to store user home directories and personal "scratch" space.

Home directory - Backed up space

As the name suggests, this is your personal file storage space that is backed up regularly - usually nightly. Backups are written to "paired" servers in remote physical locations to provide a degree of disaster resilience. Files produced by you should be stored in your home directory (mapped as "drive M:" on the Windows PCs).

e.g. /home/jbloggs/my_important_file or "My Documents\my_important_file.doc"

This data can be expected to survive, even in the event of (for example) a catastrophic fire.

Recovering from such a catastrophe is the principle reason for the backups. We may also be able to retrieve files accidentally or erroneously damaged or destroyed by their owners. We cannot promise to be able to do this. However if you do lose or damage a file, please send an email to IThelp@geos telling us where the file was, and as accurately as you can the dates and times the file was created, modified and damaged or destroyed.

Resilient scratch space

This is stored on a resilient file system. In this context, "resilient" means that there is an element of redundancy in the way the hardware is configured. This is usually a mechanism, known as RAID, which allows data to be spread over a number of discs in such a way that the failure of any one disc does not result in data loss. Data which you store on this space should be retrievable by some means (such as re-installing from an original source, re-running a model, retrieving from an occasional backup or a location on the Net, or some combination of these).

e.g. /scratch/jbloggs/my_model_output or "My Documents\scratch\my_model_output.csv"

This data can be expected to survive minor hardware failures. The data owner will be responsible for ensuring suitable retrieval options are available (e.g. re-downloading data from an online data repository) in the unlikely event of a catastrophic failure causing a loss of data.

Local Scratch space

Disposable data! Typical uses for this would be for intermediate input and output during program runs. One way of using this type of space is as a local cache of data stored elsewhere, usually to speed or ease processing.

e.g. /scratch/local/jbloggs/copy_of_my_data

This data should be expendable - it should be merely an inconvenience if it is lost. (But there is no explicit time limit on storage, nor any automatic removal scheme.)

Temporary space

Self clearing! Some disc areas are cleared when the relevant machine is rebooted. In some other cases a deliberate policy exists to dispose of items beyond a certain age

e.g. /tmp/some_temporary_data_storage