UNIX System Administration Handbook - Evi Nemeth [108]
Limit activity during dumps
Filesystem activity should be limited during dumps because changes can cause dump to make mistakes. You can limit activity either by doing the dumps when few active users are around (at night or on weekends) or by making the filesystem accessible only to dump.
This precaution sounds fine in theory, but it is rarely practiced. Users want 24/7 access to all filesystems. These days it is impossible to do a backup with no disk activity.
Filers such as Network Appliance’s F700 series provide on-line backups with snapshots of the filesystem at regular, tunable intervals. This feature enables safe backups to be made of an active filesystem and is one of the important advantages of using a dedicated filer.
See page 503 for more information about filers.
Check your tapes
We’ve heard many horror stories about system administrators that did not discover problems with their dump regime until after a serious system failure. It is essential that you continually monitor your backup procedure and verify that it is functioning correctly. Operator error ruins more dumps than any other problem.
The first check is to have your dump software attempt to reread tapes immediately after it has finished dumping. Scanning a tape to verify that it contains the expected number of files is a good check. It’s best if every tape is scanned, but this no longer seems practical for a large organization that uses hundreds of tapes per day. A random sample would be most prudent in this environment.
It is often useful to run restore t to generate a table of contents for each filesystem and to store the results on disk. These catalogs should be named in a way that relates them to the appropriate tape; for example, ocean:usr.Jan.13. A week’s worth of these records make it easy to discover what tape a lost file is on. You just grep for the filename and pick the newest instance.
See page 180 for more information about restore.
In addition to providing a catalog of tapes, a successful restore t indicates that the dump is OK and that you will probably be able to read the tape when you need to. A quick attempt to restore a random file will give you even more confidence in your ability to restore from that tape.3
You should periodically attempt to restore from various tapes to make sure that it is still possible to do so. Every so often, try to restore from an old (months or years) dump tape. Drives have been known to wander out of alignment over time and to become unable to read their old tapes. The tapes can be recovered by a company that specializes in this service, but it will be expensive.
A related check is to verify that you can read the tapes on hardware other than your own. If your machine room burns, it does not do much good to know that the dump tapes could have been read on a tape drive that has now been destroyed.
Consider the following story, which circulated through the grapevine a couple of years ago: A major research firm in California had an operator who was too busy hacking to do dumps. He opened tapes, labeled them, and filed them without ever putting any data on them. This charade went on for two or three months until someone insisted on having a file restored. What happened to him? Fired? No, he was transferred internally, but was eventually arrested and convicted on unrelated electronic fraud charges. Rumor has it he got 40 years.
Develop a tape life cycle
Tapes have a finite life. It’s great to recycle your media, but be sure to abide by the manufacturer’s recommendations regarding the life of tapes. Most manufacturers quantify this life in terms of the number of passes that a tape can stand: a backup, a restore, and an mt fsf (file skip forward) each represent one pass.
Design your data for backups
With disks so cheap and new storage architectures so reliable, it’s tempting not to back up all your data. A sensible storage architecture—designed rather than grown willy nilly as disk needs increase—can do a lot