When Californians had to evacuate in the wake of the Thomas Fire in Ventura County, they grabbed what was important—family, pets, documents, photos, and other essentials.
Most didn't try to carry off their business' servers. (Nor should they!) But unless they made Disaster Recovery and Business Continuity plans beforehand, they may not have a business to come back to.
One of our customers, a general contractor headquartered in Ventura, DID plan ahead. We helped them prepare for any disaster, natural or man-made. They weathered the fire and are still going strong.
This is how we did it. Our January WOOF! is a case study in disaster recovery.
Disaster Preparations Ahead of Time
From late 2016 to mid-2017, we implemented several preparatory measures for this customer:
- Daily Backups, both on-site and mirrored to our datacenter (which is in a different geographic location from the customer's offices)
- Asset tagging on all computers & company-issued mobile devices
- Logged software licenses, incorporated into backups
Why do all this ahead of time? One, it preserves data. We can replace a computer. We cannot replace lost data. Two, it protected the business' intellectual property, their customer and accounting files, their IT hardware (as much as possible), and their ability to work.
Emergency Actions During the Disaster
When the Thomas Fire started, our customer knew the danger to their office and their trailers at job sites throughout Ventura County. They took care of evacuating staff and records, while we took care of IT preservation.
- First, we verified all their server backups. That way we knew the data was safe and recoverable.
- Next, we increased the backup frequency. By capturing as much up-to-the-minute data as possible, we minimized the risk of data loss.
- All employee personal devices were enabled for company email & communication (if they weren't already).
- We also advised all employees to keep their laptops with them, and not to leave them at the office when visiting customers or leaving for the day.
- We kept an eye on their systems with network monitoring. Any sign of fire damage and we could switch to Action #6 (below).
- As a precautionary measure, we set up standby servers (a “hot site”) in PlanetMagpie's datacenter, hundreds of miles from the fire location.
- Finally, we collected any important IT hardware & software left in the customers' offices. We had staff scheduled to visit the site for support anyway, so we used the opportunity to safely collect & remove important computers as well as servers.
While the customer's Ventura office experienced a number of power outages due to burned-down power lines, we opted to wait for the Ventura PG&E substation to burn down before flipping all computers to use the hot site servers. Luckily, the substation was spared.
The Post-Thomas Fire Recovery Process
Once the fire was safely under control, we assisted with rebuilding the customer's workflow.
- First, returning the collected IT hardware/software to the office locations.
- Next, restoring backups for any devices lost or damaged.
- Testing network connectivity to confirm everyone can do their work, and repairing any damaged network connections.
- Reducing the frequency of server backups back to once per day.
- Finally, resuming normal IT support operations.
How it All Worked Out
The company had to evacuate from its offices and many employees from their homes as well. The fire came so close that it burned down the owner's backyard shed!
The main office did escape the fire. Other businesses only blocks away suffered total destruction.
So was all of this effort wasted? Not at all. In fact, it's a net positive for the company. Now they know they can respond quickly to a disaster. They know the company can survive disruption, and resume operations shortly thereafter.
Most importantly, they know their data is safe in the event of disaster.
The Thomas Fire now stands as the largest in California history. We're all saddened by the losses. At the same time, we're glad we could help at least one business—and its people—recover from the disaster.