Belgium data center struck causing errors on Google Compute Engine persistent disks
Four lightning strikes affecting the power grid around the Google Compute Engine data center in St. Ghislain, Belgium, caused a small data loss and a number of persistent disk errors.
Google uses persistent disks as the primary storage method for virtual machine instances; the disks are independent of the machine, so the data is retained after a virtual machine instance is ended.
The incident was tracked in a post to the company’s Cloud Status page.
“Errors occurred on a small proportion of Google Compute Engine persistent disks. … The affected disks sporadically returned I/O errors to their attached GCE instances, and also typically returned errors for management operations such as snapshot creation,” the post noted. “In a very small fraction of cases … there was permanent data loss.”
For the Western Europe zone covered by the data center, the amount of data loss was calculated as 0.000001% of total.
While not going so far as to spell out how the lightning affected ops, Google’s cloud team offered reassurances to customers.
“Google takes availability very seriously, and the durability of storage is our highest priority,” the company said. “We apologize to all our customers who were affected by this exceptional incident. We have conducted a thorough analysis of the issue, in which we identified several contributory factors across the full range of our hardware and software technology stack, and we are working to improve these to maximize the reliability of GCE’s whole storage layer.”
The St. Ghislain data center is one of four Google data centers in Europe. The others are in Hamina, Finland; Dublin; and Eemshaven, Netherlands. The company has Asian data centers in Changhua County, Taiwan and Singapore. In the Americas, Google data centers are located in Berkeley County, S.C.; Council Bluffs, Iowa; Douglas County, Ga.; Jackson County, Ala.; Lenoir, N.C.; The Dalles, Oregon; and Quilicura, Chile.
The St. Ghislain facility became fully operational in September 2010, following an April 2007 announcement of the 250 million euros ($280 million) building. In 2013, the company announced additional investment of 300 million euros to expand the data center to keep up with global demand.
