SSRL Weathers Power Outage Caused by Lightning by John Schmerge and Tom Rabedeau Friday the 13th arrived about 90 minutes early on Thursday night when a rare Bay Area electrical storm crashed the 230-kV power feed to SLAC. Nestled safely at home, such a power loss represents an inconvenience, perhaps accompanied by a sense of awe for nature's fireworks display. A significant power loss at a light source, however, represents much more than an inconvenient loss of beam time. The first order of business for SSRL Duty Operator Wes Leonard was to ensure staff and users were safely evacuated from the now dark experimental facility, illuminated by emergency lights only. This accomplished, the DO's focus turned to beam line systems protection. Leading the list of concerns were the 11 LN-cooled monochromators which are quite sensitive to loss of LN coolant flow. Fortunately, the two emergency diesel generators, which were funded and installed as part of an ARRA project in 2010, started flawlessly and provided "life sustaining" power to the LN-cooled monochromator heat exchange systems as well as other crucial systems, such as the experimental floor oxygen deficiency warning systems and key lift pumps. As night turned to day Friday morning a skeletal SSRL staff arrived to monitor and maintain systems and participate in the SLAC-wide plan for the site-wide power recovery effort. With the plan developed by mid-day Friday, it remained for PG&E to restore the 230-kV power feed, which was accomplished Friday evening. Per plan, out of concern for staff and system safety, the skeletal night staff monitored systems and further redistribution of the power was deferred until a full SLAC recovery team arrived with the sun Saturday morning. The SPEAR3 complex was targeted for early power restoration. SPEAR3 began the recovery process minutes after power was re-established around 10:30 AM on Saturday April 14th. Personnel were called in to restore the control system, search and secure the accelerator tunnels, monitor the vacuum system, restart water systems and checkout accelerator power systems. These systems were fully recovered by early evening and the only equipment damage discovered at that time was a failed UPS in one of the computer systems in B137. Accelerator system managers were then alerted to monitor their systems and turn them over to operations after check outs were completed. All checkouts were complete in the booster LINAC a few hours later and beam was established at the LINAC exit by 10 PM. By Sunday morning the booster checkouts were finished and beam was established in the booster at 12:50 PM. By Sunday afternoon the final interlocks in SPEAR3 were cleared and the RF system could be turned on. Once beam was established in SPEAR3, an orbit check was done and finally SPEAR3 was filled to 350 mA by 7:30 PM. The only accelerator equipment that was damaged and needed replacement was a BL13 EPU row phase encoder. This was accomplished during the already scheduled maintenance day on Monday the 16th and additional maintenance was completed that could not be delayed. In parallel with this SPEAR3 accelerator systems recovery effort, beam line staff commenced beam line system recovery with the power delivery Saturday morning. In a flurry of activity, vacuum staff and the DO restarted beam line vacuum systems, monitored LN-cooled monochromator heat exchange systems, and cleared a myriad of machine protection system faults while other staff attended key experimental systems and network issues. This recovery effort continued through Sunday when additional staff attended to control processes that failed to restart cleanly as well as experimental end station equipment recovery. Monday the beam line personnel protection system functionality was revalidated. An abbreviated accelerator physics schedule was conducted Monday night through Tuesday morning and beam delivered to beam lines by Tuesday at 6 AM. Some user experiments began by Tuesday mid-day and by Wednesday all beam lines were up and taking data. "All's well that ends well" - ShakeSPEAR This article originally appeared in the April 2012 edition of SSRL Headline News.