Where Were You When the Lights Went Out

Where Were You When the Lights Went Out

Where Were You When the Lights Went Out ? (Disaster Contingency) By: Larry Lemieux May 17, 2004 Where Were You When the Lights Went Out? Overview The blackout the big one Great expectations - 24 x 7 Disaster Contingency IT infrastructure Questions May 17, 2004 Where Were You When the Lights Went Out? The Event:

The Great Northeast Blackout A single transmission line from the Niagara generating station tripped (opened). Within 2.5 seconds, five other transmission lines became overloaded. The Northeast Power System became unstable within 4 seconds. Outages occurred throughout New York, Ontario, most of New England, and parts of New Jersey and Pennsylvania. The massive blackout left 30 million people without electricity. This was "the big one" The design and operation of electric utility systems changed after that, due to the lessons that were learned from this event.

May 17, 2004 Where Were You When the Lights Went Out? The Event: The Great Northeast Blackout of November 9, 1965 A single transmission line from the Niagara generating station tripped (opened). Within 2.5 seconds, five other transmission lines became overloaded. The Northeast Power System became unstable within 4 seconds. Outages occurred throughout New York, Ontario, most of New England, and parts of New Jersey and Pennsylvania.

The massive blackout left 30 million people without electricity. This was "the big one" The design and operation of electric utility systems changed after that, due to the lessons that were learned from this event. May 17, 2004 Where Were You When the Lights Went Out? CBC News: Aug 20, 2003 At 4:11 p.m. on Aug. 14, 2003, Ontario and much of the Northeastern U.S. were hit by the largest blackout in North America's history. Electricity was cut to 50 million people, bringing darkness to customers from New York to Toronto to North Bay. Ontario state of emergency to end Friday night The Ontario government will end its state of emergency at 8:00 p.m. local time Friday (Aug 22), said the province's premier.

May 17, 2004 Where Were You When the Lights Went Out? Where Were You When the Lights Went Out? The title of a 1968 movie staring Doris Day and Robert Morse Comedy based on the1965 blackout. May 17, 2004 Where Were You When the Lights Went Out? Where Were You When the Lights Went Out (Aug 14th)?

Physically? From a Disaster Contingency point of view? How ready were you to deal with the blackout at your campus? Did we learn anything? Are we better prepared for Summer 2004? May 17, 2004 Where Were You When the Lights Went Out? Great Expectations May 17, 2004 Where Were You When the Lights Went Out? Our Clients (at Ryerson)

16,000 full time students 28,000 C.E. 3,000 D.E. 3,500 employees Alumni Researchers Applicants / prospects / parents General public May 17, 2004 Where Were You When the Lights Went Out? Some Services

E-mail Course Management Corporate web site HR-Payroll SIS Electronic faxing Network services ISP access Backup services Security services (f/w, anti-virus, anti-spam) May 17, 2004 Where Were You When the Lights Went Out?

Expectations Access: anytime, anywhere Wireless: Pagers Cell phones Blackberrys Palms Tablets Laptops Self-service in real-time Want it ALL, want it NOW ! May 17, 2004

Where Were You When the Lights Went Out? Expectations Students want: Empowered Self-service Interactive Web access (for school, for fun, for business) Intuitive interfaces Immediate information 24 x 7 Little tolerance for poor service May 17, 2004

Where Were You When the Lights Went Out? How do you provide 24 x 7? Technology to the rescue Fault Tolerant High Availability Server Clustering Distributed RDBMS Redundant components Hot Backups Load Balancers May 17, 2004

Redundant Network Multiple ISPs RAID UPS Auxiliary power Generators Firewalls End User - my.ryerson.ca Technology to the rescue Internet /RIN

Outer Firewall Loadbalancer DMZ Portal Server 1 Portal Server 2 LDAP Inner Firewall SSL-Accelerator Prod Web & Appl. Server 1 Loadbalancer Prod Web & Appl. Server 2 File Server

Prod Web & Appl. Server 3 Test Crystal Server Test Web & Appl. Server Prod Crystal Server Trusted Firewall (non-routable) H/A Interface ProdDatabase Server 192.168.x.x Cognos & Reporting Server

Test Cognos & DB Server SAN SAN May 17, 2004 D is a s te r S ite R m : P O D -B 4 8 -O DARS Server Where Were You When the Lights Went Out? Issues with 24x7 Expensive Complex Difficult to maintain

Specialized h/w & s/w Specialized staff & skills Difficult to replicate for test & disaster sites Hot backups - no restores Oxymoron phrase: A fault-tolerant system has more points of failure! May 17, 2004 Where Were You When the Lights Went Out? Expectations (24x7) Students want it

Employees want it Departments want it So, my executive wants it too Because my executive wants it, they have to provide funding May 17, 2004 Where Were You When the Lights Went Out? Types of Outages (Things that keep me up at night)

Hardware failure Software failure Human failure System capacity / limitations Brown out, Black out Sabotage (employees, hackers, viruses) Disasters (natural or man made) Accidents (construction, cleaning staff) Scheduled outages (min 8 hrs) May 17, 2004 Where Were You When the Lights Went Out? Great Expectations Anytime Anywhere Nothing else will do May 17, 2004

Where Were You When the Lights Went Out? The Evolution of Disaster Contingency at Ryerson May 17, 2004 Where Were You When the Lights Went Out? Disaster Contingency Definition (At Ryerson) Service is designated as Mission Critical Service is designated as needing D.C. Service expected to be down > 2 days

Activation Approval: 2 Exec VPs Handles 3 situations: Loss of Main Computer Room Loss of Network Services Loss of department offices/building May 17, 2004 Where Were You When the Lights Went Out? DC Evolution Pre-1990s:

Mainframes (MVS & VM) Expensive Prod & Test appln on single mainframe Disaster Contingency = off-site backups Explored reciprocal arrangements with UofT & York and looked at outsourcing Could not afford disaster site to re-instate services Issues of proprietary hw/sw May 17, 2004 Where Were You When the Lights Went Out? DC Evolution Mid-1990s:

Migrate to Unix platform Less expensive More open architecture & standards Prod & test appln on separate servers 400 times smaller (server & disk) May 17, 2004 Where Were You When the Lights Went Out? DC Evolution Late-1990s Purchased and installed HR/Payroll Must GUARANTEE employees get paid

Need Disaster Site and Contingency Decided to move TEST server to Telecom room Telecom room: PBX (phones and voice-mail) Secure and part of CCS On auxiliary power (incl A/C) In different building External Auditors review and approve May 17, 2004 Where Were You When the Lights Went Out? 1st DC Plan is Reality June 1999: HR/Payroll goes live

Scope / issues: Set expectations (deliverables, timeframe, performance) Roles & responsibilities of Disaster Team Documentation: Procedures Emergency contact info of team members Step-by-step instructions Various disaster scenarios

Exercise The Plan annually Technical instructions tested by different staff User Department sign-off May 17, 2004 Where Were You When the Lights Went Out? Procedure Retrieve most recent backup tape from AB50B storage. Restore the following directories and files to /prod40/ORACLE/hrisprod on Alice. The directories follow the pattern /orabackup/yymmdd/hrisprod/*. Open hrisprod in a nomount state. To do this run . oraenv and respond to the prompt with hrisprod. Next run svrmgrl. Connect internal and open the database in nomount state. To do this issue the following command: startup nomount pfile=$ORACLE_BASE/admin/hrisprod/pfile/inithrisprod.ora. Recreate the control files and reset the file headers. To do this run the following SQL script. This should be run after finishing step 6 and while still logged in as internal. The

script name is /prod15/ORACLE/hrisprod/control.sql. Now the redo log headers must be synchronized with the rest of the database. To do this issue the following command. This should be done while still logged in as internal. The command is: alter database open resetlogs;. This may come back with a message indicating that reset logs only need to be done in a recovery situation. This is ok and just re-issue the command as: alter database open; . Shut the database down in normal mode. To do this issue the following command while still logged on as internal: shutdown normal hrisprod Startup and open the database. To do this issue the following command while still logged on as internal. startup open hrisprod pfile=$ORACLE_BASE/admin/hrisprod/pfile/inithrisprod.ora Change password of Oracle User HRIS from the bob password to the alice password. The procedure is finished.

May 17, 2004 Where Were You When the Lights Went Out? DC Evolution Post Y2K New DC plans for: Financial Systems SIS system Issues:

Manage expectations This is not 24 x 7 What if Martial Law or State of Emergency? What if SARS outbreak quarantines your staff/site? May not be able to handle certain expected or unexpected disasters May 17, 2004 Where Were You When the Lights Went Out? IT infrastructure (at Ryerson) May 17, 2004 Where Were You When the Lights Went Out?

Disaster Site evolved to full computer room Each computer room has: Standby generator & UPS Redundant network access paths Multiple ISPs with redundant feeds Fully automated robotic tape library Automated backups & cloning Lights Out Operations High Availability configured applications May 17, 2004 Where Were You When the Lights Went Out?

Lights Out Operations No Computer Operations Staff since 2000 More self-serve appls (> online, < batch) Users submit overnight job requests CRON scripts are triggered Shutdown appls and start backups Reboot servers as scheduled Restart appls and run batch jobs (< 2hrs) Errors generate auto-alerts On-call staff dial-in remotely Printing: online access, local, central

May 17, 2004 Where Were You When the Lights Went Out? DC: Departmental Data Backups Critical data in departments Departmental servers: Nightly automated backups Clone bkups as needed, restores on demand Departmental micro computers: Each user has 500mb folder Shared folders for departmental use Bkup/copy data directory to NAS

Remote access available NAS: Users bkup when they want (manual or cron) Self-serve restores (5 days of online archives) May 17, 2004 Where Were You When the Lights Went Out? DC Strategy Prevention: Avoid 2 day outages Use high availability technology UPS, generator, multiple ISPs, etc. 2 computer rooms

Physical Security Access control (alarms, motion detectors, Netbot) Environmental controls (temp alerts) Automatic fire-sprinklers (what triggers them?) IT Security: Firewalls, anti-virus, anti-spam Intrusion Detection & Intrusion Prevention Encryption, authentication, ACLs May 17, 2004 Where Were You When the Lights Went Out? DC Summary Leveraging technology at Ryerson Unattended Operations Unattended backups Automated off-site backups Highly Availability systems Remote ops & mgmt of services by:

Network Services IT Security Technical Support DBA & Application Support groups 2 computer rooms May 17, 2004 Where Were You When the Lights Went Out? Disaster Contingency What did we learn from Aug 14th? May 17, 2004 Where Were You When the Lights Went Out?

Disaster Contingency Did not expect State of Emergency Did not expect rotating blackouts Did plan for SARS (after Nursing students quarantined) Netbot and web-cam in computer room Could run most systems remotely Could get Campus Security to go push buttons May 17, 2004 May 17, 2004 Where Were You When the Lights Went Out? Disaster Contingency Are we better prepared

for Summer 2004? SARS? Blackouts? Act of Terrorism? ??? May 17, 2004 Where Were You When the Lights Went Out? Disaster Contingency Questions Larry Lemieux Asst Director, IT Support Ryerson University [email protected] May 17, 2004

Recently Viewed Presentations

  • Scholars of Human Expression: The Experiences of Autistic ...

    Scholars of Human Expression: The Experiences of Autistic ...

    Scholars of Human Expression: The Experiences of Autistic Performing Arts Professionals and Attitudes of Performing Arts Employers in the UKEllie Buckley, Liz Pellicano, & Anna Remington"The real barrier and the real thing I struggle with is just other people's perceptions,...
  • I Pronomi Personali

    I Pronomi Personali

    PRONOMI PERSONALI IL PRONOME «CI» Significa anche «in questo luogo»: Ci siete? Chi c'è? Anche in senso figurato: Ci pensate spesso? PRONOMI PERSONALI IL PRONOME «NE» Complemento di moto da luogo= «da lì» Basta, me ne vado di casa! 2)...
  • How Do We Classify Matter?

    How Do We Classify Matter?

    physical ways TO SEPARATE MIXTURES: NOTCHEMICALLY SEPARATED BY DOING CHEMICAL REACTIONS BREAKING ELECTRON BONDS. SIFTING (2 different-sized solids) MAGNETICS. FILTRATION (solid from a liquid) EVAPORATION (solute from a solvent
  • A quick note for teachers - Murray-Darling Basin Authority

    A quick note for teachers - Murray-Darling Basin Authority

    A quick note for teachers. This presentation is one element of the freely downloadable lesson plan The. Water Cycle. in the Murray-Darling Basin. Comprehensive presenter notes are available in the 'notes' view of this presentation. For an accompanying student worksheet...
  • Welcome to SOSP Women&#x27;s Workshop - Duke University

    Welcome to SOSP Women's Workshop - Duke University

    Welcome to SOSP Women's Workshop October 13-14, 2007 Carla Ellis, Duke Goals of the Workshop Build a strong community of women researchers in systems Providing networking opportunities Encourage you to pursue systems research Illustrating the exciting opportunities & challenges Enhance...
  • self-paced eLearning series Creating Solana Users Optimize your

    self-paced eLearning series Creating Solana Users Optimize your

    Objectives 1st. Include enabling objectives, measurable objectives w/ 'enablers', knowledge or skills- these are called Terminal Objectives. Creating Solana Users. User Groups and Rights Settings. Creating and Configuring Solana Users. Key Topics.
  • Atomic Structure - PC&#92;|MAC

    Atomic Structure - PC\|MAC

    Atomic Structure Early Theories of Matter Science as we know it did not exist several thousand years ago Democritus (460-370 BC) Democritus' Theory Matter is composed of _____ through which atoms move Atoms are solid, _____ , indestructible, and indivisible...
  • District PLC MeetingGrade One

    District PLC MeetingGrade One

    A quick reminder of our norms (Stay positive) mathematical practice standard #1:Make sense of problems and persevere in solving them ... PLC Meeting will continue the focus on the implementation plan for Go Math. Chapter One Teacher Edition. Designing your...