Oops

I broke the home page today. We had scheduled a move. There were a number of things that fell apart. The main problem was that They rolled out a redesigned home page a month ago, and added a "This week on campus" feature to it. To get this working, they were making a DB connection to a machine out on campus somewhere. They convienently didn't tell us about the connection, (it breaks our architectural model.) so we weren't really worried about it. The data fromt he connection is cached locally, and periodically updated, so after I moved and tested the first server, everything was working. When I cut production over and moved the 2nd server, everything fell apart, and we couldn't figure out why. It was down for about 2 hours.

I tried to keep the OPS folks updated, but gave up after 30 minutes, since we didn't have anytthing useful to say, and they kept asking the same questions over and over.

The main problem with this is that today was Philly beef steak and Buffalo chicken nugget day at the Morris center. I was planning on going. Tara was kind of planning to go. Everyone in the office was planning to go. The outage occured over lunchtime. We missed lunch. Philly Steak and Buffalo chicken nuggets only happens every 4 weeks, so we have to wait until April 19th...