:::: MENU ::::

Growing Pains-Building for the future

Well as many of you know our new Server Room/Datacenter is in the construction phase.  Yesterday the Electricians were on site to wire in the additional ATS (automatic transfer switch) for the generator, which is great…   We have to install the new ATS so we can have the entire power load of the new Datacenter on the generator….

Well we might say that is all good news… Except our staff was off campus all day for our Staff Christmas party and I found out after the fact the electricians cut the power that feeds the existing ATS.  This ATS powers our emergency lighting and temporarily parts of our old Server room. 

Well long story short our soon to be replaced battery backups don’t last 45 minutes and they are consumer grade units so the software doesn’t scale to the number of devices we have plugged into them….so take a guess what happened.

Can anyone else guess what time the power went out?

We can’t get relocated to the new Datacenter soon enough!

Free from Dell!

Recently we have had some issues with our orders from Dell.  We placed an order for several monitor mounts for our new DataCenter along with KVM extension cables for the monitors.  I gave very specific instructions as to what cables and mounts we wanted.  When the cables arrived they were Belkin KVM cables to connect the KVM to the servers… not what we wanted. So contacted Larry, our Dell Parts Specialist, to set up the cable RMA. 

One funny note, after we setup the RMA someone else called from Dell offering me 20% off the price of the cables we were RMAing if we would just want to keep them for another use… i declined that ‘generous offer’.

Then Larry did something really cool, he emailed me and told me because there was so much trouble getting the monitor mounts SKUed up and the cables were wrong he would try to ship us the cables we wanted for free to say ‘sorry’… well Dell didn’t carry the cable we needed. So Larry writes back and says just pick something off the Dell website that is less than $99 and he would give it to us since they value our business… and even send it next day.

Well there isn’t too much on the Dell website that is less than $99… But Jeremie found a Bluetooth™ iGPS-500 Receiver from Pharos that works with GPS software on a PDA or laptop.  So I asked Larry and he overnighted the GPS receiver…

It arrived today, and here I am being a hand model…Really Awesome Customer Service!!! And Merry Christmas to the Northwoods’ IT Dept…. Maybe it will prevent Jeremie from doing 360′s on the interstate when we go to the next CITRT.

SharePoint Training for Church Consultants/Vendors!

An update to our SharePoint Training we will be offering on January 7-11.

Bill English has extended a very gracious offer for Consultants/Vendors who serve the church.  Previously we had limited the training to individuals directly employed by a church or ministry and no-one in the profit sector, serving a church or otherwise could attend.  Well that has changed.  We are now able to welcome individuals or employees of companies that provide services to the church.

The news even gets better, Bill has discounted the training by almost $1000.  The cost of the training including all the courseware is $2495.

Register Now

Restoring SQL

Our recent SAN crash resulted in our needing to recover all 13 of our virtual servers from backup and has hampered my blog posting, but the blog must go on… so back on track….

One of or servers MIA was our SQL server.  We reinstalled a base SQL installation with the appropriate service packs and then began to restore our databases. 

As many of you know we use SonicWall’s CDP 4440i for our backup.  This is a disk to disk to disk solution that has worked really well.  It has a SQL agent , and we had the agent installed prior to our SAN failure which had been backing up a full backup daily, 3 hour differentials.    So we reinstalled the agent and planned to bring the databases back online. 

Since most were 3rd party databases, we elected to allow the applications to recreate the databases and then restore the data back over the empty databases.  This allowed the connectivity to the SQL server to be restored before we tried to restore the databases. 

SonicWall’s agent has a feature when you select the data set you want restored and you select restore to application. The database is taken offline, restored and then brought back online.

No problems, well no problems except for one database.  Our hourly staff use TimeClock Plus to document their hours worked and that was the DB that caused problems.  For some reason we couldn’t restore the DB to SQL with the SonicWall tool.  So we elected to use option 2, save the .bak file locally on the SQL server and use the SQL restore function.  This option didn’t work either, in-fact after calling Microsoft we still couldn’t restore the database.  When you would choose to restore the DB in SQL you couldn’t find the .bak file when you would browse for it.  The SQL manager console would just display a blank folder.  So we contacted SonicWall thinking the .bak was corrupt.  Engineering at SonicWall had me upload the .bak file and worked on our case for several days with no avail.  Finally they remotely connected to see if they could help that way.  It finally dawned on me to save the .bak file on the e: drive rather than the c: drive since that is where SQL is installed.  Suddenly SQL recognized the .bak file.  I took the database offline, ran a SQL command and we were in business. 

In case you want the SQL command:
Restore database <databasename> from disk=’<Disklocationof.bak>’ with replace, recovery

Some things we learned:

  • SQL needs the .bak file on the same drive as SQL is installed. 
  • The SQL DB has to be offline before you can run the recovery command.
  • Microsoft SQL support sometimes overlooks the easy things. They didn’t think to tell us to move the .bak file to the e: drive.

So once again the CDP has paid for itself, and some experience has helped us for the future.

Awesome Customer Service

Recently around the CITRT (Church IT RoundTable) there has been some disappointment and frustration expressed about customer service so I feel obligated to share a really positive customer service experience.

While catching up on my TIVOed episodes of Chuck (i am just getting into the show… I love the nerd herd cars) I received a dreaded phone call from my wonderful wife,Natalie.  She called rather, calmly I most note, asking me to come an pick her up.  She had been at an early evening meeting at work and was on her drive home.  She told me she had blown a tire on the car and was sitting along side the highway.  So I changed into some old jeans and headed out the door…. Calling Jeremie, my network admin, to ask where the closest turn around on the interstate near Natalie’s disabled car would be. 

Side note: when we hired Jeremie he commuted 45 minutes each way to work for 2-3 months so he was quite familiar with the highway Natalie was on.

So I change the tire, which has a hole in the sidewall as big as 3 of my fingers, and head home for the night.  I wasn’t terribly worried about replacing the tire since we purchased road hazard on the tires when we purchased them two years ago.

Wednesday AM I called the Goodyear location in Springfield, IL where we had purchased the tires before we moved.  They told me I could go to any Goodyear location.  So I drive to our local Goodyear location and show them the receipt from the Purcell Goodyear store in Springfield and they say they can’t honor it.  For those who don’t frequent Goodyear, some locations are company stores others are partner locations.  Well the location is Springfield is a partner location… Hence the store here can’t help. 

So we call the store in Springfield and they tell me their road hazard program is the same as the corperate Goodyear stores… but I should have a set of stamps for the road hazard… I don’t.  In fact I notice the invoice says the customer needs to sign the bottom indicating they have receieved the stamps… well I there is no signature on the bottom of the invoice… so well… you draw the conclusion.

So Glen at the Peoria Goodyear says they will sort it out and he will call me.  So I head back to the office. A little later Glen calls and tells me the Springfield store has no record of my purchase and they aren’t able to do anything…. No record of my purchase I have the original invoice in hand… they didn’t keep a copy are you kidding me?

So decide as a last ditch effort to call Goodyear corporate customer service.  I explain the whole story to the customer service agent and he puts me on hold. (Calling Glen in Peoria) A few minutes later the customer service person comes back on and explains Glen has ‘more options’ and will be able to assist me.  Long story short, the customer service person approved replacing my tire even-though there wasn’t any record of my purchase of the tires or of the road Hazard…. That’s really cool.  A company who does something to say ‘yes one of our affiliates screwed up but we value our customers and will make it right.’

2 hours later Glen calls and my car is ready to pick up…  Now I just need to call Purcell in Springfield, IL and get them to refund my illegitimate, non-existant road hazard that no one has record of…

Now I have to be thankful for a couple things, 1. my wife wasn’t hurt when the tire blew out, 2.  i didn’t have to buy a new tire… 3. the tire didn’t blow when we had a full trunk of stuff traveling back from Thanksgiving in Ohio and it was pouring down rain…

Elf Yourself

While we were spending some time with my parents over the Thanksgiving holiday my wife, Natalie, came across this website.  While we were watching the football game Natalie was sitting on the couch laughing so hard she was crying… after about 10 minutes of saying “just wait its funny” and tears streaming down her face she showed us this page….While I didn’t find it as humours as my mom, dad and wife it is fairly cute.  So maybe you will cry laughing… or maybe just say that’s nice… but I’ll share the ‘hilarity’ with you. 

I do have to say this is one the more creative Internet advertising campaigns I have recently seen.  Just beware, its a pretty broad fair use clause attached to uploading your photos.. So you can’t sue OfficeMax when your photo appears on their commercials like the girl with the cellular telephone company.

Thanksgiving Thoughts…

So thinking of what I am thankful for today in no specific order:

  • A great family to spend the day with (family and in-laws)
  • A wonderful wife who is my best friend and supports me in ministry
  • A awesome staff who serves tirelessly to support Northwoods’ IT needs
  • A growing relationship with Christ
  • A supervisor who is a helping me grow as a leader
  • Being part of a growing church family

Tivo Lifetime is Back?

I received my normal Tivo e-newsletter yesterday and I noticed something intriguing… They are offering Tivo Lifetime Contracts on their smaller HD Tivo.  For $689 you can have a Tivo, Lifetime Service and a network adapter.  The catch, this program is aimed at existing customers ‘giving’ the HD… so you have to have an active Tivo contract to be eligible.

$299 for the box, $100.00 for the wireless adapter and $299.99 for the Lifetime Service

So  if you have a Tivo HD for more than 3 years you start breaking even… 

How Bad is it? Answer: Really Bad

After another exciting day on the phone with tech support I am left wondering, when do you say enough is enough.

The morning started at 7am and I found out support didn’t open till 9am, I should have known it was going to be a long day.

The first 5 hours were spent calling every 30 minutes for updates from Intel.  While the support agents speak clear English, they weren’t exactly helpful.  They were only able to let me know that engineering was looking at the case and had requested remote access to our network.  Now they did note, engineering NEVER asks to remotely connect… wow wasn’t I excited to hear that.

A little after 1pm a support tech named Ruonan called my cell phone and said she would be taking the case.  I have to say she is probably one of the most pleasant tech support agents I have spoken with.  So Ruonan contacted Galen at LeftHand Networks and we started digging into the problem.  Galen connected to the SAN via SSH and noticed that the files that should be on the drives to create the configuration were missing… Hence we can’t find the data on the SAN.

Ruonan instructs us to then boot to slax pocket Linux and browse for the files on the DOM.  So we load slax Linux on a CD and boot from disk to create a bootable jump drive…  One item learned, SLAX boot iso doesn’t have an installer as the instructions state so we can’t make the USB jump drive bootable.  So Jeremie downloads DSL (D*** Small Linux).  We load that on to the jump drive.  All this monkeying around takes about 3 hours….  We boot up the SAN to find that the files are missing on the old DOM too…

So where does this leave us, according to Galen 80% chance of needing to rebuild the SAN.  So when do you wave good-bye to your data you know is on the SAN and start the backup recovery process?  Ugh, the though of restoring 13 virtual servers including a SQL server doesn’t give me warm fuzzies.

So at 5 pm I make the dreaded call… we are going to restore the servers from backup.  We contact Mark Moreno our SAN channel partner and the rebuilding of the SAN begins.

For the rest of the night we build and boot up virtual server templates and start the restore process.  At 1 am its clear we aren’t going to be up and running by morning so I tell Jeremie its time for us to head home.

I arrive at the office at 7am and we continue to work on rebuilding the virtual servers. We did a system state restore to the print server and all the printers are back.. but the server isn’t stable… So we take the info about the print shares etc and start up a new template.   Around 9 am we have our Print Server back-online (minus 2 printers)…

I do have to say the restore process from our SonicWall CDP is really nice… We install the CDP client on the new Virtual servers and then right click on the directories and tell it to restore deleted directories and the process is under way. Need i say, no searching for tapes.  As soon as the data is back online, the directory is a watched folder again.

Around 2 pm the SAN is configured and ready for us to start restoring data. One bonus for the whole process, we configure the SAN with Load-balancing this time to utilize both NICs on the VM_Hosts and configure the mirroring and array in RAID 5 and gain almost a full TB of storage space.

We attempted to restore our SQL databases to the newly built SQL server but no luck.  A 2 hour call to Microsoft and a few tweaks on the permissions and using the cilconfig tool and we are able to remotely access to the SQL server.  But now the Desktop Authority application fails when we install… And of course DA support is closed.

While I am talking with MS about SQL, Jeremie starts a case with MS Support about restoring the system state of the file server.  We have the virtual server with the data restored but we need to use the system state to bring back the permissions and shares.  We called MS since we hadn’t had much luck with the system state restore on other servers earlier in the day, and our lives would be much easier if we could restore the file server properties.  So we snapshot the virtual server and MS helps restore the registry.. No luck on reboot one, but suddenly after reboots 2 and 3 file shares appear.  A little tweaking of the registry and we are back up… but minus permissions.  The tech tells  us that we will have to rebuild the permissions… So at least we got the shares back…So now that its 2 am I tell Jeremie its time for us to go home.

A 7:15 am I call Scriptlogic and the tech points us to the fact we just need to do a clean install for DA and all is well… And once again our CDP works well… right click on the database in the client application on the SQL server and magically the DB is restored.  Logging into the DA console all our settings are there… we can now finish our Print server since we know the printer share names we need. 

It takes about 2 hours to restore the permissions to the file server but that is now back online.  So after 4 days we have our File, SQL, Print, Web, Antivirus servers back online.  The help-desk, SharePoint, Ghost and a few other less important servers will have to wait  until after thanksgiving weekend.

Our next steps, finish restoring the remaining servers, and review and evaluate the crisis.  This will include asking the question what can we do to prevent this from happening.

We found some issues with our backup process, primarily documenting configurations.  You don’t realize how much stuff  you store on your file server or in specific application databases until your file-server and SQL server are MIA.

One thing we will seriously look into: a storage space and process to backup the whole Virtual server.  If we had a backup of the whole VMDK (and other files) for each virtual server we could just restore the data to the offline server and this process would have been much quicker… Maybe a rack mounted NAS might be a good solution.  It needs to be off the SAN, but still on drives that are a Raid Array.

One thing we have learned, how to and that we should backup the configuration from the SAN.. Even though the support at Intel said that recovery of the SAN after a DOM failure would probably work if we had a backup of the config, they said that restore sometime fails too… But for the future, we have that config saved and and archived just in case.

I must say a few say thanks for the prayers!  Several including some Northwoods’ staff, JP and Ed called or emailed just to check in on us or say they were praying for us… very cool… this is what CITRT is all about.  I even had a vendor call and ask how they could help… Dean Lisenby from ACS is top notch in my book.  He calls after his work day is done on his way home to remind me that I have his cell phone number and can call for any reason…even just to check in… Dean isn’t your normal vendor.

I have to give huge Thanks to my staff.  Each person on my team has responded extremely well during this crisis; no task was trivial no matter if it was laughing at me when my thoughts are less than clear when they come out of my mouth at 1 am, fixing my forgetting to run a Virtual server as a service rather than under the local login account, answering the question of “when will I be able to….” for the rest of the staff, or unlocking doors after I leave my keys in the server room or my office, or the new server room, or in the bathroom, or.. well you get the picture.  Thanks Jeremie, Jim and Linda you 3 are an awesome team!!!!

Now I must sleep, 10 hours of sleep since Monday morning leaves a sleepy IT Director.