:::: MENU ::::
Monthly Archives: January 2008

“There is no way to retrieve the messages”

Those words are all to familiar to the Staff at Northwoods who survived the Great Crash of 2006, but now those words are being heard… by customers of Charter Communications. 

Fox news reports that Charter Communications  was performing maintenance Monday night to their systems and inadvertently deleted over 14,000 email accounts that were used by their active customers.  And oh yea there was no way to recover the data… but they are giving the affected customers $50 credit.

My favorite quote in the article “It’s never happened before. They are taking steps to make sure it never happens again.” Duh, you think?  Talk about a credibility killer.

Maybe its my total lack of compassion and I should feel sorry for the poor staff at Charter Communications…. for whom I dare assume this was a resume generating maintenance night, but I have to admit it makes me chuckle when it happens to the big guys. 

I should have some minor compassion especially when we experienced the crash of 2006 the second week I was on campus at Northwoods. But while there wasn’t a backup of the data we were able to recover the mailbox store we thought was a complete loss… 
Note: We don’t use that Backup Strategy anymore…

Learning Moment: Is your backup plan configured in a way that you can recover from any dumb admin error any of us could make during a maintenance window at 2 am?  Not to mention have you tested the plan? 

We learned our strategy is sufficient to have a complete failure and not lose data, but we are still tweaking the system for the future…  We learned in our recovery we had covered the basics but we need to improve the recovery window.  So we to need to test some more. (Note I use the word ‘test’ assuming a test is not a recovery of data in a actual failure :) )

We all should take note.. this could happen to any of us… we all make mistakes maybe not on this grand of scale, but we all make mistakes.  We should have systems in place that can tolerate not just our users but US too. 

Just because we don’t support as many users but a proportionally sized event in our organizations would be as catastrophic and equally as much a ‘credibility killer’.. what say you?


Great Team = Cell phone off?

Yesterday was a long over due vacation day… not long over due because my employer is overbearing…  but rather over due because I kept kicking this vacation day further out on the calendar because of ‘this’ or ‘that’… so possibly long over due because of my poor planning I suppose….  So I spent my day doing what I wanted, slept in a few extra hours, spent time with my wife and watched about 7 episodes of 24.  A fairly unproductive day by some standards, but quite a nice day by mine. 

I have to say thanks to my staff once again.  In the past it would not have been possible to take the day after a work night off or any day for that matter.  And before you scoff…. know it wasn’t just a pizza and soda work night, we moved our ChMS Terminal server to a new host and installed two HP2800 switches (not too shabby of work for one night).  Now that I have an awesome staff in place I can take a day away without a lot of stressing that my phone will ring and my 24 watching will be halted by an ‘emergency’.  I think it says a lot about my staff when I can turn my cell phone off (insert surprised look and gasps here) and know things will be in great hands. Once again my team handled several minor bumps in the road with flying colors!  Linda C., Jim and Jeremie you guys are a great and I am very fortunate to have you working with me.

Now another note, as IT professionals, when should we make it a best practice to not just silence our phones but actually turn them off?  Do we use this handy little feature frequently enough to create healthy margin?  Its not like the off button is a new feature but my guess would be no.  Thankfully my supervisor, Linda M, does encourage this behavior (within reason of course :)), but do you like me not take advantage of this encouragement often enough?  Do you utilize this freedom to put your family first?  I think I might need to become a little better acquainted with that little button on my phone so rarely used.

So with those thoughts I must thank my staff for the opportunity to power down to take a day to recharge.


MasterFlex is Online

Our IT staff had another work night on Monday night; as I have mentioned about before, we have regularly scheduled work nights that allow us to take our servers and services offline while we perform mantiance.  Well Monday night was one of those nights.  During this night we were able to put “MasterFlex” our new Intel Flexbox blade system into production.  We moved our ACS terminal server to FlexHost-01, one of the blades in the chassis.  As with our other servers our host boxes FlexHost-01 is also running VMware Server so this migration was as smiple as copying the virtual server over to the new host.

Well lets just say almost simple…We learned a few things in the process… Once again VMware’s tech support was very professional and knowlegable to suppport us. 

After brining the terminal server online it was locking up and the screen would freeze.  After uploading the log files to support we idnetified the problem was with the VMware tools.  We had moved this guest server from a 32bit host, and moved it to a 64bit host… and didn’t think to reinstall the tools.  So after reinstalling the new VMware tools things were better…

But then Tusday morning trouble struck again…the terminal server froze again sometime overnight.  Jeremie came into the office in the morning and saw the server was locked up…. He was able to fairly quickly identify the machine was locking up during the search of the CD hardware. The problem, there was a CD drive mapped incorrectly on the guest which was causing the machine to hang while it tried to poll the CD drive.  He removed the CD drive from the guest’s config and all was well.  Great work Jeremie! 

Several users have noted that their termial sessions are now as quick as working locally.  We haven’t run our benchmark testing yet, but I am expecting a significiant improvement… we’ll post the results soon.


Post 101

Can you believe we made it already?  Last week the blog hit post 100.  I thought it was rather fitting that the Spring RoundTable annoucement would be the 100th post on the blog. With this milestone I would like to thanks a few people:

  • Good people of Church IT RoundTable.
  • The other Jason for setting a precedent.
  • Tony Dye for being the patriarch of Modern Church IT and encouraging me to get the blog started.
  • Jeremie Kilgore, the blog hosting provider of this site.
  • Justin Moore for teaching me the ways of WordPress.
  • My wife for tolerating the new surge of photo taking to share with the readers and the time spent writing these posts.
  • Dean Lisenby for being an encouragement and a great vendor.
  • The leadership of Northwoods for encouraging our IT team to actively participate in the CITRT community.

So with that, thank you to my loyal readers (I always wanted to say that) I look forward to the next 100 posts!


Spring CITRT Save the Date!

The Spring IT Round Table is Confirmed for Saturday April 5th 9 am – 5 pm at Crossings Community Church in Oklahoma City.

Thanks to Michael Foster, Crossings’ MIS Manager and the leadership of Crossings Community Church for being willing to host us for our Spring Event.  Registration will come online in the next few weeks.

This Spring RoundTable will be a one day event since it will follow after the last day of the MinistryTECH Church Information Technology Conference on April 3-4 so if yo haven’t registered for MinistryTECH what are you waiting for?


FatPipe or Radware progress… sort of.

Progress or lack there of…

Well our Demos of FatPipe and Radware arrived.. both companies did a great job getting the hardware here quickly… one interesting note the Radware box shipped straight from Jerusalem where their factory is.. that was interesting getting a box with Hebrew writing on it.

Radware

Well both vendors have matched each others price, so the playing field is pretty much the same.  Each have supplied two devices for redundancy.  For both the second box was less than 25% of the first box, so it seems logical to run the devices in a pair… Seeing one major reason for this device is to provide fault tolerance in our ISP connections in addition to load balancing.

So the reason for the lack of progress is our Cable provider just finished installing  the cable run from the road for our new service.  First the thawing snow made the ground to soft, then they delayed because it was too cold, then this week the soil was frozen, now late this week it was just right… I’ll just call Insight/Comcast the 3 bears. 

I must say Radware continues to impress me, our Sales rep called and asked which days he could come down from Chicago and help us configure and install the LinkProof hardware.  He said he would like to spend an afternoon and morning with us getting everything setup.  Granted, this doesn’t mean their product is better, but I was really pleased to hear they would travel 6 hours round trip to help us get it configured to have a ‘successful’ demo.  This is currently more than I can say for FatPipe, they just sent the boxes and said to call them with questions.

So yesterday Jeremie installed the FatPipe hardware (photos soon) and the boxes look very clean in the rack… but several initial concerns:

  • Since we elected to request the 1U version of the FatPipe, there are only 4 ports on the hardware.  We were expecting a device expandable to 12 ports.  Our FatPipe rep said that he made a mistake…”I made a mistake & thought the 1 U boxes could handle 12 ports. Sorry.” Its ok I suppose, but that is on minor disappointment.  One note if you want the expandable version, you are going to have to coincide over 8u’s of rack space to the FatPipe hardware… ouch.
  • The FatPipe rack rails that shipped with the hardware are way too small for a normal rack.  Even when we expanded them as far as possible the rack rails don’t reach the back of the mounting brackets.  Now I will note, you can expand them far enough if you adjust the front of the rails, but then the front of the FatPipe hardware sits 5 inches back into the front of the rack.. not exactly normal rack mounting.
  • FatPipe tech support is now saying we will need a second firewall for the hardware to live behind…

So will that that said, we are running the FatPipe thru the tests… And Radware staff is on-site Friday to get their gear configured… more info to come.


And the GrandPrize IS…

A Garmin 200 was given away as our final door prize for the SharePoint Server 2007 Training.  The lucky winner of our final prize was Dennis Swinehart from Mission to the World.

Congrats Dennis!

Dennis Swienhart


New DataCenter Photos

Well now that Intel is ‘offically selling’ our FlexBox Blade Server we can now talk about our new blade server.  Here are some new photos of the chassis before it was installed in our DataCenter and after we installed it in the rack.

We are populating two blades with dual processors and 16 gb of memory per blade. The HardDrive pool is from 14 SAS drives.

flex1

Flex 2

Flex 3

Flex 4

and here are some photos of the DataCenter Online.  We have been online for just over a week in the new DataCenter and all is working well. 
DataCenter Rack DataCenter Rack2

DataCenter Rack3 DataCenter Rack4

Flex 5 Flex 7


SharePoint Group Photos

Thursday afternoon we took a set of group photos of all the attendees for the SharePoint Server 2007 training.

SharePoint Group Photo

Left To Right
Top Row:Paul Schaeflein, Justin Moore, Mark Burleson, Jeremy Marx, Jason Lee, Fady Eldeiry, Jason Powell, Mark Moreno, Isaac Neibeling, Steve Huffman, Alan Adcock, Jared Barden, Terry Rugg, David Hutchison
Middle Row: Jeremie Kilgore, David Albright, Matthew VanAndel, Ed Buford, Shawn Ross, Mark Johnson, Dennis Swinehart, Timothy Larson , J. Mac Brown, Jason Blanchard, Jeremy Good , Kurt Donnan, Terry Dabbs, Bill English
Front Row: Jim Cusey, Pam Matthews, Dave Mast, Cisco Ospina, Linda Cleveland, Beth Hayes, Anna English, Nancy Graham, Gregory Fisher, Doug Alvis

If your blog isn’t linked to and you would like to have your name link to your blog, please leave a comment.

Download Higher Resolution versions of the photos (5mb)


SharePoint Training day 3

So we have had 3 days of our SharePoint Training…. Bill is a great instructor and has adapted the course to our group very well.  His humor is a bit dry, but very entertaining… not to mention the songs he has written about SharePoint.

Here are some photos thru today..

SharePoint Training

SharePoint2
Where did the Northwoods Staff Go? All 4 are missing….

Fady
Fady, working hard or Hardly working???

NerdHerd
A few were hanging around after the service tonight so they came over for a little Stand up comedy by Jerry Seinfeld… There was a marginal drop in bandwidth at my house when all the laptops came out of the bags…

NerdHerd2

Well its time for sleep for an early morning tomorrow… Training starts back up at 8:30 am…

This morning Fady won a gift card to Best Buy (aka the mothership) in our morning door prize… it helps to be ‘In Your Seat’ at start time… (sorry ed… i really do feel badly).


Pages:12
UA-2932131-1