I’m going to start the blog again with an update on our current status. More info to follow.
Latest news
March 11th, 2006On vacation
October 10th, 2005I’ll be out of the office for the next 3 weeks. Back on the 31st when we will being the “Proof of Concept”. The POC is designed to prove out that all the hardware works together and that everyone knows how to build, fix and recover the environment. I’ll be posting near-daily updates once I’m back.
HVAC issues
September 30th, 2005After 4 weeks of checks, double checks and starting over a few times, the HVAC units have now been ordered. Adequate cooling for the data centre is always critical, but when you start using high density solutions like blades, it doubles in importance.
The main reason is that blades are ultra-dense. The small form factor precludes the ability to essentially put the BIG heat syncs you find in the latest desktops. For our configuration, we are using dual CPU 3.6 GHz Xeon chips. These puppies run hot, very hot. Thankfully we did NOT order the SCSI hard-drives on the blades or it would have been even worst.
The calculations show that each rack will be putting out between 35-45,000 BTU. That is the output of a big BBQ for comparison purposes.
We did not go for redundant systems at this time, as we want to complete the “proof-of-concept” first. This unit will eventually be cycled with a second in-rack unit once we can verify the total system load.
Now to figure out the UPS and genset requirements. I keep praying that the 200 AMP panel is enough.
SANs
September 14th, 2005The SANs have been ordered. EMC CX-500 units with 2 TB each. Nothing fancy but with everything setup for boot-from-SAN kind of critical. We will not be using the SAN Copy features initially, but the idea is to setup long-term replicaton between our sites to provide inter-site backup and DR facilities.
Now to get the HVAC and other “base” utilities ordered so I can actually plug this stuff in.
Details, details and more details…
September 12th, 2005I need to put a little context on this project. It has taken just over 1 year to get the equipment ordered. A large part of that time was evaluating the needs and requirements, but an equal amount of time was spent looking at the realities of the situation. The data centres, or should I say network closets, where the equipment needs to be install are severely lacking in terms of HVAC, power and other amenities like raised flooring, fire suppression and security.
The original idea was to fit within the existing space as much as possible. To do that we planned on using IBM blade servers and 1 U units with VMWare for just about everything. Getting 5-10 servers/blade maybe was a stretch, but it was an idea that would let us grow as we needed over time.
With head office, jumping in at the 11th hour and saying that we had to eliminate VMware and use Dell equipment, things just exploded. Instead of cramming everything into a single rack, I am now up to 3 racks per site minimum.
Unfortunately that means everything now needs to go in via a rip and replace mode. Walls are coming down all over; new flooring needs to be put in. For environmental, HVAC units, condensers, UPSs, gensets, power panels, and new racks need to be sized and ordered.
At the first site, the list of details is just amazing, even to me. First a common wall needs to be taken down to double the size of the server space. That wall contains power feeds that need to be removed from the power panels, fibre and network feeds that need be re-run. The ceilings need to be removed in both rooms so that wiring racks can be move around easily. Fire suppression (i.e. sprinklers) in the 2nd room need to be removed and our Inergen system expanded cover the larger room. Carpets need to be removed and a new floor laid down, but since we can’t get the same tile, I now need to do BOTH rooms. The existing server infrastructure needs to be relocated for the construction. The power panel needs to be removed from the common wall and temporarily re-mounted until the new panel can be installed. Overhead wiring trays need to be removed and re-installed. The lighting will need to be redone as we won’t be able to accommodate a T-BAR ceiling due to the ducting required by the new HVAC units. One outside building wall needs to be opened up to get the old UPS out and the new HVAC units in as they won’t fit through the doors. We need to build an enclosure outside the building to house the condensers and new genset.
There’s a million other details that will come up I’m sure before we even get to power on the first server.
Trust me, if you ever get asked to re-do the infrastructure for your company, start by demanding that you get to re-do the server room at least a year before the project start date! Don’t try and do both at the same time.
Servers ordered
September 1st, 2005Ok the servers have been ordered. 172 at once. One hell of a PO.
Because of our parent company corporate policy, we are going all Dell. The servers will be split up between 2 primary sites and 2 backup sites. We are focusing on consolidation as much as possible.
At the physical consolidation level, we will be using the Dell 1855 blades, and Dell 1850 1U servers to try and stay within the existing data room foot prints.
Unfortunately this entails a number of challenges including how to deal with the increased heat, power and standby power requirements. Dell has a great excel calculator on their site to assist with the planning effort. It will allow you to predict relative heat and power loads a lot better than the traditional 100% load assumptions based solely on power supply maximum wattage.
The second level of consolidation is at the application level. Trying to see what services can be effectively combined without complicating things too much or putting all of our eggs in one basket. A good example of this is combining DNS/DHCP/WINS and similar functionality on one box. We will also consolidate internal web systems etc.
The last level is the Virtual Machine. We use VMWare ESX server extensively, and where a server is non-production (i.e. we can afford to take it down without shutting this place down) and does not have physical requirements (i.e. USB dongles, serial hookups etc) it will get a slot on a virtual machine.
All servers are diskless and will use Boot-From-SAN.
Again corporate policy has dictated that we use the EMC product line, so we will be using the EMC CX-500 product line. Base size of each SAN is 2TB, with a Brocade fabric to tie it all together.
I’ll post some diagrams in the next few days.
Project overview
August 28th, 2005The idea is to completely refresh the server infrastructure enviroment. That means replacing 60+ existing servers that are a complete mess and mix of everything with a single consistent platform. All servers are a mix of Linux RedHat and Windows 2003. Over the next 8-10 weeks, we will rip out the existing equipment and build the physical infrastructure required to support the new servers.
To date that means physically moving the equipment around, documenting their configurations and software deployments and preping the rooms for the new equipment.
More to follow
Welcome
August 20th, 2005When IT Architects gave me an opportunity to be heard, I though it important for you to be heard right along with me.
This is the place for you to respond to my musings and follow along as I re-create IT in my own little part of the world.
Please join along and let me know what you think.
Cheers
Michel
The big start
August 20th, 2005My day job is to manage the IT infrastructure for a contain port management company in Vancouver, Canada. A small company with just over 160 employee’s. Sounds easy right.
Unfortunately, life is always complicated. You see the environment I manage actually includes 180+ servers running 4 separate ERP systems servicing 2 container ports with just over 1000 actual users. My workstations are terminals in industrial equipment, pickup trucks and 10 story gantries, on top of the normal office stuff.
My wireless networks include VHF 800/900 MHz radio modems and 802.11 NICs. You see my wireless foot prints are just over 76 acres. Not your typical office in any respect.
Just over a year ago I was given the opportunity to re-architech the entire infrastructure to accomodate a refresh to a new ERP system. My day-to-day musings here will reflect and document those changes.
Let me know what you think.
Michel