This Saterday, we switched the subnet over, and moved all the remain criticle systems over to a new serverroom across the country.
I didnt get to make as many pics as I liked, and no video, this was mainly cause I was so bussy of course. :D
Half the pics below are curtesy of Arnold.
Back of the Digital Alpha box, refernece sho for recabling.
Packing up the servers in special locked crates. You could see these movers where the right stuff, big burly guys, but they handled the servers like feathers.
Justin felt at home at the new location.
x3800 waiting to be converted.
We layed out the servers in the hall in the order we where building them into the racks.
Our project operation center, round the corner of the serverroom. From here the managers of the project coordinated the downtime, and the business on-site testing.
Tom, our WAN guy, on the phone with Mohamed, our Unix guy, who was supporting us remotely.
Very good food and snacks where provided by Arnold and Jan. Thank you guys! You know the best way to an engineers heart is through his stomach!
Justin loves Legos, so this pictures seems to make sense.
Justin working on the rack conversion kit for the IBM systemx 3800
Tom and me discussing the Proxy server and internet line out. Old proxy, new Internet line, and some firewall rules where needed.
Arnold:“2 pizza please!” ; Mustafa is thinking: “I dont like Pizza”…
Paul being technical. We hold our collective breath.
Sliding in a server into its new home.
Bottom of racks SR3 and SR4
Top Left rack, SR4
A suprise box. This witebox FTP server turned out to be running 4 essential ustomer FTP/EDI flows. We had space for it thankfully, resting on the IBM x3800. Hopefully it will be gone in 2 weeks, but nothing is temporary. Made this picture of it for the Visio rack diagram.
One of the 2 redundant FTP servers failed on transport. Justin and Paul spend 2 hours putting a new one together out of spare parts of old ones. HP Netservers here, P2 machines, a decade old.
Behind locked door and locked racks, all servers humm quitely, content in their new home.
A day later, back in the first location:
Empty racks moved out of the server rooms
Mail Cluster going to get sent back to UK
Gertjan visibilly enjoying dismanteling the place. 6 years of mental burden of supporting tihs stuff being dealt with here ;)
The NAS, all the data for the Netherlands, with all volumes deleted and now unplugged, ready for storage.
Videos of “demolition”:
Mustafa and Gertjan have taken apart the entire second server room in 1 day. (click here for link if you cant see the embed above)
Oooh.. Nobs! Lovely nobs!! (click here for link if you cant see the embed above)
I love the display, which has some pre-customized WoW settings in there. I also downloaded a G15 Teamspeak addon that shows who is speaking on Teamspeak, very cool. The keys are very pleasant on the fingers, and the keyboard has some special macro buttons down the side, which come with quite powerfull macro/scripting software.
The backlit lighting is very nice. Though I prefer the blue that the G11 model has.
Its very nice, but after playing around with Lia’s new MX1000, I think I might actually prefer that for the feel. The Sidewinder is pretty good though, though it is a bit bulky. It comes with weights that you can insert into the side, which is kinda smart. I went for the max load. It also comes with a cable block to keep the cable at the right managable length. You can customise the buttons, but not the buttons at the top which I thought was a bit weak. The buttons at the top control on-the-fly DPI switching which can be usefull, the little diplay shows what DPI the mouse is set to.
Because I am rather jellous of Lia’s new MX1000, I decided to get the MX Revolution for my work.
Now the MX Revolution is, I guess, the successor to Lia’s MX1000.
Logitech MX1000
Now while that is true, let me say that these are in fact two very different beasts. Lia doesnt like the second scroll wheel for example and while I dont mind it, I cant blame her.
Estetically they are both gorgeous and very well designed, and fit in your hard totally naturally, supporting the thumb very well.
The revolution does a funny thing with the main scroll wheel, it actually adapts the way it rolls during scrolling, on the fly, depending on the kind of page you are on. You feel it physically switch between the common “clicks” and the “smooth” mode that some Microsoft mice have standard.
Anyhow, to conclude, I am very happy with my new purchases, and am sure they are gonna serve me very well. And at work, I am totally pimpin of course
A tour of the current state of the office, and one of the server rooms, where most racks are now empty or near-empty.
An Epic moment. Gert-Jan uninstalls Citrix on the last 2 servers in the farm, effectively ending 6 years of the 2000-user, 60-server Metaframe XP Citrix farm that served our Netherlands users. I would have had Marcel do this, but he has gone on holiday.
Various IBM servers and almost all the blades lie ready to be moved to the new location. We don't have a use for any of these currently, so they will go into storage.
Various IBM servers and almost all the blades lie ready to be moved to the new location. We don't have a use for any of these currently, so they will go into storage.
Remember when we where young and the world (servers) where new?
Both blade centers are going into storage, we have no use for them.
Blade Centers in they Hayday
The racks are becoming quite bare now.
Before:
After:
Before:
After:
Before:
After:
Before:
After:
The pile for the garbage container crows and grows.
Marcel posing with all off the DL360 G3 severs (going into storage). he built the farm all those years ago, now he bears witness to its demise. Its a little sad for all of us that spent so years maintaining it all.
I managed to get my hands on one of my favorite servers, the IBM systemx 3650 with the 8 SAS disks in it. We reinstalled it as a new SQL2005 system, that will host, amungst other things, the HPSIM, IBM Director, Websense and Sharepoint databases. All for internal IT use.
Spent today going over some cabling details mostly. Its strange how stressed we all became over these little details, while we will have larger issues to worry about on the day.
During the move, we will be moving the Voip servers, RF-Controllers (used by various sites to do handheld-scanning), an Authentication server, the Wyse-Terminal (thin client) management server, and 1 PDC.
We are also moving 2 very criticle FTP/EDI (Electronic Data Interchange) servers, that handle lots of customer-related FTP flows and internal EDI between warehouse management systems.
All the above systems (in the rack diagram in RED) rely on keeping their old IP adress for now, and thus we are moving the entire subnet over to the new site. This is why we call it the big bang. After that, the old site will no longer be routable to the rest of the network, and all that remains is stripping it and storing the hardware that is left.
Also the former Domain Controllers / DNS servers are an issue. Their IP adresses have long since been used in clients all over the place, the config of many of which we cant control centerally (for example through DHCP). Therefore I am moving 1 of the old PDC’s, and its keeping 2 of the 3 legacy IP adresses we need to keep alive.
We spent yesterday doing last cabling work and other such things in preperation for Saterday (see pics below). I still have some things to do, there are some management/reporting scripts still running on the old site I need to migrate tomorow, and I have yet to get round to re-installing HPSIM again. I might start on it after this post, actually.
Here are yesterdays pics:
Starting to look more like the planned diagram now. As you can see we, we replaced the Sun Storagetek with the HP Storageworks MSL9000 that we salvaged from one of the Warehouse Management Systems that was migrated to Prague last week. We dont know if we can use it though, its kinda overkill for the amount of data we now need to backup anyway.
Mustafa is set to become my new right-hand man after his project, he certainly has got the right attitude ;)
Oh thats Justin, he has been added to this project to take some of the load of me. He really knows his stuff and is a wirlwind of highly-opinionated energy ;)
Ninja-Consultant. Implements ESX-virtualisation solutions when you are not looking!
Ser is our resident LAN guy. Here we created a patching/switchport diagram for the racks (on the laptop screen), and he is tightening up the VLAN configs on the stacked core switches. Justin and me then quickly repatched everything before anyone noticed there was a service interruption ;) (dont worry, the really criticle stuff goes in during the Big Bang)
This is more or less how its gonna be. All the bits are now in place, half the rack stuff has been put in place already, the last stuff goes in coming Vriday.
One of the biggest oversights we made was with power requirements. If you want redundant power for a lot of servers, you need to make sure you have a LOT of plugs, enough PDU’s, and plenty of PSU to back it all up. We didnt, partly due to time contraints, budget, but also short-sightedness, and my reliance on people that I, to put it bluntly, shouldnt have relied on to do the math.
Above pic also available this time is very high res, so you can see all the exact server types. These Visio stencils are very detailed, I love em!
So a few weeks ago, we spent our Saterday doing major recable work on the new Technical Computer room (TCR) in the new location, in preperation for our Big Bang services move coming up on the 5th of July (it was delayed from the 22nd))
So thise pics and videos are a few weeks old now, but who cares.
PICTURES (scroll down for movies)
Not enough room to neatly get all the fat KVM cables sorted, so later I actually extended the room for the cables to 2U, and used the cable guide rail only for ethernet cables and power.
2U above the KVM not just being used for those fat cables, but also for the Avocent Switchview box, and its adapter, that is attached to the HP KVM box. Advocent KVM/IP hardware rocks by the way. If you have a non-IP KVM swich, consider the Switchview IP 1020, it makes life so much easier and they are super simple to set up!
Too many cables, not enough room. Once we are done with the move, I am gonna seriously tie this shit down. Meanwhile I curse those HP 10K racks for not having ANY room at the sides for large cable bundles. Oh.. not to mention ANYTHING to attach a tierib to!
The lineup of servers. The missing one was at the old location being synced up with another FTP box.
The master at work ;)
Not easy to work with all the cables all over the place.
The patch panel guys where not done yet as you can see.
Hello!
Mustafa putting some of the servers into cold storage. You wouldn’t believe the kind of hardware we will end up putting into storage cause we have no immediate use for it!
This is what happens when you dont move your servers professionally. We dont have the budget for it, so that is our exuse. Servers where fine, though I would not recommend this.
Carting around servers.
So very little room to work with. Look at the space at the sides, notice the PDU, then imagine it being full of servers. See my point?
So here we decided to move some servers over to the middle rack. This is partly to do with the fact that I discovered we dont have nearly enough power connectors in these racks to fill them.
I havent been back there since, I really hope they cleared these bundles up now.
I hate loose cables like that.
HP-UX on a 9000 box. Very old, but still used. Hosts a warehouse management system, all terminal based.
The proud admin of the HP box.
Richard wasnt very usefull during our work, so he caught up on project admin ;)
MOVIES
A lot of engineers in a small space. This movie gives you an idea of the space :)
Me working on the cables. All the cables all over the floor was half the point of tackling this place, what a mess.
Me working on getting the KVM set up. The little box is the Avocent IP View 1020
The most important element of our productivity right there!
All the “temporary” plugs and cables finally being cleared out. I wonder how many admins out there wish they had the time or the downtime to do the same and really clean house.
Overview of our LAN and WAN racks.
Making slow but steady progress on the cables. 3 ours later it would look very diffferent!
Labling ALL of your cables on both ends means you can accuractly document what hooks up to what from A-Z. Doing the initial labeling though is a pain, so this is where our “certified labeler” came in ;)
The amount of old hardware our IT department has in stock is just crazy. They dont know what to do with it all! We should have a CRT yardsale or something. Anyone need a Lexmark multifunctional? The IBM xseries 336 being stored here are only a fraction of the hardware we will be storing.
Its 2 am and the end of our scheduled downtime. Clearing up the last stuff and wiping the servers off, and turning em all back on. Hooked up to the new UPS, we could finally have them ALL on without burning out the old grid. The best bit of all was we managed to clear almost all cables from the floor.
At the KPN ommunications center at Schiphol airport. There are bout 100 people crowded round the bar at the far end of terminal, watching Netherlands-France, and the Dutchies just scored their first goal
Flight leaves in 45 mins, I have 16 mins remaining on the internet code. Will read Wired magazine in Flight, as its almost not worth sticking yer ipod on. They dont let you listen during takeoff or landing, and the flight is only 45 mins. Go figure.
