As said earlier, our server room is not equiped with the latest and most powerful cooling equipment available. As it is summer in Norway, the temperature in the room could get above 35C. This room is not optimal, but it is the best solution available to me. So, in order to track the temperature changes I have made a quick hack with MRTG, perl and ipmi-sensors to read the temperature from the IPMI 1.5 BMC on the PE2850 and PER200s. The result is available at http://decagon.frode.biz/ipmistat. As you can see, it also displays fan speeds.

Posted: July 21st, 2008 under Hardware by Frode.
Comments: none

Well, no success so far. As long as I stay away from installing VMware Tools I have networking, but as soon as I install the enhanced vmxnet driver the network connection is lost. I get the broadcast traffic on my network, but it does get any replies when I try DHCP. And yes, I have allowed promiscuous mode on the vSwitch. So, no luck here.

Another problem is that as soon as I boot the Xen enabled kernel, everything slows down to an almost halt. CPU usage peaks at 100%. So much for trying to make it easy to play around with Xen. I guess I have to reinstall one of my servers with a Xen enabled kernel instead…

Posted: July 17th, 2008 under VMware, Xen by Frode.
Comments: none

One of the goals for this project was to try out the fancy thing that VMware calls VMotion. Basicly it is a system where you can move virtual machines around between physical hosts without having to power them down. By enabling VMotion you create an opportunity for virtualized load balancing in virtual data centers which is quite cool :-)

The VMotion requires that you have your virtual machines on some kind of centralized storage, for example iSCSI or Fibre Channel. When you or the load balancers initiate the transfers, ESX takes a snapshot of RAM content, moves it to the new hosts while concurrently logging changes from the log to the current state. When the RAM content are safely relocated to the new physical machine, the virtual machine are powered off and the change log are sent to the new host which are then powered on. This should appear as nothing have happened for the users outside the data center.

After setting up my lab, this was one of the first things I wanted to try out. After a quick install of Ubuntu on my new iSCSI disk, I initiated the migration. After some 20 seconds the virtual machine was on its new host and that without missing a single ping packet. I must say, I am quite impressed!

20 seconds is quite a difference from the usual work flow involved in moving systems from an old host to a new host before the world became virtualized. Either you had to move some disks around between the machines or you had to install both the operating system and the necessary applications on the new host before moving the data. Both methods gives you a significant amount of time where the service is unavailable, especially if you hit some kind of problems. Now, it is done with the push of four buttons without having to the take service down at all.

Needless to say, I just love it! :-)

Posted: July 5th, 2008 under Storage, VMware by Frode.
Comments: none

After installing ESXi to the PE R200 servers and doing some initial testing it was time to move the servers to their new home. We have a primitive server room here at our hall of residence which I am happily abusing. I say primitive since there are no rack cabinets installed and the cooling is inadequate. For example, the current temperature in the room is 37C – way over the limits for a healthy server room. Before you ask: yeah, it is much cooler when it is not 20C and above outside…

But, since I am not too keen on having four noisy rack servers less than two meters away from my bed, it is a good alternative.

As I am trying to simulate a large network, I have installed a number of NICs in each server to give them multiple paths to both storage, management and the internet. The current setup is more or less:

• One NIC for administration
• Two NICs for iSCSI / VMotion traffic
• One NIC for internet traffic

Paragon is a bit different since it is hosting most of my critical stuff such as documents, pictures, email and web pages. Since this is a PowerEdge 2850, I only have three PCI-X slots available. There are two onboard NICs and I had two Intel PRO1000/MT cards from another project. Since I need two NICs to the internet due to some restrictions on the active MAC addresses on the residence network, which is out of my control, I need a fifth NIC for management. The only available card I had was a 3Com 3c905. As I was pretty much out of luck at this point, the card was too old for the PCI-X slot. So, no management NIC in paragon.

The solution was to take advantage of VLAN seperation which I had already planned to use to seperate management traffic and iSCSI/VMotion. So, I just added the two NICs from paragon which should carry iSCSI traffic to the management VLAN. Problem solved :-)

As said, my setup involves both NIC teaming and VLANs. The teaming is a bit of a problem since the support in VMware is rather poor compared to Linux’. The only available options for splitting traffic between the two NICs are:

• Route based on the originating virtual port ID
• Route based on IP hash
• Route based on source MAC
• Use explicit failover order

All of these modes failes to give me the full 2Gbit/s the NICs are capable of since most of the traffic are headed either to or from the iSCSI server which will, naturally, have the same IP address every time. Since I do not have a lot of virtual machines running iSCSI, the routing based on the originating virtual port ID will solve my problem either.
In Linux you have a number of options in the bonding module, including both ip hash and source MAC. Since I want throughput, I have chosen to use the simplest mode where packets are transmitted using RR (Round Robin) between the NICs available. Another possibility could have been the 802.3ad mode which my switch supports, but it looks like the RR mode is doing a better job with the load balancing. I have not done any significant test to verify how incoming frames are distributed, but my switch are configured to group the different ports together in 802.3ad mode.

Moving further along with the setup, I installed a simple KVM switch which I had from another old project. It just makes some administration easier since I probably will be reinstalling the R200s quite some times before I am done with this project.

Pictures of the setup will be available as soon as my laptop is back online :-)

Posted: July 5th, 2008 under Hardware, Storage, VMware by Frode.
Comments: none

The first release candidate (RC1) for VMware Server is out. VMware Server is VMware’s free virtualization product. It requires either Windows or Linux as the operating system and installs itself as an application/daemon/service which you can run on your private machine.

VMware Server 1.0 was more or less the GSX server revamped and released to the public for free. It was quite slow and had some problems keeping up with the Linux kernel release cycle. As the application requires some kernel modules to be built it is dependent upon the kernel source. The kernel source can differ quite a lot from release to release and will therefore break the module building process from time to time. For those of use which are either living in the “bleeding edge” world or are patching regularly due to security problems, this turned out to be a major problem.

VMWare Server 2.0 have been a much nicer experience, at least for me. The modules are no longer a problem since VMWare have released beta 1, beta 2 and now RC 1 in accordance with the Linux kernel release cycle. The performance issues are mostly resolved too since 2.0 is using larger parts of ESX internally. Or, I can not say that for sure, but the config files suggests that 2.0 is using ESX components.

So, why would anyone pay for ESX or ESXi when you can get Server for free? Well, the first problem is manageability. ESX and ESXi can be managed from VirtualCenter Infrastructure where you can add your whole data center and play around with the machines. The status for Server 2.0 RC1 is that you only can use the VirtualCenter Infrastructure Client to manage a single host at the time. You have to connect directly to the server running VMware Server to manage it.

Ocourse, in VMware Server you have to do all the iSCSI magic in Linux instead of letting VMware handle it. There are no support for VMotion, DRS or HA either. So, VMware Server is not an enterprise class application.

Another problem is that you have to maintain an operating system in addition to VMware Server. By being dependent on an operating system which is not tailored for the hypervisor you also get some more overhead. This means that you will have fewer resources available for the virtual machines.

But, VMware Server is quite fun to play around with. It is actually perfect for my storage server which runs Linux instead of ESX because I need access to it by SMB, NFS and iSCSI at the same time.

So, quickly summarized: VMware ESX & ESXi for your production servers, VMware Server (hey, it’s free!) for your single server at home. Or, naturally, if you are on a budget :-)

Posted: July 3rd, 2008 under VMware by Frode.
Comments: 1

My Dell PowerEdge R200 servers arrived yesterday. New hardware is always fun and two servers is twice the funof just one!

The PE R200′s are quite simple. Built in an 1U case, Dell have fitted the servers with two disk drives, a single CPU (in my case, a Xeon X3320, Quad Core 2.5GHz with 2x3MB cache), four RAM slots supporting  a maximum of 8Gig, two PCI-E slots (x8 and x4) and two NICs (Broadcom NetExtreme BCM5721). The SATA controller is the Intel CH9 chipset. More information is available at Dell’s site. For those interested I have some pictures in my gallery.

After some fooling around with the hardware, changing one of the disk drives (from 80GB -> 640GB), adding an Intel PRO1000 PT Dual Port NIC and 4Gigs of RAM in both the servers they were ready for some experimenting.

As the goals is to learn virtualization software, I downloaded the VMware ESXi Installable ISO and burned it to a CDRW. The installation is surprisingly fast – you just have to select which disk drive it should install to and then you are done. Everything is done in just a few minutes. Installing the VMWare ESX 3.5 is more advanced with a lot of unnecessary choices (time zone, keyboard language etc.) because of the service console.

Given the choice between the fast, uncustomizable install and the slow, customizable, I usually select the customizable one because of the fear of having to customize the install afterward. A lot of the choices done in the install is often hidden in config files and databases after the install is done, so I just take the hard way during the install to save time later on. But, with ESXi, there are not a lot of options to customize. If I first wish to run a hypervisor as the operating system, I do not need support for running Linux or Windows on the same host at the same time, I can just run them as virtual machines. Therefore, it is no problem that the install wishes to use the whole disk for VMWare. The clock synchronization issue is solved by using NTP and I really do not care about the keyboard language as I am using VMvare Infrastructure Client to manage the host. As long as the system recognizes my password locally I am happy. VMware have done quite a nice job to hide a lot of the unnecessary questions with ESXi. All the relevant options are available after the install by logging on to the service program locally. As it turned out, I only had to change the NIC for management and set a root password.

Another nice feature with ESXi is that it has its own interface for hardware monitoring. This is something ESX 3.5 is missing and requires me to install a third party package in the service console for IPMI support.

So, if you ask me, I would say that ESXi is the way to go with hypervisors. Out with huge packages with their own operating system for management, in with small, easy installable packages. I wonder if I could make the ESXi network installable so that I do not have to fiddle around with CDs….

Posted: July 1st, 2008 under Hardware, VMware by Frode.
Comments: 17

To really unlock the potential in VMWware Infrastructure you need some kind of network storage. The alternatives are few and are mostly very expensive. Fibre Channel has been the only alternative for quite some time, but it is so expensive that only very large organizations could afford it.  It demands both dedicated host adapters (HBA), switches and storage devices.

iSCSI is the new player on the block. By using IP to transfer blocks to the storage devices, a large amount of infrastructure can be reused to create a SAN – or you can just put your SAN on your existing network. This eliminates the cost problem while still delivering quite good performance. As with FC you can buy dedicated HBAs and switches, but they are expensive.

So, I have taken the low-cost road to iSCSI. As you can see on the on the lab page, my storage server is a home made 3U server with 15 500Gig disk drives. The operating system of choice is Linux which gives me a number of choices for the iSCSI software package. After testing a few of them, I chose ietd, or “The iSCSI Enterprise Target”. The reason for choosing ietd was that it supports VMware and are still being developed.

Some early benchmarks suggests that ietd in combination with Open-iSCSI gives me close to 108MB/s over a single 1Gbit/s link. Quite nice if you ask me :-)

Posted: July 1st, 2008 under Storage by Frode.
Comments: none