I promised some more details on the R200 and ESXi 4.0. Read more »

Posted: June 10th, 2009 under Hardware, VMware by Frode.
Comments: 2

So, it was time to make the virtualization a bit more serious and the same time solve my hosting problem. After a quick survey of the hardware available, I bought a Dell PowerEdge R710: Read more »

Posted: June 10th, 2009 under Hardware, VMware by Frode.
Comments: 2

Yes, the Dell PowerEdge R200 runs vSphere/VMware ESXi 4.0. More details to come as I revive this blog :-)

Posted: May 23rd, 2009 under Hardware, VMware by Frode.
Comments: 1

I’m getting some comments about the “Can’t write to device” errors. At first I thought it might be a problem with 3.5U2 since I had no problems installing the 3.5U1 on the R200, but my reinstall with 3.5U2 went without a hitch. So, here are some other ideas:

• Bad disk drive / disk drive firmware / disk related problems (I have changed from the disk delivered by Dell to a WDC 640Gig)?
• Old bios?
• Corrupted CD?
• Bad luck?

If anyone know the source of this problem, please make a comment :-)

Posted: August 29th, 2008 under Hardware, VMware by Frode.
Comments: 1

So, ok, the new semester started today. As always, I’m a bit lazy on the first day so I took the opportunity to work from home instead of going to school.

It turned out be a rather ok day when I finally got out of bed. I’ve made a quick front page and a suggestion for the content in my project report. As you can read on my “Why”-page, the goal is to virtualize computer labs for students to reduce both cost and administration and at the same time deliver a better service.

After a quick look around, I chose to focus on the following virtualization solutions in the next months:

• VMware ESX/ESX(i)/Server/VDI
• Microsoft Hyper-V
• Xen
• Sun Virtualbox/xVM/Solaris Containers
• VirtualIron
• Virtuozzo
• QEMU

Are there any other solutions I should look at? I know about the different IBM/HP solutions, but they requires special hardware which are pretty expensive.

Posted: August 18th, 2008 under Hardware, Hyper-V, VMware, Xen by Frode.
Comments: 1

Just to follow up on the post about Serial console over IPMI: The ESX(i) supports serial console, but you have to configure it under “Configuration –> Advanced Settings –> VMKernel –> Boot”. I didn’t get a real console on the VMware ESXi (I didn’t try to hard neither), but I got the debug output from the kernel which should be enough if things are working properly. Later on I might try to find out how I get a real, input enabled console so that I for example can remotely change the IP address on the management interface since it is reported that the kernel might change interface names during upgrades which makes the service console unavailable.

Posted: July 31st, 2008 under Hardware, VMware by Frode.
Comments: none

The PowerEdge R200 is equiped with an IPMI controller which enables you to control power settings and read sensors over LAN. The version on the R200 is 1.5 which is rather old and outdated, but it does the job.

One of the more hidden features of the IPMI controller is the ability to support serial console over IPMI. The documentation for this feature is rather non-existent as far as I know, and it requires a number of steps to work properly.

To get it working you need the following:

• Enter the BIOS and set Serial to BMC NIC under Integrated Pheriphals
• Set the Serial Port to Serial 1 and 19200
• Make sure the IPMI controller is enabled, have a administrative password and an IP address
• Install the Dell Linux Remote Access Utilities from Dell’s support pages  (you just want the “linux/bmc/osabmcutil9g”-package.
• Start the dsm_bmu_solproxy32 service and telnet to localhost on port 623. From there you can connect to your IPMI hosts.
• Please read http://support.euro.dell.com/support/edocs/systems/pe2850/en/ug/t1390ab.htm to see how different system keys such as F1, F2 and Ctrl+Alt+Del are remapped on the serial console.

That’s it. No more visits to the server room :-) This procedure should work for all the IPMI v1.5 BMC’s from Dell. The IPMI v2.0 have Serial-Over-LAN support directly in the ipmitool package so you don’t need the solproxy. The host should be configured in the same way though.

Posted: July 31st, 2008 under Hardware by Frode.
Comments: 3

As said earlier, our server room is not equiped with the latest and most powerful cooling equipment available. As it is summer in Norway, the temperature in the room could get above 35C. This room is not optimal, but it is the best solution available to me. So, in order to track the temperature changes I have made a quick hack with MRTG, perl and ipmi-sensors to read the temperature from the IPMI 1.5 BMC on the PE2850 and PER200s. The result is available at http://decagon.frode.biz/ipmistat. As you can see, it also displays fan speeds.

Posted: July 21st, 2008 under Hardware by Frode.
Comments: none

After installing ESXi to the PE R200 servers and doing some initial testing it was time to move the servers to their new home. We have a primitive server room here at our hall of residence which I am happily abusing. I say primitive since there are no rack cabinets installed and the cooling is inadequate. For example, the current temperature in the room is 37C – way over the limits for a healthy server room. Before you ask: yeah, it is much cooler when it is not 20C and above outside…

But, since I am not too keen on having four noisy rack servers less than two meters away from my bed, it is a good alternative.

As I am trying to simulate a large network, I have installed a number of NICs in each server to give them multiple paths to both storage, management and the internet. The current setup is more or less:

• One NIC for administration
• Two NICs for iSCSI / VMotion traffic
• One NIC for internet traffic

Paragon is a bit different since it is hosting most of my critical stuff such as documents, pictures, email and web pages. Since this is a PowerEdge 2850, I only have three PCI-X slots available. There are two onboard NICs and I had two Intel PRO1000/MT cards from another project. Since I need two NICs to the internet due to some restrictions on the active MAC addresses on the residence network, which is out of my control, I need a fifth NIC for management. The only available card I had was a 3Com 3c905. As I was pretty much out of luck at this point, the card was too old for the PCI-X slot. So, no management NIC in paragon.

The solution was to take advantage of VLAN seperation which I had already planned to use to seperate management traffic and iSCSI/VMotion. So, I just added the two NICs from paragon which should carry iSCSI traffic to the management VLAN. Problem solved :-)

As said, my setup involves both NIC teaming and VLANs. The teaming is a bit of a problem since the support in VMware is rather poor compared to Linux’. The only available options for splitting traffic between the two NICs are:

• Route based on the originating virtual port ID
• Route based on IP hash
• Route based on source MAC
• Use explicit failover order

All of these modes failes to give me the full 2Gbit/s the NICs are capable of since most of the traffic are headed either to or from the iSCSI server which will, naturally, have the same IP address every time. Since I do not have a lot of virtual machines running iSCSI, the routing based on the originating virtual port ID will solve my problem either.
In Linux you have a number of options in the bonding module, including both ip hash and source MAC. Since I want throughput, I have chosen to use the simplest mode where packets are transmitted using RR (Round Robin) between the NICs available. Another possibility could have been the 802.3ad mode which my switch supports, but it looks like the RR mode is doing a better job with the load balancing. I have not done any significant test to verify how incoming frames are distributed, but my switch are configured to group the different ports together in 802.3ad mode.

Moving further along with the setup, I installed a simple KVM switch which I had from another old project. It just makes some administration easier since I probably will be reinstalling the R200s quite some times before I am done with this project.

Pictures of the setup will be available as soon as my laptop is back online :-)

Posted: July 5th, 2008 under Hardware, Storage, VMware by Frode.
Comments: none

My Dell PowerEdge R200 servers arrived yesterday. New hardware is always fun and two servers is twice the funof just one!

The PE R200′s are quite simple. Built in an 1U case, Dell have fitted the servers with two disk drives, a single CPU (in my case, a Xeon X3320, Quad Core 2.5GHz with 2x3MB cache), four RAM slots supporting  a maximum of 8Gig, two PCI-E slots (x8 and x4) and two NICs (Broadcom NetExtreme BCM5721). The SATA controller is the Intel CH9 chipset. More information is available at Dell’s site. For those interested I have some pictures in my gallery.

After some fooling around with the hardware, changing one of the disk drives (from 80GB -> 640GB), adding an Intel PRO1000 PT Dual Port NIC and 4Gigs of RAM in both the servers they were ready for some experimenting.

As the goals is to learn virtualization software, I downloaded the VMware ESXi Installable ISO and burned it to a CDRW. The installation is surprisingly fast – you just have to select which disk drive it should install to and then you are done. Everything is done in just a few minutes. Installing the VMWare ESX 3.5 is more advanced with a lot of unnecessary choices (time zone, keyboard language etc.) because of the service console.

Given the choice between the fast, uncustomizable install and the slow, customizable, I usually select the customizable one because of the fear of having to customize the install afterward. A lot of the choices done in the install is often hidden in config files and databases after the install is done, so I just take the hard way during the install to save time later on. But, with ESXi, there are not a lot of options to customize. If I first wish to run a hypervisor as the operating system, I do not need support for running Linux or Windows on the same host at the same time, I can just run them as virtual machines. Therefore, it is no problem that the install wishes to use the whole disk for VMWare. The clock synchronization issue is solved by using NTP and I really do not care about the keyboard language as I am using VMvare Infrastructure Client to manage the host. As long as the system recognizes my password locally I am happy. VMware have done quite a nice job to hide a lot of the unnecessary questions with ESXi. All the relevant options are available after the install by logging on to the service program locally. As it turned out, I only had to change the NIC for management and set a root password.

Another nice feature with ESXi is that it has its own interface for hardware monitoring. This is something ESX 3.5 is missing and requires me to install a third party package in the service console for IPMI support.

So, if you ask me, I would say that ESXi is the way to go with hypervisors. Out with huge packages with their own operating system for management, in with small, easy installable packages. I wonder if I could make the ESXi network installable so that I do not have to fiddle around with CDs….

Posted: July 1st, 2008 under Hardware, VMware by Frode.
Comments: 17