VMware isn’t very keen on sharing some of the details of ESXi, so it’s very nice that some of us virtualizing geeks are hacking their software to find new features.

One of the major problems with ESX(i) is the support for storage controllers – quite a few have discovered that their controller isn’t supported. Well, that’s a problem of the past from now on: You can just boot ESXi with PXE and still retain the configuration after reboot. You can even exploit this for making completely automatic deployment of ESXi. Kudos to Jim McCann for the guide!

Diskless ESXi: http://docs.google.com/View?docid=ddcwgcd6_4fs6s7jcf – check the buttom part to find out how you save your configuration so that it isn’t lost upon reboot :-)

Posted: August 26th, 2008 under Storage, VMware by Frode.
Comments: none

Being both a Norwegian, and a newbie when it comes to iSCSI and ESX(i), it shoudn’t come as a surprise that I had my fair share of problems getting iSCSI on ESX to work.  As for those of you who doesn’t know Norwegians, we are pretty much known for completely ignoring every piece of documentation related to a new gadget. This might pose some problems when the specifications have strict requirements on for exmaple the format of strings. As I discovered after quite some time debugging with tcpdump, debug mode of ietd and a massive amount of retries, ESX have some requirements for the iSCSI Qualified Name (IQN). The iQN should be in the following format according to RFC 3720:

Format: iqn.yyyy-mm.{reversed domain name} (e.g. iqn.2001-04.com.acme:storage.tape.sys1.xyz)

As I was pretty eager to get things up and running, I just constructed my own IQN:

iqn.2008.06.biz.frode:storage.vmware

As you can verify, this IQN is valid. I also tried to connect and mount this target with open-iscsi on my linux host. It worked.

So, the next challenge was to get ESX to recognize this target. After sorting out the firewall issues and reading some changelogs for ietd to verify that ESX actually acepted this software target (it does) I added my iSCSI server to ESX’ iSCSI software adapter. As I soon discovered, it did not work.

After checking with tcpdump, I saw that packets were flowing back and forth between my ESX host and the iSCSI server. But, there were no targets showing up in the ESX configuration. A bit frustrating since I had no idea about what was wrong. ESX did not help me either since it did not return a single error message, not even when I tried to scan the host from the ESX console. Since the same target worked as a charm in Linux, I was even more puzzled.

After quite some time trying different options in ietd and rescanning my iSCSI server while duming the network traffic, I tried to change the IQN on the target from iqn.2008.06.biz.frode:storage.vmware to iqn.2007.04.biz.frode:storage.vmware. Voilà! Suddently my targets and LUNs came up in ESX!

As you can see, the only difference in the IQN is the date. As specified in RFC 3720 – Internet Small Computer Systems Interface section 3.2.6.3.1, the IQN date should reflect a date when the naming authority owned the domain name. The RFC says nothing about what happens if you aquire the domain less than a month before you create the target and therefore sets the date to the same month and year as you are currently in as dictated by the RFC. Linux and Open-iSCSI didn’t care about the date being the same as the current one and happily connected to it. ESX on the other hand rejected it without saying a word as the current month and year was June 2008 (2008.06) when I tried it. Maybe a bug in the implementation of iSCSI used in ESX? I don’t know, I’m still a bit puzzled :-)

But, the lesson learned from this story is simple: Don’t use the same month and year as you are currently in if you want ESX to see your targets :-)

Posted: July 26th, 2008 under Storage, VMware by Frode.
Comments: none

One of the goals for this project was to try out the fancy thing that VMware calls VMotion. Basicly it is a system where you can move virtual machines around between physical hosts without having to power them down. By enabling VMotion you create an opportunity for virtualized load balancing in virtual data centers which is quite cool :-)

The VMotion requires that you have your virtual machines on some kind of centralized storage, for example iSCSI or Fibre Channel. When you or the load balancers initiate the transfers, ESX takes a snapshot of RAM content, moves it to the new hosts while concurrently logging changes from the log to the current state. When the RAM content are safely relocated to the new physical machine, the virtual machine are powered off and the change log are sent to the new host which are then powered on. This should appear as nothing have happened for the users outside the data center.

After setting up my lab, this was one of the first things I wanted to try out. After a quick install of Ubuntu on my new iSCSI disk, I initiated the migration. After some 20 seconds the virtual machine was on its new host and that without missing a single ping packet. I must say, I am quite impressed!

20 seconds is quite a difference from the usual work flow involved in moving systems from an old host to a new host before the world became virtualized. Either you had to move some disks around between the machines or you had to install both the operating system and the necessary applications on the new host before moving the data. Both methods gives you a significant amount of time where the service is unavailable, especially if you hit some kind of problems. Now, it is done with the push of four buttons without having to the take service down at all.

Needless to say, I just love it! :-)

Posted: July 5th, 2008 under Storage, VMware by Frode.
Comments: none

After installing ESXi to the PE R200 servers and doing some initial testing it was time to move the servers to their new home. We have a primitive server room here at our hall of residence which I am happily abusing. I say primitive since there are no rack cabinets installed and the cooling is inadequate. For example, the current temperature in the room is 37C – way over the limits for a healthy server room. Before you ask: yeah, it is much cooler when it is not 20C and above outside…

But, since I am not too keen on having four noisy rack servers less than two meters away from my bed, it is a good alternative.

As I am trying to simulate a large network, I have installed a number of NICs in each server to give them multiple paths to both storage, management and the internet. The current setup is more or less:

• One NIC for administration
• Two NICs for iSCSI / VMotion traffic
• One NIC for internet traffic

Paragon is a bit different since it is hosting most of my critical stuff such as documents, pictures, email and web pages. Since this is a PowerEdge 2850, I only have three PCI-X slots available. There are two onboard NICs and I had two Intel PRO1000/MT cards from another project. Since I need two NICs to the internet due to some restrictions on the active MAC addresses on the residence network, which is out of my control, I need a fifth NIC for management. The only available card I had was a 3Com 3c905. As I was pretty much out of luck at this point, the card was too old for the PCI-X slot. So, no management NIC in paragon.

The solution was to take advantage of VLAN seperation which I had already planned to use to seperate management traffic and iSCSI/VMotion. So, I just added the two NICs from paragon which should carry iSCSI traffic to the management VLAN. Problem solved :-)

As said, my setup involves both NIC teaming and VLANs. The teaming is a bit of a problem since the support in VMware is rather poor compared to Linux’. The only available options for splitting traffic between the two NICs are:

• Route based on the originating virtual port ID
• Route based on IP hash
• Route based on source MAC
• Use explicit failover order

All of these modes failes to give me the full 2Gbit/s the NICs are capable of since most of the traffic are headed either to or from the iSCSI server which will, naturally, have the same IP address every time. Since I do not have a lot of virtual machines running iSCSI, the routing based on the originating virtual port ID will solve my problem either.
In Linux you have a number of options in the bonding module, including both ip hash and source MAC. Since I want throughput, I have chosen to use the simplest mode where packets are transmitted using RR (Round Robin) between the NICs available. Another possibility could have been the 802.3ad mode which my switch supports, but it looks like the RR mode is doing a better job with the load balancing. I have not done any significant test to verify how incoming frames are distributed, but my switch are configured to group the different ports together in 802.3ad mode.

Moving further along with the setup, I installed a simple KVM switch which I had from another old project. It just makes some administration easier since I probably will be reinstalling the R200s quite some times before I am done with this project.

Pictures of the setup will be available as soon as my laptop is back online :-)

Posted: July 5th, 2008 under Hardware, Storage, VMware by Frode.
Comments: none

To really unlock the potential in VMWware Infrastructure you need some kind of network storage. The alternatives are few and are mostly very expensive. Fibre Channel has been the only alternative for quite some time, but it is so expensive that only very large organizations could afford it.  It demands both dedicated host adapters (HBA), switches and storage devices.

iSCSI is the new player on the block. By using IP to transfer blocks to the storage devices, a large amount of infrastructure can be reused to create a SAN – or you can just put your SAN on your existing network. This eliminates the cost problem while still delivering quite good performance. As with FC you can buy dedicated HBAs and switches, but they are expensive.

So, I have taken the low-cost road to iSCSI. As you can see on the on the lab page, my storage server is a home made 3U server with 15 500Gig disk drives. The operating system of choice is Linux which gives me a number of choices for the iSCSI software package. After testing a few of them, I chose ietd, or “The iSCSI Enterprise Target”. The reason for choosing ietd was that it supports VMware and are still being developed.

Some early benchmarks suggests that ietd in combination with Open-iSCSI gives me close to 108MB/s over a single 1Gbit/s link. Quite nice if you ask me :-)

Posted: July 1st, 2008 under Storage by Frode.
Comments: none