As reported earlier, the Update Manager (UM) does not use the fully qualified domain name (FQDN) when it connects to the UM server. The update will therefore fail if you don’t have the UM Server on the same search domain as the ESX(i) servers you are trying to update.

After reading some comments on the net and doing some research, I realized that the UM server no longer sets the PatchDepotUrl variable under HostConfig in vci-integrity.xml. This file is the configuration for the UM server and it located under C:\program files\vmware\Infrastructure\Update Manager if you haven’t modified your installation. By changing, or rather setting the PatchDepotUrl to a path containing the FQDN of the UM server (for example http://winserver.lab.frode.biz/vci/hostupdates/hostupdate) the clients use this url to get updates instead of the default, non-working one with the windows machine name. I seem to remember that it was using the FQDN in Update 1. Anyone know why VMware changed the default behaviour?

Posted: July 31st, 2008 under VMware by Frode.
Comments: 1

Just to follow up on the post about Serial console over IPMI: The ESX(i) supports serial console, but you have to configure it under “Configuration –> Advanced Settings –> VMKernel –> Boot”. I didn’t get a real console on the VMware ESXi (I didn’t try to hard neither), but I got the debug output from the kernel which should be enough if things are working properly. Later on I might try to find out how I get a real, input enabled console so that I for example can remotely change the IP address on the management interface since it is reported that the kernel might change interface names during upgrades which makes the service console unavailable.

Posted: July 31st, 2008 under Hardware, VMware by Frode.
Comments: none

The PowerEdge R200 is equiped with an IPMI controller which enables you to control power settings and read sensors over LAN. The version on the R200 is 1.5 which is rather old and outdated, but it does the job.

One of the more hidden features of the IPMI controller is the ability to support serial console over IPMI. The documentation for this feature is rather non-existent as far as I know, and it requires a number of steps to work properly.

To get it working you need the following:

• Enter the BIOS and set Serial to BMC NIC under Integrated Pheriphals
• Set the Serial Port to Serial 1 and 19200
• Make sure the IPMI controller is enabled, have a administrative password and an IP address
• Install the Dell Linux Remote Access Utilities from Dell’s support pages  (you just want the “linux/bmc/osabmcutil9g”-package.
• Start the dsm_bmu_solproxy32 service and telnet to localhost on port 623. From there you can connect to your IPMI hosts.
• Please read http://support.euro.dell.com/support/edocs/systems/pe2850/en/ug/t1390ab.htm to see how different system keys such as F1, F2 and Ctrl+Alt+Del are remapped on the serial console.

That’s it. No more visits to the server room :-) This procedure should work for all the IPMI v1.5 BMC’s from Dell. The IPMI v2.0 have Serial-Over-LAN support directly in the ipmitool package so you don’t need the solproxy. The host should be configured in the same way though.

Posted: July 31st, 2008 under Hardware by Frode.
Comments: 3

As said earlier, VMware is releasing ESXi for free starting today. The CD image is available from VMWare’s page. This is a great offer from VMware :-D

Posted: July 28th, 2008 under VMware by Frode.
Comments: 5

After upgrading from 2.5 Update 1 to Update 2 the Update Manager failed to scan hosts. It said that the metadata was missing. The metadata is supposed to be downloaded by the update manager on a regular basis and it worked nicely in Update 1.

What puzzled me even more was that both my VI servers failed with the same message. Downgrading the Update Manager (UM) to the version from Update 1 fixed the problem, but that didn’t comply with my wish for running Update 2.

After reading some logs I discovered that my ESX 3.5 Update 2 host is looking for “http://winserver:80/vci/hostupdates/hostupdate/esx/esx-3.5.0/contents.xml.sig” when it tries to scan for updates. This file is created by the UM when it downloads updates. Or, it should have been created.

After looking in the default datastore for the UM I realized that there were only a single file there, namely “update_metadata.xml”. This file describes the different updates available so that the Update Manager Client (UMC) can display the available patches. In UM Update 1 there were a number of folders and subfolders in the datastore folder too: “hostupdates/hostupdate/esx/esx-3.5.0″ for example. As of UM Update 2, these folders are no longer created. No wonder why the update scan fails miserably…

After taking a quick look at the UM configuration file (C:\Program Files\VMware\Infrastructure\Update Manager\vci-integrity.xml in my case) I see that the download location for the patches are https://www.vmware.com/PatchManagementSystem/patchmanagement . This page is just a placeholder for a redirect to another page: http://vip-patchmgmtwlc.vmware.com:7004/patchMdSvc?WSDL . The error is quiteeasy to spot: the hostname doesn’t resolve.. But, the same page is referred to in UM Update 1. So, no solution yet…

Update 1 @ 2008-07-28: After reinstalling the Update Manager, changing ports and flushing the database it looks like it is downloading files again. Minus 10 points to VMware for extremely poor and irrelevant logging, plus 10 points to me for fighting with UM for most of the evening. VMware: please, fix your error messages :-)

Update 2 @ 2008-07-28: Even though I specify that the UM should identify itself as 192.168.20.3, the esx host tries <machine name>. This gives me some trouble since the machine is outside the default search domain in DNS and therefore it will not resolve. The solution is to set the DNS settings manually, but I really believe that the UM should use the FQDN to identify itself instead of just the machine name as it would solve a lot of potential problems.

Update 3 @ 2008-08-02: Solved!

Posted: July 27th, 2008 under VMware by Frode.
Comments: 1

In my previous post I asked for hot cloning/backup and IPMI support. What I didn’t know at the time of writing was that VMWare already had plans for implementing these features. Reading through the changelog for VMware ESX(i) 3.5 Update 2 I notice that both IPMI support and the hot migrating is listed as two of many new features. That was quick :-> For all the other changes, please read the changelog :-)

Currently I’m busy updating my servers :-)

Update 1 @ 2008.07.27: Looks like I was a bit quick to declare victory. I can’t find any IPMI settings or support for Storage VMotion in the new VIC :(

Update 2 @ 2008.07.27: I’m happy to report that VMware ESX 3.5 Update 2 now have support for reporting hardware status to the VIC. The support is the same as ESXi.

Posted: July 27th, 2008 under VMware by Frode.
Comments: none

After using the Infrastructure for some time, I have a list of features I want to see in the next VIC:

1) Live cloning or backup:

I know that VMware have consolidated backup to provide this function, but it requires either another backup program acting as a frontend or a number of command line commands. I’m not scared by the CLI, but the windows command line is rather ugly and pretty much useless. Powershell is supposedly better, but it requires that I read some documentation before using it. I want a quick command integrated in the VIC which allowes me to take a complete copy of my virtual machine and archive it somewhere outside the datastore which it currently resides in. Give me an option to create a complete backup ready to be powered on somewhere else without having to mess around with CLI commands and/or third party backup systems!

2) Storage VMotion GUI integration:

As storage VMotion is one of the new and fancy features in ESX 3.5 / ESX(i), it would be quite nice to do this from the GUI. I suspect that it is scheduled to be integrated in the VIC in either 2.5 Update 2 or 3.0, but I want it now. Again, I know about the different CLI options, but then again: when I have a fancy GUI, why can’t we use it? :-) Thanks to Andrew Kutz for the third party VIC plugin which enables SVMotion in the VIC :-)

3) Linux support for VMware Infrastructure:

I’m a big fan of open source systems and are currently running Linux on all my desktops and laptops. As I have access to the academic part of MSDN (MSDNAA), the cost of aquiring Windows 2003/2008 are virtually zero. Even so, i really want VMware Infrastructure, both the server and the client part, to be able to run on Linux. Currently, my Infrastructure installation is running inside a virtual machine on one of my ESX hosts. This isn’t an optimal solution and I’m pretty sure that VMware does not support this configuration. Being able to run Infrastructure on a linux host instead would mean that I could move it to my storage server. That would mean that I will have one less VM to worry about.

Another problem with the dependence on Windows Server is that it adds significantly to the total price of the virtualization solution. Windows 2003 costs at least $999 while Linux is free. Removing $999 from the total would be nice :-) Removing a Windows host from the list of systems to administrate is even nicer :-)

But, please note this: I don’t want a web client instead of the VIC. The web clients are both difficult to use and extremely slow compared to the current VI client.

4) IPMI integration:

I would be pretty cool to be able to control the power settings of my hosts by sending IPMI commands to the hosts directly from the VIC. As the DRS have some support for controlling the power status (VMware Distributed Power Management) of hosts it might be valuable to have IPMI support directly in the VIC.
Disclaimer: I don’t know how VMware are powering on hosts again after taking them offline when they are idle. I guess that they are using Etherwake to wake up hosts since they have no way of knowing the ip address of the IPMI service processor.

Well, this list might be extended later on :-)

Posted: July 26th, 2008 under Random thoughts, VMware by Frode.
Comments: 2

Being both a Norwegian, and a newbie when it comes to iSCSI and ESX(i), it shoudn’t come as a surprise that I had my fair share of problems getting iSCSI on ESX to work.  As for those of you who doesn’t know Norwegians, we are pretty much known for completely ignoring every piece of documentation related to a new gadget. This might pose some problems when the specifications have strict requirements on for exmaple the format of strings. As I discovered after quite some time debugging with tcpdump, debug mode of ietd and a massive amount of retries, ESX have some requirements for the iSCSI Qualified Name (IQN). The iQN should be in the following format according to RFC 3720:

Format: iqn.yyyy-mm.{reversed domain name} (e.g. iqn.2001-04.com.acme:storage.tape.sys1.xyz)

As I was pretty eager to get things up and running, I just constructed my own IQN:

iqn.2008.06.biz.frode:storage.vmware

As you can verify, this IQN is valid. I also tried to connect and mount this target with open-iscsi on my linux host. It worked.

So, the next challenge was to get ESX to recognize this target. After sorting out the firewall issues and reading some changelogs for ietd to verify that ESX actually acepted this software target (it does) I added my iSCSI server to ESX’ iSCSI software adapter. As I soon discovered, it did not work.

After checking with tcpdump, I saw that packets were flowing back and forth between my ESX host and the iSCSI server. But, there were no targets showing up in the ESX configuration. A bit frustrating since I had no idea about what was wrong. ESX did not help me either since it did not return a single error message, not even when I tried to scan the host from the ESX console. Since the same target worked as a charm in Linux, I was even more puzzled.

After quite some time trying different options in ietd and rescanning my iSCSI server while duming the network traffic, I tried to change the IQN on the target from iqn.2008.06.biz.frode:storage.vmware to iqn.2007.04.biz.frode:storage.vmware. Voilà! Suddently my targets and LUNs came up in ESX!

As you can see, the only difference in the IQN is the date. As specified in RFC 3720 – Internet Small Computer Systems Interface section 3.2.6.3.1, the IQN date should reflect a date when the naming authority owned the domain name. The RFC says nothing about what happens if you aquire the domain less than a month before you create the target and therefore sets the date to the same month and year as you are currently in as dictated by the RFC. Linux and Open-iSCSI didn’t care about the date being the same as the current one and happily connected to it. ESX on the other hand rejected it without saying a word as the current month and year was June 2008 (2008.06) when I tried it. Maybe a bug in the implementation of iSCSI used in ESX? I don’t know, I’m still a bit puzzled :-)

But, the lesson learned from this story is simple: Don’t use the same month and year as you are currently in if you want ESX to see your targets :-)

Posted: July 26th, 2008 under Storage, VMware by Frode.
Comments: none

As reported in the previous post, ESXi will be avaiable for free on July 28th.  As this is great news for all the virtualization geeks and sysadmins out there, I would like to share some of my experiences with ESXi.

First of all, installation is a piece of cake: Boot with the CD (I haven’t tried to boot the installation from iSCSI or network yet), press return a few times, watch the server reboot and you’re done. Configurating the server is just as easy, the administration interface is clean and self explaining. As long as all your hardware is supported by ESXi, the installation should be done in just a few minutes.

As ESXi lacks the service console, you don’t have the option of installing a bunch of administration software from your hardware vendor. I don’t miss those features as the ESXi have a hardware monitor service installed which is available from the VirtualCenter Infrastructure Client. As most of the hardware vendors are using IPMI for LOM (Lights Out Management), you can aggregate your monitoring to either another host or a virtual server running on the ESXi and pull the information from the network instead. By removing the service console, the ESXi is also a lot more secure since there are less software which can be exploited and just a few services available from the network. Naturally, if you make the service console available only from a dedicated management network, you are home free, but that is not an option on a number of installations.

Managing ESXi is just as easy as the VirtualCenter Infrastructure Client (VIC) handles all the necessary configuration options. Rebooting the ESX host is a thing of the past as long as you don’t change the iSCSI host identification. Adding new network switches and storage are done with just a few clicks as with the ESX version. All in all, the VIC makes it easy to manage servers.

The biggest advantage of ESXi is the small footprint and the ease of installation. Making it small makes it more stable since there are fewer applications that can fail. With fewer applications, there are fewer patches needed to make it secure too so that the number of required reboots are kept on a minimum. This is important since enabling VMotion requires a number of expensive licenses and an iSCSI system which adds to the total cost of the solution.

As said before, I believe that the ESXi is the right way to develop virtualization solutions. In the future I believe that there will be few machines sold without an integrated virtualization solutions since it gives the users the opportunity to maximize the utilization of their servers. Moving further into the future, I think that we will see desktops with the same solutions. Possible not for the same reasons (preferable utilization), but for security reasons.

Posted: July 23rd, 2008 under VMware by Frode.
Comments: none

According to InformationWeek, VMware ESXi will be available for free from July 28th. Previously priced at $495, the battle with Microsoft Hyper-V has made VMware to release the ESXi as a free download. As a fan of VMware, I am happy to say that this is a bold move. ESXi is an enterprise virtualization solution running without an extra operating system. As you probably know, to run Hyper-V you need WIndows Server 2008. This operating system is pretty expensive ($999). So, the previous price of $495 was a bargain if you do not need the Windows Server to run applications. Needless to say, the new price is unbeatable!

VMware ESXi is the old ESX server without the service console. It can be installed to a disk drive or you can boot it from a USB stick or from flash memory. With a footprint of only 32MB you do not lose disk space to unimportant stuff you do not use anyway. The performance is great, and the administration is pretty easy due to the integration with VMware Infrastructure.

Sadly, you still need to pay for the VMware VirtualCenter to be able to use all the cool stuff such as VMotion, VMware HA and DRS. You also have to use VirtualCenter if you want to manage your entire datacenter within a single client application. The VirtualCenter solution is pretty cool, although it is too expensive for students as myself which plays around with the software to learn instead of saving money by reducing the number of servers in an organization.

It is easy to make recommandations if you want to play around with virtualization: Get VMware ESXi on July 28th for free. There are no real alternatives on the market today unless you want another operating system together with your virtualization solution. If so, there are a number of alternatives for free: VMware Server, KVM and Xen for example.

As a sidenote: VMWare, your hostname (little-black-box.vmware.com) keeps showing up in my web statistics. Can you give me a license for VirtualCenter to support this blog? :-)

Posted: July 23rd, 2008 under VMware by Frode.
Comments: 3