Sunday, March 9, 2008

ESX 3.5 on a whitebox

It has been very quiet from my end for the past weeks because I was very busy at a client & at the same time spending all my free time working on my ESX-on-whitebox hardware project. After being inspired by some colleagues, I decided to order the following hardware:
  • Asus P5BP-E/4L motherboard
    This motherboard supports an Intel S775 processor, has VGA and audio onboard and most importantly, the LAN controllers on this motherboard are ESX certified (Broadcom 57xx chipset).

  • Intel Q6600 Quad Core processor (2.4 GHz) and 8 GB ECC RAM (4x 2GB)
    Just to be sure I have enough CPU power and memory resource pools :)

  • Dell Perc 5i Integrated SAS Controller
    My colleagues advised me that storage was the biggest bottleneck in their ESX whiteboxes (based around the very nice Asus P5M2/SAS board). I decided to go for a dedicated hardware controller. I picked up the Dell Perc 5i controller, which is more or less a rebranded LSI Logic 8408 SAS controller on EBay with 256MB of RAM and a battery backup unit for about 175 EUR.

    The main advantage of SAS controllers is that they also support the (cheaper) SATA consumer drives. A quick test confirmed this; I had absolutely no problems at all with this controller & even flashed the latest LSI Logic firmware to it :).

    Maybe of interest for some: the later Dell firmwares and also the later LSI logic firmwares for this controller provide support for Write Back without a BBU present.

  • SATA to SAS cables
    The Dell Perc 5i has SFF-8484 SAS connectors on board, so I purchased two Adaptec SFF-8484 to 4xSATA cables from a nearby store to attach all the drives.

  • 8 Seagate SATA harddisks (4x 1TB and 4x 200GB)
    Space... loads of space.
The hardest thing was getting all these disks in my Silentmaxx ST11 casing; it required some case modding and loads of patience to get everything well fitted. The 500W PSU that is necessary to provide enough juice, was recycled from an Antec Sonata case. I also added a small 3Com 3C905 100Mbps card for my ISP modem connection.

The installation of ESX 3.5 was a piece of a cake & and I can confirm that the above hardware works like a charm. For those interested, I also noticed that ESX 3.5 supports the ICH7 SATA controllers (found on many consumer motherboards as well). I think -- but this has to be confirmed by someone else -- that you need to configure your ICH7 disks in a RAID before the ESX kernel will accept them as a storage pool.

20 comments:

Anonymous said...

Hi there,

Just wanted to report that you do not need to create a raid in order to have the sata hdd recognized by ESX. I have tested this with an Asus barebone (V3-P5945GC@90 Euro)with a single samsung 750GB drive. ESX installs successfully the only issue I have is that the onboard nic is not recognized. Hope this helps.

Grtz,

Herc.

Gavin Adams said...

One reason for an actual SAS/SATA controller is RAID support. If you don't have RAID I assume ESX 3.5/3i will see the the storage as JBOD. A drive failure could be bad.

Anonymous said...

How many VMs are you running? How's the performance?

Cheers,
James

Anonymous said...

Do you know if 3.5 works on intel ESB2 controller?

Tim Jacobs said...

James,

At the moment I am running 7 virtual machines, which is quite a low load:
- Two Windows 2008 servers
- Two Windows 2003 servers
- Two Windows XP machines
- One m0n0wall firewall/router

The total CPU taken by these 7 virtual machines at idle is 440MHz, the total memory taken 2.65 GB (512Mb for the W2K8 machines, roughly 300Mb for the XP machines, the rest for the W2K3 servers).


Regarding the question whether the Intel ESB2 controller is compatible, the only answer I can give is to look at the official hardware compatibility guides (http://www.vmware.com/support/pubs/vi_pages/vi_pubs_35.html) which document the kernel modules that ESX supports. If it is not there, it is not supported. Feel free to provide a piece of hardware for testing :D.

Bye,

Tim

Dave Mishchenko said...

Regarding the ICH7 controller ESX will only recognize drives connected in a non-RAID mode. That's the case with most of the embedded (low end) RAID controllers found on lower end motherboards, as they require a software component to function which ESX does not provide.

Anonymous said...

I've had my eye on that PERC5i SAS controller for some time now. But I keep hearing rumors about it being slow. Care to comment how's the beast performing, from your own experience?

Tim Jacobs said...

Regarding the speed of the Dell Perc 5i controller: I've read people mentioning write speeds of up to 250 MB/s on that controller, so the throughput capacity surely is there. It depends a lot on the disks you use on the controller and whether you put it in a 1x/4x/8x PCIe slot.

I regularly see peaks of 60MB/s write on a four-SATA-disk 7200 RPM RAID5 array (a network copy to a virtual fileserver), without any other virtual machine noticing anything of that load. That 60 MB/s is roughly the network throughput you can attain over a gigabit network.

By stressing a virtual machine itself, I can achieve write speeds of up to 150 MB/sec, again without any other virtual machine noticing performance loss. I suspect that even higher performance can be gained by using 10k/15k RPM disks in a real SAS setup.

Having said that, do realize that the Perc 5i is an "old" controller and that there are more performant controllers on the market. However, for a second hand $175 deal (which is already expensive if you look at some eBay bargains you can get today), you don't hear me complain at all :D.

Chris said...

Tim,

Where did you buy your P5BP-E/4L? The only sites online that have it are in Australia or the UK?

Thanks,

Anonymous said...

Tim,

Does ESX 3.5 also work with the local SATA controller on this mainboard? I first want to use local storage and after a few weeks a dedicated RAID controller.

Thanks!

Tim Jacobs said...

Yes, this board uses an Intel ICHx SATA controller which is supported by ESX 3.5.

Anonymous said...

Hi tim

this looks ideal for my requirements, have you tested ESXi with this setup ?

cheers

Thomas said...

Hello,

I also want to know if you have tested this setup with esxi?
Would appreciate an answer, as I am in the process of building an esx/esxi compatible system my self.

Tim Jacobs said...

Hi Thomas & anonymous poster,

I have not explicitly tested the board with ESXi, but since the hardware compatibility list is the same for ESX 3.5 and ESX 3i, I do not expect any issues at all.

Kuniko said...

Keep up the good work.

Hannes said...

Tim, nice system ... may I ask what disk you built in?

Cheers,
Hannes

Tim Jacobs said...

Hi Hannes,

I put in 8 Seagate Barracuda disks. 4 x 7200.10 200GB series for VM storage and 4x 1TB 7200.11 series storage for data storage.

Bye!

Tim

Anonymous said...

Tim,

Nice machine. I have been putting a whitebox together as well. This is my first experience with raid and a server quality machine. I am a web developer not really an expert in Systems or networks.

Anyway, I also bought a perc5i off ebay. I have a question, I can definitely configure this via setup ctr+r, but what if I need to configure this remotely running esx 3.5? Do you know of a way?

Also, I configured mine raid 10 64k stripe. Do you have any performance tips for me? I do have the battery backup. I will be running only a few VMs - Centos Server, Window 2008, and mono firewall.

-Thanks
Adam

DM said...

How do you tell if a disk has failed?

Tim Jacobs said...

Hey DM,

I blogged about how you can use the LSI tools under VMware ESX to detect disk failures. For more information, see:

http://timjacobs.blogspot.com/2008/05/installing-lsi-logic-raid-monitoring.html

Bye!

Tim