VMWare ESXi 5 I/O performance problem on IBM x3250

After upgrading to ESXi 5.0 from ESXi 4.1 Update 1 because I could not install it on my LSI 1064E hardware RAID, my IBM x3250 4364 server showed poor I/O performance when downloading stuff. In the logs the message was :

Device xxyyzz performance has deteriorated. I/O latency increased from average value of 80631 microseconds to 3395272 microseconds.

You can see the screenshot with the error logged below.

ESXi 5 hardware controller problem

ESXi 5 problem on LSI 1064E

My solutions are : try another pair of HDD’s, try another controller model (i only have this one), or try VMWare ESXi 3.5 U3, according to this post.

The RAID Bios version is the latest : 6.22.00
The firmware version is : 1.18.86.
It seems that there are 2 versions of LSI 1064E controller : 1 MB and 2 MB.
My IBM x3250 4364 has the 1 MB version. For those who have the 2 MB version (newer servers), here is a link.

I will update this post as soon as I find the best solution.

Update 1 – 23.11.2011: Leaving only one HDD made the ESXi 5 unable to boot. I’ve used the other HDD and i had the same result. So RAID1 in my case didn’t work for booting, but the data was there.
I used only one HDD, and installed ESXi on a USB stick. But the “performance has deteriorated” warning it’s still present.

Update 2 – 3.01.2012: Write caching was disabled for the LSI 1064E card . I enabled it, but i still have the same message.

5 comments

Skip to comment form

  1. An IO latency this high is mostly because of heavy copy actions.
    A linux environment could scale this by using ionice during heavy copy actions.

    The ESXi kernel has this functionality and you can scale IO in the vSphere client.

    If you execute a copy action on the command shell of the ESXi server though I’m not sure whether it’s possible to address the kernel functionality in copy actions.

    A note on your values; I consider both of them to be pretty high.
    A default SATA idle disk will probably have a latency between 5 and 12 milliseconds.

    80 Milliseconds (as you state) seems to be very high especially when it grows to 3,3 seconds.
    I’d say with these results either your controller has a defect or the disks.

    1. I’ve tested the HDD’s and they are OK. Also the controller works if i install directly an OS (Ubuntu or Windows7 – tested and no lags seen).
      The only problem that i see is with installing VMWare 4.1 U1, it gives me errors if my HDD’s are in RAID 1 Hardware (i didn’t test RAID 0). Also i used another 2 HDD’s just to test that. VMWare 4.1 U1 installs fine if I don’t use RAID.
      VMWare 3 works fine with HW RAID 1, but i can’t use 64 bit guests.
      The fastest tests that i can do are : leave only one HDD in RAID, or just use both HDD’s in non-RAID mode to see what happens.
      I/O performance suffers when i use either ftp or wget.

    • Cello on November 29, 2011 at 10:54 pm
    • Reply

    I had the same issue.
    I fix it enabling the cache of the hard disk in the raid array.
    The LSI 1064E disable cache write by default, but you can enable it using the LSIUtil.
    You have to go in the RAID ACTIONS -> CHANGE VOLUME SETTINGS and then write YES on the “Enable write caching” request.
    My VMs are a lot faster and I have no more messages in log file.

    • Johnny on November 14, 2012 at 11:02 am
    • Reply

    I have this issue as well.

    ESXI 5.0u1 on an IBM x3250 with an LSI 1064E controller.

    Did you ever find a solution to this?

  2. Probably the only solution is to use a better RAID controller. I’m having the same issue with other onboard RAID controllers like, IBM ServeRAID 8k-i, and SAS3041E (PCI-EX low value)

Leave a Reply to Bogdan Cancel reply

Your email address will not be published.

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.