a couple of weeks ago i began experiencing a seemingly unending nightmare. it all began like so….
i was attempting to update the BIOS to the latest release on a dell poweredge R710. the server was running vsphere 4.x. this is a pretty routine thing to do, that i’ve done several times before with no problems. i use the dell update packages (DUPs) for red hat linux when i update firmware in ESX. sometimes i push out the updates with IT assistant, other times i do it manually. i was trying to update the BIOS manually on this occasion and the update simply refused to work. it basically just crapped out and exited. so i was like okay…maybe something is just wrong with this particular release so i downloaded the release prior and the same thing happened. also of note is that i just used the same exact DUP file to upgrade another R710 a week prior. at this point i was perplexed and did some searching around. i found some articles that gave a howto on deconstructing the DUP and then _really_ manually upgrading the BIOS. the instructions seemed very specific and i eventually decided this wasn’t a path i wanted to go down.

plan B involved using the lifecycle controller – unified server configurator (USC). which i had just found out about. so i got the server hooked back up to the internet and booted into USC then did the platform update using ftp.dell.com as a source. by default USC wants to update the firmware to everything. when i first saw this i was like wow that’s really cool…basically take care of everything all at once. even though i was originally interested in updating the BIOS only…i decided to pull the trigger and update everything.

and that’s when i was transported straight to hell. the firmware updates all succeeded except failure on 2 very expensive 10Gb NICs. the failure error was “nicwrapper.efi error” and the only dell docs i could find that referenced the error said to make sure you have the latest BIOS and iDRAC updates applied. well i had the latest iDRAC update and i was attempting to install the latest BIOS update. if the NIC firmware update couldn’t be applied it shouldn’t have been checkbox’d by default . 🙁 anyways, at this point vsphere would boot up and during driver loading the server would reboot. at the time, it wasn’t plainly obvious to me what the problem was. i figured the NIC update failure wouldn’t have caused any real damage. system logs were showing a PCI bus error and a CPU IERR. i was still thinking it was a problem with the BIOS… like maybe it didn’t apply correctly. of course, if that was the issue the system probably wouldn’t have been booting at all. it eventually became clear to me that the NICs were the problem when the system would boot fine with both NICs removed.

if i had been more familiar with USC at the time i probably would have tried an update rollback on the NICs and maybe would have saved myself all kinds of trouble. instead, i tried to apply the firmware update to both NICs again with the same unsuccessful result. so then i saw there was a DUP for a newer NIC firmware update and tried that which also ended in failure. then i wanted to try that newer update and run it from USC…well you can’t just do that with DUPs you have to download another utility called the dell repository manager. then with the repository manager export the update as SUU. i did all that and tried to apply the update with USC…still no luck.

this basically goes on forever…i tried with an dell server update utility (SUU) bootcd. i even put the NICs in a windows server and tried to apply the windows DUP and received a “bmapi error”. at this point i gave up…the NICs were obviously shot. they were in a state that was beyond my knowledge on how to repair.

what did i learn from all of this: don’t ever try to update the firmware on broadcom NICs. unless for some reason you really really have to.