============ UPDATE==============
I just want to add that this behaviour is fixed on IPMI firmware level.
There is an check-box added in the Configuration -> LAN Select, where one can choose whether to use the dedicated IPMI LAN or On-Board LAN. However still a little crippled menu, because you can not choose which Onboard port to use.
============ UPDATE==============
These days I got my hands on a bunch of supermicro servers with Tylersburg chipset ( -f types of MB with embedded IPMI) and 55xx series processors.
For the MBs from supermicro designed for 54xx and below CPUs, I got used with the IPMIs with dedicated LAN port which were working as expected. The IPMI is going out via the dedicated LAN port no matter what.
I wanted to set up the servers ready for delivery in the data center. All I wanted to do is to set up a proper IPs on the IPMI and make sure I can see it. Everything else can be done remotely.
The servers arrive with a sticker on them which shows the current IP set for IPMI. What I do is to plug the servers in the power sockets, without powering them on, then connect to IPMI, change the IP address and GW, then save and that is all. No noise in the office no head ace.
With all servers with chipset different from Tylersburg so far, no problem.
What was my astonishment when I realized that I can not see the IPMI. My first assumption was that the IP on the sticker was wrong. Ok not a big deal, just boot into BIOS and change it from there. I did boot but all I saw was that the IP address is set as on the sticker and I still did not have any ping to the IPMI.
I tried to set it to something else, still the same result unable to even get an ARP response from the IPMI card. These kind of problems really drives me mad.
I got on the phone my HW vendor to ask for the weird behaviour. After a few minutes conversation, I started doing something which was against any logic. I started to unplug the network cable and try the other LAN port on the MB which were supposed to be dedicated LAN port for the MB, and guess what, IPMI was located on the first MB port instead of the dedicated IPMI port.
From this point on, in order to make the stupid IPMI to move to the dedicated port I had to do a cold reboot with a network cable attached to the IPMI port.
If you plug the server in the power socket without network cables connected, the IPMI is so "smart" that it goes on the wrong port again. It does not start on the dedicated IPMI LAN port but on the first port of the MB. This is completely stupid logic for me. Why should the stupid IPMI play smart. This is a server after all. You can not play with the remote management because you think you are smart. I want to be able to pin down the IPMI LAN port, to something unchangeable. Everything else is incredibly stupid, although it may look cool to someone.
Why do I hate this. I find this behaviour unacceptable because for example you have lots of servers. All of them have IPMI cards. All the IPMI cards are connected in a private LAN which is visible only via VPN for example.
Now for some reason you want cold reboot of some server. You call the datacenter support and request a cold reboot.
All is fine so far, but imagine that during the cold reboot the support guy by mistake unplug the IPMI of the server, or the RJ45 connector is not well connected or something like this. After the cold reboot the "smart" IPMI will be pinned down to the MB LAN port and depending on the IP address set it might become visible to the public, which is really, really not a very good idea.
I thought that this is not normal behaviour, and started to look for a way to disable it. However I found a raised middle finger in the docs:
Notes:
1. If you wish to use the IPMI-dedicated LAN port for your network connections, be sure to connect an RJ45 cable to your dedicated LAN port before you activate the BMC (at first power-on or cold reset). Otherwise, the BMC will look for a shared LAN port to connect to if the IPMI-dedicated LAN cable is not detected upon BMC activation.
2. However, should you decide to use the IPMI-dedicated LAN port for a network connection, please perform a BMC cold reset or power cycle reset in order for the dedicated LAN to be detected.
And the worst thing, you can not disable this stupidity so far.
Supermicro are saying that they are working on a resolution of this issue. I hope to fix this soon and to be able to choose from the BIOS on which port I want the IPMI, not someone else to choose for me.
I just want to add that this behaviour is fixed on IPMI firmware level.
There is an check-box added in the Configuration -> LAN Select, where one can choose whether to use the dedicated IPMI LAN or On-Board LAN. However still a little crippled menu, because you can not choose which Onboard port to use.
============ UPDATE==============
These days I got my hands on a bunch of supermicro servers with Tylersburg chipset ( -f types of MB with embedded IPMI) and 55xx series processors.
For the MBs from supermicro designed for 54xx and below CPUs, I got used with the IPMIs with dedicated LAN port which were working as expected. The IPMI is going out via the dedicated LAN port no matter what.
I wanted to set up the servers ready for delivery in the data center. All I wanted to do is to set up a proper IPs on the IPMI and make sure I can see it. Everything else can be done remotely.
The servers arrive with a sticker on them which shows the current IP set for IPMI. What I do is to plug the servers in the power sockets, without powering them on, then connect to IPMI, change the IP address and GW, then save and that is all. No noise in the office no head ace.
With all servers with chipset different from Tylersburg so far, no problem.
What was my astonishment when I realized that I can not see the IPMI. My first assumption was that the IP on the sticker was wrong. Ok not a big deal, just boot into BIOS and change it from there. I did boot but all I saw was that the IP address is set as on the sticker and I still did not have any ping to the IPMI.
I tried to set it to something else, still the same result unable to even get an ARP response from the IPMI card. These kind of problems really drives me mad.
I got on the phone my HW vendor to ask for the weird behaviour. After a few minutes conversation, I started doing something which was against any logic. I started to unplug the network cable and try the other LAN port on the MB which were supposed to be dedicated LAN port for the MB, and guess what, IPMI was located on the first MB port instead of the dedicated IPMI port.
From this point on, in order to make the stupid IPMI to move to the dedicated port I had to do a cold reboot with a network cable attached to the IPMI port.
If you plug the server in the power socket without network cables connected, the IPMI is so "smart" that it goes on the wrong port again. It does not start on the dedicated IPMI LAN port but on the first port of the MB. This is completely stupid logic for me. Why should the stupid IPMI play smart. This is a server after all. You can not play with the remote management because you think you are smart. I want to be able to pin down the IPMI LAN port, to something unchangeable. Everything else is incredibly stupid, although it may look cool to someone.
Why do I hate this. I find this behaviour unacceptable because for example you have lots of servers. All of them have IPMI cards. All the IPMI cards are connected in a private LAN which is visible only via VPN for example.
Now for some reason you want cold reboot of some server. You call the datacenter support and request a cold reboot.
All is fine so far, but imagine that during the cold reboot the support guy by mistake unplug the IPMI of the server, or the RJ45 connector is not well connected or something like this. After the cold reboot the "smart" IPMI will be pinned down to the MB LAN port and depending on the IP address set it might become visible to the public, which is really, really not a very good idea.
I thought that this is not normal behaviour, and started to look for a way to disable it. However I found a raised middle finger in the docs:
Notes:
1. If you wish to use the IPMI-dedicated LAN port for your network connections, be sure to connect an RJ45 cable to your dedicated LAN port before you activate the BMC (at first power-on or cold reset). Otherwise, the BMC will look for a shared LAN port to connect to if the IPMI-dedicated LAN cable is not detected upon BMC activation.
2. However, should you decide to use the IPMI-dedicated LAN port for a network connection, please perform a BMC cold reset or power cycle reset in order for the dedicated LAN to be detected.
And the worst thing, you can not disable this stupidity so far.
Supermicro are saying that they are working on a resolution of this issue. I hope to fix this soon and to be able to choose from the BIOS on which port I want the IPMI, not someone else to choose for me.
Thanks for this. We were just beating our heads against the wall trying to work out why some SuperMicro boards were looking for the DHCP server for their IPMI address on the LAN port. Nothing at all in the BIOS.
ReplyDeleteBut sure enough, the power cables had been put in before the IPMI cables. Cold reboot did the trick. Nothing in the BIOS, and judging from the date of this post, if SuperMicro are working on a resolution of this issue they're not working very fast!
Thanks again!
had the same problem, found setting in web interface. Silly "feature" indeed... Should be disabled by default, at least.
ReplyDeleteI had the same problem too, weird it is not documented yet in supermicro manuals...
ReplyDelete