Showing posts with label lenovo. Show all posts
Showing posts with label lenovo. Show all posts

Wednesday, January 6, 2016

Lenovo G8272 and EN4093R Invalid Signature Firmware Upgrade Problem

While trying to upgrade the firmware of brand new Lenovo G8272 switches from the initial release of 8.2.1.0, I got an error after uploading the new firmware:
Failure: image contains invalid signature.
G8272(config)#
Feb  9 18:58:41 G8272 ERROR   mgmt: Firmware download failed to image1

I only got 2 results online and both pointed at Changelogs that mention the issue has been fixed, but not how! I contacted a great person within Lenovo who checked internal documents and it turned out that this issue affects G8272 and EN4093R switches manufactured on December 2015 (specifically, 12th week of 2015). (Thank you Zeeshan!)

Cause

"The switch software uses it hardware serial number and the public keys on its kernel file system to generate a private key to decrypt the OS or Boot image being uploaded to it and then proceeds to install it. If the serial number of the switch is changed for some reason, the combination of the hardware serial number and the public keys will fail to generate the appropriate private key to decrypt the uploaded image and reports that the image has an invalid signature."

In my case, the switches were fresh & no one changed any serial code, but were still affected.

Fix

"In order to remedy this situation, the way out is to remove the public keys installed on the kernel file system and reboot the switch. During reboot, the switch will generate new set of public keys using the current serial number. With these newly generated public keys, the switch will be able to compute the proper private key to decrypt the uploaded images."

Requirements

  • Serial cable (mini-USB that came with the switch)
  • Serial-to-USB kit (you have to buy this on your own)
  • CAT5E or CAT6 STP or UTP cable
  • New firmware (8.2.4.0 as of this writing)
  • PuTTY or your favorite serial/telnet/ssh tool
  • admin password (default is admin:admin)
  • ftp/tftp server software. I suggest 3CDaemon (FTP & TFTP) or Filezilla (FTP & SFTP).

On a Flex chassis, you should enable Serial Over LAN (SOL) from the Chassis Management Module (CMM) to be able to access the serial port of the switches. Use UTP cable on the CMM port not the switch.

I highly recommend configuring the management port (RJ45) to use for firmware upload since it'll be very fast, as it'll take 45 minutes to upload one OS image! While it takes 1 minute on the management port via Ethernet.

Note: The initial firmware (8.2.1.0 does not support SSH). However, SSH is enabled by default once you upgrade to 8.2.4.0. Make sure you disable HTTP & Telnet after the upgrade.

Procedure

Any line that starts with # it means this is a command to be typed (without the # sign).
  1. Connect to serial port on the switch (mini-USB port)
  2. Login as admin user
  3. Reboot the switch: #reload
  4. When the switch shows Memory Test, press Shift+t to enter Manufacturer Mode.
    U-Boot 2009.06 (Feb 23 2015 - 07:27:18)

    CPU0:  P2020, Version: 2.1, (0x80e20021)
    Core:  E500, Version: 5.1, (0x80211051)
    Clock Configuration:
           CPU0:1200 MHz, CPU1:1200 MHz,
           CCB:600  MHz,
           DDR:400  MHz (800 MT/s data rate) (Asynchronous), LBC:37.500 MHz
    L1:    D-cache 32 kB enabled
           I-cache 32 kB enabled
    Board: Networking OS RackSwitch G8272
    I2C:   ready
    DRAM:   DDR:  4 GB

    Memory Test ..........

    Manufacturing Mode

    FLASH: 16 MB
    L2:    512 KB enabled
    PCIe1: Root Complex of PCIe, x2, regs @ 0xffe0a000
    PCIe1: Bus 00 - 01
    MMC:  FSL_ESDHC: 0
    Note : Operational Mode has changed.
    Net:   eTSEC1, eTSEC2 [PRIME]

    Booting OS
  5. Once the OS boots, enter the admin password (default is admin)
  6. You should now be at the prompt where it says: Diagnostics#
  7. Enter diagnostics mode: #linux
  8. List the filesystem to see if there are existing public encryption keys: #ls /user/*.pem
    > ls /user/*.pem
    /user/development_key.pub.pem  /user/production_key.pub.pem
  9. The two files above should show. Delete them: #rm /user/*.pem
  10. That's it. Now quit by typing q in the command: #q
  11. Now reboot: #/boot/reset
  12. Press "y" to confirm rebooting. The switch will now reboot and generate new keys to match the current hardware serials and whatnot.
  13. Now connect via Ethernet (or configure an IP interface on the management port then connect) and upgrade the switch
  14. #copy tftp image1 address 192.168.70.13 filename G8272-8.2.4.0_OS.man mgt-port
    Change tftp to match what protocol you're using.
    Change 192.168.70.13 to match your machine's IP where the TFTP/FTP server is running.
    Change G8272_8.2.4.0_OS.man to match the file name.
  15. You'll be asked if you want to make image1 the default boot image; press y.
  16. Repeat the same step above for the 2nd image: image2. Do NOT select it as the default image.
  17. Now upload the Boot image:
    #copy tftp boot address 192.168.70.13 filename G8272-8.2.4.0_Boot.man mgt-port
  18. We're done. If you have any config unsaved, type: #write
  19. Now that you're done, reboot the switch: #reload

Congratulations.

Tip: You may want to change the switches' timezone, date & time (in that exact order). The defaults dated to Feb 2015.

IBM POWER8 Networking via Direct Attach Cables

I recently had a project where my company sold POWER8 servers to the customer along with some Lenovo servers and Lenovo G8272 network switches. The switches have 48x 1/10 Gb ports + 6x 40 Gb ports.

To save on cost, it's possible to use Direct Attach Cables (DACs) to connect servers to the switches without buying SFPs nor FC cables. List price comparison:
  • Lenovo 10GBASE-SR SFP+ Transceiver (46C3447) = $629
  • Lenovo 5m LC-LC OM3 MMF Cable (00MN508) = $58
  • To connect 1 server (4 ports) to switches (4 ports) = 8x $629 + 4x $58 = $5,264.
In contrast, with DACs, you only need 1 cable which includes the SFPs (copper):
  • Lenovo 5m Passive SFP+ DAC Cable (90Y9433) = $210
  • Lenovo 5m Active DAC SFP+ Cable (00VX117) =  $290
  • Active are often used for switch-to-switch connectivity.
  • To connect 1 server (4 ports) to switches (4 ports) = 4x  $210 = $840.
 DACs are 16% the cost! Or 6.3 times cheaper. These prices are based on publicly available list prices. They might be different depending on your region and distributor.

The POWER8 servers (S822) have the following Ethernet adapter: EN0U -- PCIe2 4-Port (10Gb+1GBE) Copper SFP+RJ45 Adapters. According to the redbook (guide), these adapters require Active Copper DACs.

I actually used the Passive DACs that I used for the Lenovo servers, and the cables worked just fine. The AIX team configured 2 Virtual Input/Output Servers (VIOS) on each POWER8 system, and each POWER8 system had 4 of these adapters. We also configured LACP for each VIOS, so the total bandwidth available to each VIOS was 40 Gb.

So even though the redbook says that Active DACs are required, the passive ones work just fine. Also the redbook only lists 1 meter, 3 meter & 5 meter cables (since they're active) and no mention of passive cables.