Lenovo G8272 and EN4093R Invalid Signature Firmware Upgrade Problem

While trying to upgrade the firmware of brand new Lenovo G8272 switches from the initial release of 8.2.1.0, I got an error after uploading the new firmware:

Failure: image contains invalid signature.
G8272(config)#
Feb  9 18:58:41 G8272 ERROR   mgmt: Firmware download failed to image1

I only got 2 results online and both pointed at Changelogs that mention the issue has been fixed, but not how! I contacted a great person within Lenovo who checked internal documents and it turned out that this issue affects G8272 and EN4093R switches manufactured on December 2015 (specifically, 12th week of 2015). (Thank you Zeeshan!)

Cause

"The switch software uses it hardware serial number and the public keys on its kernel file system to generate a private key to decrypt the OS or Boot image being uploaded to it and then proceeds to install it. If the serial number of the switch is changed for some reason, the combination of the hardware serial number and the public keys will fail to generate the appropriate private key to decrypt the uploaded image and reports that the image has an invalid signature."

In my case, the switches were fresh & no one changed any serial code, but were still affected.

Fix

"In order to remedy this situation, the way out is to remove the public keys installed on the kernel file system and reboot the switch. During reboot, the switch will generate new set of public keys using the current serial number. With these newly generated public keys, the switch will be able to compute the proper private key to decrypt the uploaded images."

Requirements

• Serial cable (mini-USB that came with the switch)
• Serial-to-USB kit (you have to buy this on your own)
• CAT5E or CAT6 STP or UTP cable
• New firmware (8.2.4.0 as of this writing)
• PuTTY or your favorite serial/telnet/ssh tool
• admin password (default is admin:admin)
• ftp/tftp server software. I suggest 3CDaemon (FTP & TFTP) or Filezilla (FTP & SFTP).

On a Flex chassis, you should enable Serial Over LAN (SOL) from the Chassis Management Module (CMM) to be able to access the serial port of the switches. Use UTP cable on the CMM port not the switch.

I highly recommend configuring the management port (RJ45) to use for firmware upload since it'll be very fast, as it'll take 45 minutes to upload one OS image! While it takes 1 minute on the management port via Ethernet.

Note: The initial firmware (8.2.1.0 does not support SSH). However, SSH is enabled by default once you upgrade to 8.2.4.0. Make sure you disable HTTP & Telnet after the upgrade.

Procedure

Any line that starts with # it means this is a command to be typed (without the # sign).

1. Connect to serial port on the switch (mini-USB port)
2. Login as admin user
3. Reboot the switch: #reload
4. When the switch shows Memory Test, press Shift+t to enter Manufacturer Mode.
   U-Boot 2009.06 (Feb 23 2015 - 07:27:18)
   
   CPU0:  P2020, Version: 2.1, (0x80e20021)
   Core:  E500, Version: 5.1, (0x80211051)
   Clock Configuration:
          CPU0:1200 MHz, CPU1:1200 MHz,
          CCB:600  MHz,
          DDR:400  MHz (800 MT/s data rate) (Asynchronous), LBC:37.500 MHz
   L1:    D-cache 32 kB enabled
          I-cache 32 kB enabled
   Board: Networking OS RackSwitch G8272
   I2C:   ready
   DRAM:   DDR:  4 GB
   
   Memory Test ..........
   
   Manufacturing Mode
   FLASH: 16 MB
   L2:    512 KB enabled
   PCIe1: Root Complex of PCIe, x2, regs @ 0xffe0a000
   PCIe1: Bus 00 - 01
   MMC:  FSL_ESDHC: 0
   Note : Operational Mode has changed.
   Net:   eTSEC1, eTSEC2 [PRIME]
   
   Booting OS
5. Once the OS boots, enter the admin password (default is admin)
6. You should now be at the prompt where it says: Diagnostics#
7. Enter diagnostics mode: #linux
8. List the filesystem to see if there are existing public encryption keys: #ls /user/*.pem
   > ls /user/*.pem
   /user/development_key.pub.pem  /user/production_key.pub.pem
9. The two files above should show. Delete them: #rm /user/*.pem
10. That's it. Now quit by typing q in the command: #q
11. Now reboot: #/boot/reset
12. Press "y" to confirm rebooting. The switch will now reboot and generate new keys to match the current hardware serials and whatnot.
13. Now connect via Ethernet (or configure an IP interface on the management port then connect) and upgrade the switch
14. #copy tftp image1 address 192.168.70.13 filename G8272-8.2.4.0_OS.man mgt-port
    Change tftp to match what protocol you're using.
    Change 192.168.70.13 to match your machine's IP where the TFTP/FTP server is running.
    Change G8272_8.2.4.0_OS.man to match the file name.
15. You'll be asked if you want to make image1 the default boot image; press y.
16. Repeat the same step above for the 2nd image: image2. Do NOT select it as the default image.
17. Now upload the Boot image:
    #copy tftp boot address 192.168.70.13 filename G8272-8.2.4.0_Boot.man mgt-port
18. We're done. If you have any config unsaved, type: #write
19. Now that you're done, reboot the switch: #reload

Congratulations.

Tip: You may want to change the switches' timezone, date & time (in that exact order). The defaults dated to Feb 2015.