Petter Reinholdtsen

At work we have heaps of servers. I believe the total count is around 1000 at the moment. To be able to get help from the vendors when something go wrong, we want to keep the firmware on the servers up to date. If the firmware isn't the latest and greatest, the vendors typically refuse to start debugging any problems until the firmware is upgraded. So before every reboot, we want to upgrade the firmware, and we would really like everyone handling servers at the university to do this themselves when they plan to reboot a machine. For that to happen we at the unix server admin group need to provide the tools to do so.

To make firmware upgrading easier, I am working on a script to fetch and install the latest firmware for the servers we got. Most of our hardware are from Dell and HP, so I have focused on these servers so far. This blog post is about the Dell part.

On the Dell FTP site I was lucky enough to find an XML file with firmware information for all 11th generation servers, listing which firmware should be used on a given model and where on the FTP site I can find it. Using a simple perl XML parser I can then download the shell scripts Dell provides to do firmware upgrades from within Linux and reboot when all the firmware is primed and ready to be activated on the first reboot.

This is the Dell related fragment of the perl code I am working on. Are there anyone working on similar tools for firmware upgrading all servers at a site? Please get in touch and lets share resources.

#!/usr/bin/perl
use strict;
use warnings;
use File::Temp qw(tempdir);
BEGIN {
    # Install needed RHEL packages if missing
    my %rhelmodules = (
        'XML::Simple' => 'perl-XML-Simple',
        );
    for my $module (keys %rhelmodules) {
        eval "use $module;";
        if ($@) {
            my $pkg = $rhelmodules{$module};
            system("yum install -y $pkg");
            eval "use $module;";
        }
    }
}
my $errorsto = 'pere@hungry.com';

upgrade_dell();

exit 0;

sub run_firmware_script {
    my ($opts, $script) = @_;
    unless ($script) {
        print STDERR "fail: missing script name\n";
        exit 1
    }
    print STDERR "Running $script\n\n";

    if (0 == system("sh $script $opts")) { # FIXME correct exit code handling
        print STDERR "success: firmware script ran succcessfully\n";
    } else {
        print STDERR "fail: firmware script returned error\n";
    }
}

sub run_firmware_scripts {
    my ($opts, @dirs) = @_;
    # Run firmware packages
    for my $dir (@dirs) {
        print STDERR "info: Running scripts in $dir\n";
        opendir(my $dh, $dir) or die "Unable to open directory $dir: $!";
        while (my $s = readdir $dh) {
            next if $s =~ m/^\.\.?/;
            run_firmware_script($opts, "$dir/$s");
        }
        closedir $dh;
    }
}

sub download {
    my $url = shift;
    print STDERR "info: Downloading $url\n";
    system("wget --quiet \"$url\"");
}

sub upgrade_dell {
    my @dirs;
    my $product = `dmidecode -s system-product-name`;
    chomp $product;

    if ($product =~ m/PowerEdge/) {

        # on RHEL, these pacakges are needed by the firwmare upgrade scripts
        system('yum install -y compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686 procmail');

        my $tmpdir = tempdir(
            CLEANUP => 1
            );
        chdir($tmpdir);
        fetch_dell_fw('catalog/Catalog.xml.gz');
        system('gunzip Catalog.xml.gz');
        my @paths = fetch_dell_fw_list('Catalog.xml');
        # -q is quiet, disabling interactivity and reducing console output
        my $fwopts = "-q";
        if (@paths) {
            for my $url (@paths) {
                fetch_dell_fw($url);
            }
            run_firmware_scripts($fwopts, $tmpdir);
        } else {
            print STDERR "error: Unsupported Dell model '$product'.\n";
            print STDERR "error: Please report to $errorsto.\n";
        }
        chdir('/');
    } else {
        print STDERR "error: Unsupported Dell model '$product'.\n";
        print STDERR "error: Please report to $errorsto.\n";
    }
}

sub fetch_dell_fw {
    my $path = shift;
    my $url = "ftp://ftp.us.dell.com/$path";
    download($url);
}

# Using ftp://ftp.us.dell.com/catalog/Catalog.xml.gz, figure out which
# firmware packages to download from Dell.  Only work for Linux
# machines and 11th generation Dell servers.
sub fetch_dell_fw_list {
    my $filename = shift;

    my $product = `dmidecode -s system-product-name`;
    chomp $product;
    my ($mybrand, $mymodel) = split(/\s+/, $product);

    print STDERR "Finding firmware bundles for $mybrand $mymodel\n";

    my $xml = XMLin($filename);
    my @paths;
    for my $bundle (@{$xml->{SoftwareBundle}}) {
        my $brand = $bundle->{TargetSystems}->{Brand}->{Display}->{content};
        my $model = $bundle->{TargetSystems}->{Brand}->{Model}->{Display}->{content};
        my $oscode;
        if ("ARRAY" eq ref $bundle->{TargetOSes}->{OperatingSystem}) {
            $oscode = $bundle->{TargetOSes}->{OperatingSystem}[0]->{osCode};
        } else {
            $oscode = $bundle->{TargetOSes}->{OperatingSystem}->{osCode};
        }
        if ($mybrand eq $brand && $mymodel eq $model && "LIN" eq $oscode)
        {
            @paths = map { $_->{path} } @{$bundle->{Contents}->{Package}};
        }
    }
    for my $component (@{$xml->{SoftwareComponent}}) {
        my $componenttype = $component->{ComponentType}->{value};

        # Drop application packages, only firmware and BIOS
        next if 'APAC' eq $componenttype;

        my $cpath = $component->{path};
        for my $path (@paths) {
            if ($cpath =~ m%/$path$%) {
                push(@paths, $cpath);
            }
        }
    }
    return @paths;
}

The code is only tested on RedHat Enterprise Linux, but I suspect it could work on other platforms with some tweaking. Anyone know a index like Catalog.xml is available from HP for HP servers? At the moment I maintain a similar list manually and it is quickly getting outdated.