Linux Installation on Intel® iSCSI Remote Boot

This page describes how to install Linux onto a hard drive so that the disk is bootable using an Intel iSCSI Remote Boot-capable adapter. There are two methods to accomplish this:

For more information on SCSI device enumeration, refer to Appendix A at the end of this document.

The SUSE* Linux Enterprise Server 10 process can be found here.

The SUSE* Linux Enterprise Server 9-SP3 process can be found here.

The Red Hat* Enterprise Linux 5.1 process can be found here.

The Red Hat* Enterprise Linux 4 process can be found here.

NOTE: Some newer network device drivers are not included in either SLES9 or RHEL4. To enable these devices, the user will need to get the correct driver and load the driver before discovery is attempted when following the preinstall instructions.

SUSE* Linux Enterprise Server 10

NOTE: Users who wish to have the easiest experience when installing Linux onto an iSCSI target should use SLES10 or greater. SLES10 provides native support for iSCSI Remote Booting and installing. This means that there are no additional steps outside of the installer that are necessary to install to an iSCSI target using an Intel Gigabit PCI-Express Server Adapter. Please refer to the SLES10 documentation for instructions on how to install to an iSCSI LU.

SUSE* Linux Enterprise Server 9-SP3

Prerequisites:

  1. A host system to install Linux on.
  2. A driver disk compatible with the host system and the diskless target (either USB key or floppy disk).
  3. The SUSE Linux Enterprise Server 9-SP3 install CDs.
  4. linux-iscsi files (this requirement will be satisfied by Section I)
    1. iscsi.ko
    2. iscsid
    3. initiatorname.iscsi
    4. iscsi.conf

Section I : Creating the Linux-iSCSI driver

This is the recommended method for creating the driver disk.

  1. On the system running SUSE Linux Enterprise Server 9-SP3, insert and mount the media to be used as the driver disk.
  2. YaST -> Add/Remove Software.
  3. Search "kernel"
  4. Select "kernel-default" and install the package.
  5. Search "iscsi"
  6. Select "linux-iscsi" and install the package.
  7. Copy the following files to your mounted media
    1. /lib/modules/2.6.5-7.244-default/extra/iscsi.ko
    2. /sbin/iscsid
    3. /etc/initiatorname.iscsi
    4. /etc/iscsi.conf
NOTES: Configure the initiatorname.iscsi and iscsi.conf files according to your target configuration. To do so you should read the README provided by the linux-iscsi source code. You must at least configure the DiscoveryAddress in the iscsi.conf.

If your target supports advanced features such as CHAP, you will need to make sure that the iscsi.conf and initiatorname.iscsi match your target configuration at this point. These files will be used throughout the install and boot process.

Alternatively, you may create the driver disk using a different host machine other than SUSE Linux Enterprise Server 9-SP3. You will need to compile the iscsi.ko and iscsid using the 2.6.5-7.244-default kernel. You can learn to do this by reading the README provided with the linux-iscsi source code.

Section II : Installing to the iSCSI target system

NOTE: You need an Intel® iSCSI Remote Boot capable adapter to connect to your target and boot the OS from the iSCSI Target Disk.

Requirements:

  1. Diskless host compatible with the driver disk.
  2. The driver disk created in Section I.
  3. The full set of CDs for both SUSE Linux Enterprise Server 9 and SUSE Linux Enterprise Server 9-SP3.

Procedure:

  1. Boot from SUSE Linux Enterprise Server 9-SP3 installation CD1.
    1. Select "installation", but before pressing "enter", please enter the following boot option.
          DiscoveryAddress=<IP of target> InitiatorName=<initiator iqn> netsetup=1


  2. When you are prompted for CD1 insert SUSE Linux Enterprise Server 9 CD1.
  3. When you are prompted to accept the license agreement, press Ctrl-Alt-F2 to switch to the console
    1. mount the driver disk
    2. copy the following files to /etc
      1. initiatorname.iscsi
      2. iscsi.conf
    3. copy the following files to /tmp
      1. iscsi.ko
      2. iscsid
    4. unmount your driver disk and remove the driver disk
    5. cd /tmp
    6. insmod iscsi.ko
    7. ./ iscsid
  4. Verify that you are connected to the target

    Ctrl-Alt-F4 will display something such as:

"iSCSI target bus 0 target 0 = iqn.XYZ"

"Attached scsi disk ..."

  1. Press Alt-F7 or Ctrl+Alt+F7 to go back and continue installation. Accept the license agreement and continue until you reach the "Installation Settings" Dialog.
NOTE: When partitioning the iSCSI LUs, ensure that your boot partition is not distributed across physical partitions on different iSCSI LUs. The Intel iSCSI Option ROM will only expose one iSCSI LU and if /boot is distributed across multiple LUs it will not be visible during the boot process.
  1. Select the Software packages to install
    1. Click "Software".
    2. Choose the software you wish to install.
    3. Select "Detailed selection". Change the value of the "Filter" drop-down box to "search".
    4. Search for "iSCSI".
    5. Make sure the "linux-iscsi" checkbox is selected.
    6. Make sure the "Autocheck" checkbox is selected.
    7. Click "accept" and "continue" if prompted.
  2. Change some of the Booting options
    1. Click "Booting".
    2. Double-click "Disk Order".
    3. Ensure that the disk that you are installing the OS on is the top disk in the list.
  3. When the installation completes, the user is warned that a reboot will occur, accompanied by a 10 second countdown. Press "Stop" to stop the system reboot.
  4. Press Ctrl-Alt-F2 to go to console
    1. Run "chroot /mnt"
    2. Edit line 10 of the file /etc/sysconfig/kernel. Append "e1000" & "iscsi" driver to INITRD_MODULES variable.

      Example: INITRD_MODULES="... e1000 iscsi".

    3. If your target requires CHAP authentication add the follow to /sbin/mkinitrd at line 1901

      echo Username=<CHAP username> >> /etc/iscsi.conf

      echo Password=<CHAP password> >> /etc/iscsi.conf

    4. mkinitrd -D ethX
    5. Press Ctrl-D
  5. Modify three configuration files so that they have the correct device node listed. Refer to Appendix A for more information on SCSI device enumeration.
    1. Edit /mnt/boot/grub/menu.1st so that root= points to the correct root partition.

               For example:

                   root=/dev/sda3

    1. Edit /mnt/boot/grub/device.map so that grub knows which hard drive it should use.

                For example:

                    (hd0) <SCSI device node for Boot LUN>

    1. Edit /mnt/etc/fstab so that the partitions of your iSCSI LU are mounted to the correct mount points. Only change the device nodes, not the FS type, attributes or major/minor numbers. For example,

                /dev/sda2     /         reiserfs     ac1,user_xattr     1     1

                /dev/sda1     /boot  ext2         ac1,user_xattr     1     2

  6. Press Alt-F7 (Alt-F1 if using text mode installation) to return to installation screen.
    1. Select "OK" to reboot.
    2. Remove the install CD.
  7. After the reboot, YaST will step you through various configurations.
NOTE: After this first reboot, YaST may ask you to configure your network devices. Skip this step as it will reset the network interface that the iSCSI traffic is on.
  1. When you have booted to your iSCSI LU, edit /etc/rc.d/network. Add the following line to the script to prevent it from trying to reset the iscsi-ethernet interface. Please add carefully

            test "$a" = ethX && continue;

    immediately after line number 169 where ethX is the ethernet interface used for iSCSI communication.

NOTE: After the system has booted, iscsid will be running. The user may want to add something like "kill `pidof iscsid`" to an init script. iscsid is only needed for discovery and can be killed in this way without problems.

See Appendix A

Red Hat Enterprise Linux 5.1 Installation:

NOTES: Users who wish to have the easiest experience when installing Linux onto an iSCSI target should use Red Hat 5.1 or greater. Red Hat 5.1 provides native support for iSCSI Remote Booting and installing. This means that there are no additional steps outside of the installer that are necessary to install to an iSCSI target using an Intel Gigabit PCI-Express Server Adapter. Please refer to the Red Hat 5.1 documentation for instructions on how to install to an iSCSI LU.

RHEL5.2 has iBFT support.

Red Hat Enterprise Linux 4 Installation:

This process has two phases.

  1. Creating the Driver Update Media (DUM). This involves copying the iSCSI module, its dependencies and the utilities needed for installation onto an update disk. The disk can be an IDE floppy, USB floppy or USB flash drive.
  2. Install to the iSCSI target LU. This consists of two parts; pre-install and post-install.

Prerequisites

  1. A Red Hat Enterprise Linux 4-U3 or U4 build machine. To ensure module compatibility the architecture of the build machine should match both the install media and the diskless system with which you intend to connect to the iSCSI target LU. This means that if you will be doing a diskless boot on an x86_86 system your build machine will need be have an x86_64 architecture. The install CDs will also need to be for the x86_64 architecture.
  2. iSCSIBoot.tar.gz. Provided by Intel.
  3. Install Media. A blank 3.5' floppy disk, or a USB flash drive.
  4. The iSCSI Target LUN (Boot LUN). This information is important to know in advance so that grub can be configured correctly.

Phase I - Creating Driver and Utilities Media (DUM)

  1. Copy and untar iSCSIBoot.tar.gz on the build machine.
    1. cp iSCSIBoot.tar.gz /home
    2. cd /home
    3. tar -zxvf iSCSIBoot.tar.gz
  2. Compile iscsiboot/iscsi_boot_util.c statically.
    1. cd iSCSIBoot
    2. gcc -static -Wall iscsi_boot_util.c -o iscsi_boot_util

  3. Install the following RPM and copy modules to preinstall directory.
    1. rpm -ivh kernel-2.6.9-34.EL.i686.rpm, from the RHEL4-U3 or U4 CD2
    2. cp -f \

              /lib/modules/2.6.9-34.EL/kernel/drivers/scsi/scsi_transport_iscsi.ko \

              /lib/modules/2.6.9-34.EL/kernel/drivers/scsi/iscsi_sfnet/iscsi_sfnet.ko \

              /lib/modules/2.6.9-34.EL/kernel/crypto/crc32c.ko \

              /lib/modules/2.6.9-34.EL/kernel/crypto/md5.ko \

              /lib/modules/2.6.9-34.EL/kernel/lib/libcrc32c.ko \

              /home/iSCSIBoot/preinstall/

  1. Copy media onto the DUM
    1. Attach the DUM and mount it if necessary.
    2. cp -fr /home/iSCSIBoot/iscsi_boot_util /home/iSCSIBoot/preinstall \

          /home/iSCSIBoot/postinstall <DUM mount point>

  1. Unmount if necessary and remove the DUM

Phase II - Installation to iSCSI disk

NOTES: This assumes you have flashed and configured your iSCSI firmware to connect to your target and iscsi firmware was able to connect to the iscsi target.

If you are using USB flash drive/USB floppy as DUM, it must be attached prior to this step. If you're performing a Local Install, make sure the local disk is connected.

  1. Begin installation with the option "linux updates".
  2. Continue until the "Update Disk Source" screen.
    1. Press Ctrl+Alt+f2 to switch to command line mode.
    2. Make a mount point directory and mount the DUM. (The mount point can be anything but /mnt). Depending on the DUM media type, these are the instructions:

IDE Floppy

        mkdir /mnt/floppy /tmp/iscsi

        mount /dev/fd0 /mnt/floppy

        cp -fr /mnt/floppy/* /tmp/iscsi

        umount /dev/fd0

USB Floppy Drive

To mount a USB device you must understand which /dev node the kernel will use to represent the DUM. To do this run "dmesg" and examine the output. Look for the text which shows your USB device's registration with the system. Determine which sdX device is associated with your device.

Next run "cat /proc/partitions", the output should look similar to this,

Major     Minor     #blocks     Name

7             0             <x>         loop0

8             0             <x>         sda

Using the /dev node determine the minor number for your USB device.

Mount the USB device.

mkdir /mnt/usb /tmp/iscsi

mknod /dev/sdX b 8 <minor#>

mount /dev/sdX /mnt/usb

cp -fr /mnt/usb/* /tmp/iscsi

DO NOT unmout/remove USB floppy/USB flash drive at this point.

  1. Run the "preinstall/iscsipreinstall" script from the mount point.

/tmp/iscsi/preinstall/iscsipreinstall

  1. Run "dmesg" again to verify that discovery occurred correctly. It should show that drives were recently discovered. Note the LUNs and discovered SCSI device nodes. This information needed to set grub configuration files.
  2. Press Ctrl+Alt+F1 and select "Back".
  3. Continue with the installation.
NOTE: When partitioning the iSCSI LUs, ensure that your boot partition is not distributed across physical partitions on different iSCSI LUs. The Intel iSCSI Option ROM will only expose one iSCSI LU and if /boot is distributed across multiple LUs it will not be visible during the boot process.
  1. Configure the boot order
    1. On the "Boot Loader Configuration" select "Configure advanced boot loader options" and click Next.
    2. Change the drive order so that your target disk is on the top of the list.
    3. Continue with remainder of installation.
    4. When the install is complete DO NOT press "Reboot".
  1. Make some final post install changes:
    1. Press Ctrl-Alt-F2 to switch back to the console.
    2. Alter the grub settings:

      Modify the device.map and grub.conf files so that they have the correct device node listed. Refer to Appendix A for more information on SCSI device enumeration.

          /mnt/sysimage/boot/grub/device.map

              (hd0) <SCSI device node for Boot LUN>

          /mnt/sysimage/boot/grub/grub.conf

              #boot=<SCSI device node for Boot LUN>

    3. Run the post install script and unmount the DUM.

               /tmp/iscsi/postinstall/iscsipostinstall

  2. Ensure the network is not shutdown during halt
    1. Edit the /mnt/sysimage/etc/init.d/halt and scroll to the bottom.
    2. Change the line stating HALTARGS="-i -d" to HALTARGS="-d"
  3. Return to the GUI and reboot
    NOTE: If you are using a bootable USB or Flash drive, remove it, otherwise the system will try to boot off that drive.
    1. Press Ctrl+Alt+F7 and then select "Reboot".
    2. Remove the installation media and DUM.
  4. On your first boot, if Kudzu asks you if you'd like to configure your adapters, please select to "ignore" the device.

    Appendix A

    The iSCSI LUs are discovered as SCSI device nodes. The enumeration of SCSI devices is dynamic. The SCSI subsystem creates device nodes in the order that they are discovered, starting with /dev/sda and incrementing alphabetically. In the case of iSCSI LU enumeration, local SCSI devices such as disks or USB devices affect this process. If a device node is created for a USB device before iSCSI discovery occurs the USB device will be represented by /dev/sda and the iSCSI LUs will begin with /dev/sdb. However, if you were to reboot this system without the USB device the iSCSI LUs will begin with /dev/sda. It is important to configure grub correctly to point to the correct device node in order to boot from an iSCSI hard drive.

    It is often the case when installing to an iSCSI LU that a device driven by the SCSI subsystem is used for the install media. For example, using a USB flash drive for the DUM. As stated above this will affect how grub should be configured when installing.

    Here is an example of how iSCSI device node mappings might differ between installation and booting. In this example the user is using a USB flash drive as a DUM. Since this device is being used to copy files onto the system it is present before iSCSI discovery occurs. However, when booting from the iSCSI Remote Boot Disk no USB flash drive is necessary and therefore it isn't attached.

    Remote Installation:

    USB flash drive or USB floppy                           /dev/sda

    LUN 0                                                               /dev/sdb

    LUN 1                                                               /dev/sdc

    Booting:

    LUN 0                                                               /dev/sda

    LUN 1                                                               /dev/sdb

    If the USB device is attached after booting, it will be enumerated as such,

    USB flash drive or USB floppy                          /dev/sdc

    It is also possible that there are other USB devices on the system, it is the users responsibility to find out if these will interfere with the device node mapping during boot.

    Appendix B

    After performing a Local Install on an IDE device the user will need to modify grub so that when booting the root partition is identified as a SCSI device. When installing to a local IDE disk drive the device node used will be of the form /dev/hdX, but when this volume is exposed by the iSCSI target the initiator will see the hard drive as a SCSI device and it will be represented as /dev/sdX. This change must be made in the grub configuration file. This type of problem exposes the necessity that the user fully understand the concepts described in this document as specifics may vary greatly.

Appendix C

In some cases, if booting the system takes an unusually long time, you may try commenting the following line in the file /etc/init.d/iptables:

$IPTABLES-restore $OPT $IPTABLES_DATA

if [ $? -eq 0 ]; then

    success; echo

else

    failure; echo; return 1

fi

This may be required as the command may be resetting the connection.

Appendix D

Crash Dump for Linux:

Red Hat Enterprise Linux 4, 5.x: Netdump utility can be used to dump the disk image when the system crashes.

SUSE Linux Enterprise Server: Netdump utility can be used to dump the disk image when the system crashes.