Booting debian over iscsi

Here I will be documenting my first attempt to boot linux over iscsi.

1. Chose debian because the squashfs of debian works fine and there are no other problems.
2. Also, all the kernel modules needed for iscsi initiator are inside initramfs so, less work for me.
3. No considerable success yet

The target for these experiments is installed on my laptop (Ubuntu-9.04) with following procedure. Most of these steps are taken from blog Using iscsi over ubuntu 9.04

Installing iscsi target

sudo apt-get install iscsitarget

Configure the target by editing /etc/default/iscsitarget, /etc/ietd.conf and /etc/initiators.allow files as follows

1. Open /etc/default/iscsitarget and set ISCSITARGET_ENABLE=true
2. Open /etc/ietd.conf, and comment out all lines, then add following paragraph

Target iqn.2001-04.com.example:storage.lun1
        IncomingUser 
        OutgoingUser
        Lun 0 Path=/var/www/debian-live-501-i386-gnome-desktop.iso,Type=fileio,IOMode=ro
        Alias LUN1

Now we add following line to “/etc/initiators.allow”

  
ALL ALL

This should allow any initiator to access any lun device
and then, restart the target

sudo /etc/init.d/iscsitarget restart

The iscsi target is tested by installing iscsi initiator on same laptop. Following is the procedure.

• Installing

sudo apt-get install open-iscsi

* configure iscsi initiator to start nodes automatically\\

Edit the file /etc/iscsi/iscsid.conf to have line node.startup = automatic

* and then restart the initiator

/etc/init.d/open-iscsi restart

• Now, do the discovery of nodes available

$ sudo iscsiadm -m discovery -t st -p 192.168.0.1
192.168.0.1:3260,1 iqn.2001-04.com.example:storage.lun1

• I dont know why this command is needed, but it was in the blog, so I am adding it here

$ sudo iscsiadm -m node
192.168.0.1:3260,1 iqn.2001-04.com.example:storage.lun1

• Logging in

$ sudo iscsiadm -m node --targetname "iqn.2001-04.com.example:storage.lun1" --portal "192.168.0.1:3260" --login
Logging in to [iface: default, target: iqn.2001-04.com.example:storage.lun1, portal: 192.168.0.1,3260]
Login to [iface: default, target: iqn.2001-04.com.example:storage.lun1, portal: 192.168.0.1,3260]: successful

• and it is done, you can see the new device with fdisk -l command

$ sudo fdisk -l
[sudo] password for pravin: 

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xa9f9aa9b

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1           5       40131   de  Dell Utility
[...]
[...]
[...]

Disk /dev/sdc: 724 MB, 724785152 bytes
23 heads, 61 sectors/track, 1008 cylinders
Units = cylinders of 1403 * 512 = 718336 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table

$ sudo mount -t iso9660 /dev/sdc mpoint
mount: block device /dev/sdc is write-protected, mounting read-only

$ mount
/dev/sda6 on / type ext3 (rw,relatime,errors=remount-ro)
[...]
[...]
[...]
/dev/sdc on /home/pravin/Etherboot/git/BKO.git/pxeknife/debian/debian-live-501-i386/mpoint type iso9660 (ro) 

$ ls mpoint
doc  isolinux  live  md5sum.txt

As Now I know that my target is working correctly, I am free to try out booting from this target

I need to add all the tools, configuration files and kernel modules to initramfs so that I can use them from there to connect.

• Step 1 , Kernel Modules : on checking the contents of /etc/init.d/open-iscsi it seems that following kernel modules are to be inserted

modprobe iscsi_tcp
modprobe ib_iser 

• These modules are already in debian initramfs, so no need to worry about that :)

• Step 2 Tools needed : Again /etc/init.d/open-iscsi tells us that it uses /sbin/iscsid and /sbin/iscsiadm
  

but there are more iscsi related tools in /sbin

$ ls /sbin/iscsi*
/sbin/iscsiadm  /sbin/iscsid  /sbin/iscsi_discovery  /sbin/iscsi-iname  /sbin/iscsistart

To avoid any missing dependencies, better to add all above tools to initramfs.

• Step 3 Configuration Files : From man page of iscsiadm and iscsid, we can guess that following configuration files will be needed

/etc/iscsi/iscsid.conf
/etc/iscsi/initiatorname.iscsi

The HTTPFS code has been commented out so that initramfs will give me rescue shell, which can be used to start the iscsi initiator.
Following commands were executed once the rescue shell is provided.

modprobe iscsi_tcp
modprobe ib_iser
mkdir /var/run
/sbin/iscsid
iscsiadm -m discovery -t st -p 192.168.0.1

and no node is shown :(
I tried to run iscsid in debug mode with following command

/sbin/iscsid -d 8 -f
iscsid : sysfs_init: sysfs_path='/sys'
[...]
[...]

iscsid: reaped pid 994, reap_count now 0

and also ran iscsiadm in debug mode with following output

(initramfs) iscsiadm -d 8 -m discovery -t st -p 192.168.0.1
iscsiadm: ip 192.168.0.1, port 3260, tgpt -1
iscsiadm: Max file limits 1024 1024

This output is definitely very less compare to the output of same command executed on my laptop.
Note: This was executed over qeum. I am not sure if qemu has created any problem.

To make sure debien in virtual environment can use iscsi with the tools, and configuration files used inside initramfs

1. I booted debian live CD with virtualbox
2. inserted kernel modules just as above
3. Copied configuration files into live cd.
4. Copied tools into live cd
5. Executed above commands from inside live CD
6. The iscsiadm -m discovery command did gave output.

# iscsiadm -m discovery -t st -p 192.168.0.1
192.168.0.1:3260,1 iqn.2001-04.com.example:storage.lun1

So, the tools do work. but may be they are able to use other files, tools and modules which are not present in initrd.
How to find those dependencies?

Finally, added the strace to initramfs, and now trying find what is the problem because of which iscsiadm is dying

socket(PF_FILE, SOCK_STREAM, 0)         = 3
connect(3, {sa_family=AF_FILE, path=@"ISCSIADM_ABSTRACT_NAMESPACE"...}, 110) = 0
write(3, "\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 5244) = -1 EPIPE (Broken pipe)

This is what killing the iscsiadm imatuarly.

With help of stephana, the likely problem is located at following location

stefanha: http://git.kernel.org/?p=linux/kernel/git/mnc/open-iscsi.git;a=blob;f=usr/mgmt_ipc.c;h=e784bcac019a84522e62beff1b49b1ca62e37f5a;hb=HEAD
(07/19/2009 04:34:21 PM) stefanha: less1: Perhaps you're hitting this:
(07/19/2009 04:34:27 PM) stefanha: http://git.kernel.org/?p=linux/kernel/git/mnc/open-iscsi.git;a=blob;f=usr/mgmt_ipc.c;h=e784bcac019a84522e62beff1b49b1ca62e37f5a;hb=HEAD#l549
(07/19/2009 04:34:31 PM) stefanha: if (!mgmt_peeruser(fd, user) || strncmp(user, "root", PEERUSER_MAX)) {
(07/19/2009 04:34:43 PM) stefanha: write(2) will return EPIPE if the UNIX domain socket was closed.
(07/19/2009 04:35:17 PM) stefanha: mgmt_peeruser() is defined here: http://git.kernel.org/?p=linux/kernel/git/mnc/open-iscsi.git;a=blob;f=usr/mgmt_ipc.c;h=e784bcac019a84522e62beff1b49b1ca62e37f5a;hb=HEAD#l342
(07/19/2009 04:38:15 PM) less1: stefanha: is it checking if I have root privilages? I am not sure if I understand which peeruser credential is it refering.
(07/19/2009 04:40:08 PM) stefanha: Your user name must be "root".
(07/19/2009 04:40:25 PM) stefanha: Otherwise iscsid will not accept commands from the UNIX domain socket.

modified the iscsiadm code so that, it will not fail when username is not root.
With is modification, the discovery part is working, but now there is error at login part. Following is the error.

connection2:0: Could not create connection due to crc32c loading error.  Make sure the crc32c module is build as module into the kernel.

The first attempt will be to download the kernel module from Live CD of debian after booting it from CD.
— Doing it

There was need for two extra kernel modules

1. libcrc32c.ko
2. crc32c.ko

Fetched these modules from live-CD and added them.
There as also need for small sleep after iscsiadm –login as /dev/sda was taking little time to appear.
This was handled by adding sleep 2

Two new parameters were introduced

1. iscsifs=<IP:PORT> iscsi server IP and port number
2. target=<targetname> the iscsi target name of the disk (entire name is not needed, some part which can uniquely identify apart from other disks exported at same machine)

Following are the errors that I get when connecting to remote iscsi

Connection1:0: ping timeout of 5 secs expired, last rx 307737, last ping 308987, now 310237
Connection1:0: detected conn error (1011)
SQUASHFS error : Unable to read page, block 0x1b392
Bus error

Most of these errors are related to timing, and I believe that these errors are there because of delays in accessing required data.

sometimes, you may get following error when logging in with iscsiadm

 
iscsiadm -m node --targetname "${targetname}" --portal "${ISCSIFS}" --login 
records not found!.

this is because the ${ISCSIFS} is not IP but dnsname. Dnsname may work with iscsiadm -m discovery -t st -p “${ISCSIFS}” but it may verywell fail when same dnsname is given to login. mailing list discussion

Dont use –portal “${ISCSIFS}” option, use

 
iscsiadm -m node --targetname "${targetname}" --login