Multipath configuration on Linux Servers (RHEL or CentOS 5.5 or higher)

   Multipath on Linux Servers

In Today's post, we'll go over configuring multipath on Linux Servers.  This is applicable to RedHat 5.5, 5.6 and Oracle Enterprise Linux (OEL) 5.6.  Also, this is in working with QLOGIC HBA's, it will be different with other HBA's!

We will be configuring LUNs presented by a 3PAR or EMC SAN for use on the above Unix servers.

Necessary Information:

The person that provisioned the storage has to provide the worldwide ID (WWID) for each LUN assigned to the Linux server.  This is critical when multiple LUNs are to be assigned at the same time as the WWID is uniquely assigned to each LUN.    

Note that WWIDs should only have lower case characters(!) 

The Linux server should have a copy of the Qlogic LUN scanning software installed at /usr/local/bin/.  Otherwise, a copy should be made of the ql-dynamic-tgt-lun-disc.sh script.

Depending upon the version of Linux you are using, you will need to use a specific version of a generic copy of the multipath.conf file.   These versions do not include support for EMC LUNs at this time.  A copy of this file should be placed in the /etc/ directory on the Linux server being configured.

Configuring multipath:

You have to be root in order to execute the following commands:

1)    Execute the following command to get the SCSI ID of the internal SCSI disk.  This is done in order to blacklist it from being scanned within multipath.conf.

[root@linux01 ~]# scsi_id  -g  -u  -s  /block/sda

2)    Start up the multipathd daemon process.

[root@linux01 ~]# service multipathd start

3)    Enable multipathd to start on subsequent boots.

[root@linux01 ~]# chkconfig --add multipathd
[root@linux01 ~]# chkconfig multipathd on

(chkconfig provides a  simple  command-line  tool  for  maintaining  the 

/etc/rc[0-6].d  directory  hierarchy by relieving system administrators 

of the task of directly manipulating the  numerous  symbolic 

links  in those directories.)

4)    Execute the following commands to scan for LUNs on the Linux machine:

[root@linux01 ~]# ql-dynamic-tgt-lun-disc.sh  -cl
[root@linux01 ~]# ql-dynamic-tgt-lun-disc.sh  -s

5)    Edit the multipath.conf file to include the internal SCSI disk’s unique SCSI ID.

WAS:    wwid        # PERC controller from /block/sda
IS:        wwid <hex string from the scsi_id output> # PERC controller from /block/sda

6)    Execute the following command to verify that the LUNs are visible.  If the LUNs are from a 3PAR, you should see the string “3PARdata” visible for each LUN in the output.
You should see one “mpath?” where ‘?’ is a different letter for every LUN detected.

[root@linux01 ~]# multipath –ll

7)    Edit the multipath.conf file again to insert the WWIDs for each LUN as shown by the previous command and assign alias names to the LUNs.  This requires adding the following to this file:

multipaths {

      multipath {

                  # <some descriptive comment>

                  wwid <WWID of the LUN>

                  alias <some descriptive alias name>

      }

}

The portion above that is italicized should exist for every existing LUN.
Note that the WWID of the LUN should be provided by the person that provisioned the storage but the hex string should be proceeded by the character ‘3’.  This resulting WWID can be compared to the output of the “multipath –ll" command for correctness.
The alias names should be descriptive such as “grid1” for the first Oracle ASM LUN, “data1” for the first Oracle data LUN, or “fra1” for the first Oracle flash recovery area.

8)    Restart multipathd to reread the multipath.conf file.

[root@linux01 ~]# service multipathd restart

9)    Executing the “multipath –ll" command should now display the alias names.
[root@linux01 ~]# multipath –ll

10)    You should also verify that the LUNs now exist in the /dev/mapper/ directory.  You should see each LUN specified by it’s alias name (e.g. grid1).

[root@linux01 ~]# cd /dev/mapper
[root@linux01 ~]# ll

11)    If these LUNs are being created for Oracle, I will publish another article at a later date.

12)    If additional LUNs are added, execute the following:

# ql-dynamic-tgt-lun-disc.sh  -s
# multipath –ll

Edit the multipath.conf file again to insert the WWIDs for each new LUN as reflected by the previous command and assign alias names to the new LUNs.

[root@linux01 ~]# service multipathd restart

13)    If you wish to rename an alias, execute the following:
Edit the multipath.conf file to insert the new alias name(s) assigned to the LUN.

[root@linux01 ~]# service multipathd restart

Verify that the alias(s) have changed by looking at the /dev/mapper directory.

Cleaning up failed/faulty (dead) paths in multipath:
This situation can occur if LUNs are removed or in case of a hardware failure.

1)    Flush the device (e.g. /dev/sdx) from the multipath service:
[root@linux01 ~]# multipath -f sdx

2)    Flush any outstanding I/O to the device:
[root@linux01 ~]# blockdev  --flushbufs  /dev/sdx

3)    Remove the device file from the SCSI subsystem:
[root@linux01 ~]# echo 1>/sys/block/sdx/device/delete

4)    Verify that the device file is gone from multipath:
[root@linux01 ~]# multipath -ll | grep –i sdx