#multipathing

iSCSI connectivity is critical when used for SAN in a datacenter. High Availability and guaranteed bandwidth availability are essential for successful operation with the desired up-time and performance requirements. Different multipathing schemes are used for the iSCSI connections to ensure high availability and bandwidth. Multipathing in simple terms is the availability of redundant physical components like, adapters, cables, switches, to create redundant logical iSCSI paths between an iSCSI initiator and an iSCSI target. Having multiple paths ensures high availability in case of a failure and higher bandwidth. There are three main types of multipathing for iSCSI.

Link Aggregation

Active/Standby Multipathing

Active/Active multipathing

Link Aggregation: IEEE 802.3ad Link Aggregation Control Protocol (LACP) and EtherChannel

Link Aggregation refers to the concept of aggregating two or more iSCSI links into a single link. IEEE standard 802.3ad and EtherChannel both independently specify a method for aggregating multiple Ethernet links to form a single network link. The packet traffic between an initiator and a target is distributed among aggregated links using a hash function based on a variety of parameters, including source and destination MAC addresses, source and destination IP address, or other. The paths are generally converged at the NIC driver layer or the operating system interface layer. A single IP address is assigned to the set of links on both the initiator and the target.

IEEE 802.3ad standard does not require any particular distribution algorithm but the distribution algorithm used needs to meet the following criteria.

The algorithm will not

a) Mis-order frames that are part of any given conversation, or

b) Duplicate of frames.

These conditions are met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated. This means that the traffic might not be uniformly distributed across the aggregated links. Generally, there are enough conversations to, more or less, distribute the traffic between the aggregated links but if the traffic is composed of only one conversation, i.e. one initiator/destination, then at any given time, only one of the aggregated links will be used. In such a scenario, the effective bandwidth will be equivalent to the bandwidth of a single port. If this is a 10Gb environment and if only iSCSI traffic is passing through this port, this may be OK. Regardless, this will provide redundancy and will make the iSCSI link highly available. This is an important concept: Link Aggregation does not always mean a fatter pipe.

Using a round-robin algorithm, instead MAC or IP hash, would correct this problem, but it does not meet the criteria for the 802.3ad standard. EtherChannel and other nonstandard mechanisms do allow for the use of round-robin distribution. Using EtherChannel would provide the opportunity to get rid of this issue but not all NIC and switch hardware support EtherChannel. 

As evident, the end-user performance and redundancy requirements, available hardware and other factors need to be carefully considered before deciding to use Link Aggregation. 

Active/Standby Multipathing:

Active/Standby multipathing creates high-availability but does not increase the bandwidth of the solution. In case of Active/Standby Multipathing, iSCSI initiator will initiate a TCP connection per port to the iSCSI target. That means, for two NIC ports, two TCP connections will be established. One of these TCP connections will be the primary connection, which will be active; and other ones will be standby at any given time. Data will flow down the active connection until active connection fails. In case of such a failure, traffic will then be diverted to the standby connection. Since the standby connection is fully established, failover is very fast.

  The choice between Active/Standby and Active/Active multipathing also depends upon the iSCSI SAN storage array. Not all array controllers support Active/Active and Active/Standby both. Some might support either of the two and some might support both. 

Also, it is important to note that Link Aggregation can be used with Active/Active or Active/Standby multipathing. As mentioned before, Link Aggregation creates a single link by aggregating multiple links. These "aggregated single links" can be multipathed in Active/Active or Active/Standby mode. This is especially useful when some aspect of the design dictates the use of Active/Standby multipathing. In this scenario, multipathing is not going to increase your bandwidth. In this case, Link Aggregation can be used to improve total bandwidth by creating Active/Standby multipathing between two aggregating links of two. For example, if you need 20Gb of storage bandwidth with Active/Standby multipathing, you would need 4x 10 Gb ports in two bundles of two to achieve this. 

Active/Active Multipathing:

Active/Active multipathing increases the available bandwidth between an iSCSI initiator and target. In Active/Active multipathing, bandwidth of both of the adapters can be used. This is, of course, dependent upon the algorithm used for load balancing. If the load balancing algorithm performs hashing of IP or MAC, then both of the links may not be used simultaneously all the time. If round-robin or other similar algorithms are used, efficiency simultaneous use of both of the adapters is possible. 

Active/Active multipathing is performed by using a software agent (a multipathing driver) in the software stack above the SCSI layer. This multipathing module presents the two iSCSI initiators as a single path or link to the disk driver. As you can imagine, this is only possible when both of the iSCSI initiators are connected to the same end device (a LUN or datastore). This is ensured by using SCSI inquiry commands to check for commonality in device paths, usually the LUN serial number. 

As mentioned before, Link Aggregation can also be used with Active/Active multipathing, although it may not provide any significant benefits.

Summary:

A lot of people are moving towards iSCSI and away from FC due to costs, reduced complexity, or simplified management.  iSCSI multipathing support started with vSphere 4 and is improved with vSphere 5.  Now the individual physical NICs can be mapped to vmk port groups from within the GUI.  This saves a lot of mucking about in obscure commands from the CLI.  There is one thing VMware has not included, and that is a simple way to set Round Robin path selection for all the LUNs in a datastore and for all the hosts in a cluster.  This is particularly annoying if you had a lot of extents in your datastore, or a lot of hosts in your cluster.  So I wrote a script using PowerCLI to automate the process for me.  Now, whenever I add a new datastore, I simply run the script and let it set the Round Robin MPP for me.  Check it:

#Configure_Round_Robin.ps1

#Written 9/21/2011 by Ned Bellavance

#This script should change the path policy on iSCSI LUNs connected to the a given storage model device to Round Robin

# It will not change the policy on any other storage devices

connect-viserver -server yourvcenterserverfqdn

$HostArray=get-vmhost

foreach($VMHost in $HostArray){

Get-VMHost -Name $VMHost.Name | Get-ScsiLun -LunType "disk" | where{$_.model -like "*YourStorageDevice*" -and $_.multipathpolicy -ne "RoundRobin"} | Set-ScsiLun -MultipathPolicy "RoundRobin"

}

It's a pretty simple little script and all you need to know is your vCenter FQDN and the model of your storage device as VMware sees it.  To find that out, just run Get-VMHost -Name (a host in the cluster) | Get-ScsiLun -LunType "disk" | ft Id, Model, MultipathPolicy.  That will return all the disks connected to your host in table format.  The Model is the property you will use to filter the command.  In my case, all the datastores are connected to a Celerra, and sure enough that's what the Model comes up with.