Why SolidFire uses iSCSI

From the beginning, SolidFire storage clusters were designed to be used with the Internet Small Computer Systems Interface (iSCSI) storage protocol. iSCSI is a standards-based method of communicating between computing equipment, such as hosts and servers, and storage equipment, like SolidFire. It was standardized between 2002 and 2004 and is supported by all major operating system suppliers and almost all storage vendors, including many NetApp offerings. This is not going to be an iSCSI tutorial, many of which are readily available on the web. (You can also catch my SNIA podcast on the evolution of iSCSI live on May 24 or the recording after the fact.) Instead, this blog post examines the three main reasons SolidFire decided on iSCSI as its storage transport of choice.

iSCSI is a commodity protocol

What exactly does this mean? iSCSI defines a way of encapsulating SCSI commands on a standard TCP/IP connection, typically carried on an Ethernet network. This means that when the SCSI standards are changed by the ANSI T10 committee (responsible for maintaining the standard), iSCSI doesn’t need to change. It also means each time Ethernet speed goes up, from 10 megabits to 100 megabits to 1 gigabit to 10 gigabits and beyond, iSCSI benefits from the increase without having to make any changes. It means when Ethernet network interface controllers (NICs) add efficiency and offload capabilities, such as jumbo frames, TCP segmentation offload (TSO) and large receive offload (LRO), iSCSI seamlessly takes advantage of these benefits.


It also means that iSCSI is a truly networked storage protocol, because it’s carried by TCP/IP. TCP/IP primarily provides the guaranteed delivery and ordering of the network traffic, two things that are required for a storage protocol. It also provides the capability to route for larger networks. Using IPsec, it can encrypt data as it’s being transmitted — and a lot more. iSCSI benefits from the full feature set that TCP/IP offers, without needing to change the way it works as a storage protocol.

This isn’t true of other SCSI transport methods, such as Fibre Channel or Serial Attached SCSI (SAS). Other transports have needed to define the entire protocol from the physical signalling level to all of the network switching, reliability, and ordering. Evolutionary changes are only available if the transports, themselves, change.


Another way to look at this is that iSCSI makes use of the standard Ethernet network ports built into all data center equipment. iSCSI traffic is also switched by regular Ethernet switches, disposing of the need for a completely different network just for carrying storage traffic. There’s no need to buy specialized host bus adapters (HBAs) used by other storage protocols.


This leads to the primary reason iSCSI is a commodity protocol: It can be fully implemented in software without any special equipment. iSCSI HBAs exist, but they are more expensive than the cost of additional Ethernet ports and CPU cores necessary for handling iSCSI traffic. All of the major operating systems, VMware, Linux, Microsoft, and Solaris, offer a full-featured software iSCSI implementation. In fact, SolidFire has only qualified software iSCSI initiators, not seeing any need to support third-party iSCSI HBAs.


That iSCSI is commodity-based means SolidFire isn’t at the mercy of specialized hardware vendors or slow-moving standards bodies to take advantage of necessary features.

iSCSI’s secret sauce: login redirection


There is one feature provided by iSCSI that especially suits SolidFire (and any other scale-out storage, for that matter). In fact, it’s one of the reasons that first-generation scale-out storage, such as LeftHand Networks and EqualLogic, were iSCSI-based. That feature is iSCSI login redirection.


When an iSCSI host wants to communicate with iSCSI storage, it starts by sending a login request. This request contains identification about who is sending the request and which particular piece of storage, or “iSCSI target,” the host wants to communicate with. It can optionally contain authentication information, if required. It also contains a handful of other parameters that the host and storage negotiate. The iSCSI storage will validate the authentication, if needed and finish the negotiation of the parameters. Once the login is complete, an iSCSI “session” is established that carries SCSI commands and data between the host and storage. When one end or the other decides to end the session, it can request or issue a logout or, more rudely, simply close the TCP connection between the two. (If you’ve ever seen the error message “Connection reset by peer” in a browser or VPN, your TCP connection has been closed.) With iSCSI, it isn’t uncommon for a TCP session to stay up for days or months.


When an iSCSI login request comes in, storage can simply respond with a message informing the host that the target has temporarily moved to a different address and not continue forward with any of the regular negotiation. The host then reissues the login request to the new address. This is known as iSCSI login redirection and is a key to how SolidFire scale out and QoS work.


SolidFire advertises a single IP address (on a single node) where all volumes are available.

When a login request is received, that node decides which node should actually handle the traffic for the specified volume, and that node redirects the login to the IP address of the appropriate node. That way a node can be chosen that can handle the IOPS and capacity requirements for that volume.


If a single node is handling too much traffic for its volumes, or a new node is added and volumes are redistributed, the SolidFire cluster asks the host to logout. Immediately after logging out, the host attempts to log on to the original IP again (since it was told the target temporarily moved), and it is redirected to the new node serving that volume.


The same approach works if a node is being removed from a cluster or a node goes down.  In the case of a node being removed, all of the volumes served by that node request a logout and the host is redirected to a different node when it attempts to login. If a node goes down, the host attempts to log on again. As soon as the SolidFire cluster realizes the node is down, login requests are redirected to nodes that are up. In this way, iSCSI login redirection provides self-healing capabilities to SolidFire.

Cluster upgrades use the same approach as removing a node.  Any volumes being served by a node are redirected to another while that individual node is being upgraded. Once an upgraded node is ready, it can start serving a volume and another node can be upgraded. This is all done without intervention and provides SolidFire with a nondisruptive upgrade technique.


In its ability to provide a way to redirect a session to another location before fully setting it up, iSCSI stands alone among SCSI transports. The iSCSI host doesn’t have to “discover” where the session is redirected to and doesn’t have to maintain information about where it was redirected after the session closes. That this capability facilitates scale out, load balancing, self healing, and nondisruptive upgrading is another reason SolidFire chose iSCSI.

iSCSI eases configuration and automation


SolidFire also wanted a provisioning method that was easy to automate when customers deploy at large scale. Other SCSI transports require a complete understanding of the relationship between host initiators and target ports in the storage network so access to the appropriate volumes can be configured. This means keeping track of extensive configuration information that needs to be applied and updated any time storage or hosts are added or replaced. In large enterprises and service providers, this may amount to keeping track of thousands of entities and their relationships.


In its simplest form, iSCSI only requires the host’s identity and the SolidFire storage address to map volumes to hosts. In turn, this allows simple programmatic provisioning without the need to touch other components of the infrastructure. The reduction in complexity makes automating end-to-end iSCSI connectivity very simple using standard data center automation tools.


A lot of people have asked why SolidFire chose iSCSI. That it’s a commodity protocol, provides login redirection, and facilitates automation are certainly three of the main considerations. Hopefully you now have an understanding of the market and technology considerations that went into our decision.


Learn more about setting up your storage network in the Storage Network Design Choices technical brief.



Andy Banta

Add comment