Tuesday, August 3, 2010

How to set up a smooth-running iSCSI SAN

iSCSI is easy, but it's also easy to do wrong. Here's how to avoid the common pitfalls and ensure your iSCSI SAN hums right along

Since its arrival six years ago, iSCSI has taken the storage world by storm. Proponents usually focus on iSCSI's excellent price performance, but the real secret to its success is the widely known skill set required to implement it: iSCSI uses standard TCP/IP networking to move block-level data around, so network administrators with little storage experience feel right at home.

In fact, iSCSI is so easy to set up, it's also easy to set up incorrectly. Depending upon the situation, this can result in poor performance, poor reliability, or both. Building an iSCSI SAN that can deliver enterprise-grade performance and reliability takes a bit of planning and a thorough knowledge of how your chosen SAN platform works.

The minimalist disaster
Let's say you work for an SMB and have a few Windows servers you want to attach to an iSCSI SAN. The servers are pretty new and have four onboard 1Gbps NICs, but only one of them is plugged into a switch -- a low-end, unmanaged gigabit switch.

At first blush, it seems like you could plug the SAN into the switch, throw an IP address on it, install the Microsoft iSCSI initiator on the server, configure it to connect to the SAN, and be on your way. Actually, you could. You would be able to mount SAN storage from the server and it would probably work passably -- as long as you didn't try to do much with it. Throwing a real production load on the SAN volume, however, would quickly illustrate the inadequacies of this design.

There are several problems with attaching to iSCSI storage this way. You would have no switch or NIC redundancy, no load balancing on the SAN, none of the features you'd need to optimize the flow of iSCSI traffic over the switch, and -- perhaps worst of all -- storage traffic would fight with front-end client traffic for bandwidth.

Doing iSCSI the right way
The missing element in this scenario is obvious: a decent switch. A good switch is a critical part of an iSCSI SAN. Yes, any gigabit switch will work with iSCSI, but you miss out on some very important features with the bargain-basement, unmanaged variety.

First off, you need your switch to be non-blocking -- that is to say, it needs to be able to handle full-rate input and output on all of its ports simultaneously. Not all switches can do this; even some enterprise-grade switches share bandwidth between groups of adjacent ports (especially chassis-based switches such as older varieties of the Cisco Catalyst 4500, though there are many others).

Second, you want to be able to support flow control. Flow control is a Layer 2 Ethernet protocol that allows a receiving host to request that the sending host slow the amount of data that it's being sent. This prevents situations where either the server or SAN sends more data to the other than they are ready to accept -- something that can result in massive amounts of inefficient TCP retransmissions and generally poor overall performance.

Third, you want the switch to support jumbo framing. Typical Ethernet packets are limited to 1,500 bytes. Jumbo framing extends this to 9,000 bytes. This is important because it cuts the number of packets that need to cross the wire and results in slightly higher link efficiency and lower latency. Most modern switches support this, but not all can.

Lastly, get a switch that supports VLANs. You want to segregate iSCSI traffic onto its own VLAN -- partially for performance reasons, but mostly as a quick and easy way to ensure that unauthorized hosts can't connect to the SAN.

Once you've chosen a good switch, buy two if you can afford it. Having two won't increase performance, but it could save your bacon if one fails. A dual-core network architecture is always a good idea. It's even more important when block-level storage is running on the network.

Why? Look at it this way: If you unplug an ordinary server from an ordinary network, in most cases, the worst that will happen is that all of your users will be disconnected until you plug it back in. But when you have block-level storage running on the network, cutting access to storage could potentially destabilize things and cause services to crash. Although it's almost always possible to recover from such events fairly quickly without data loss, they are best avoided if at all possible. Implementing two switches can go a long ways toward that goal.

After you have your switches sorted out, you can attach your servers and SAN to them. On the server-side, use two of the four NICs in each server to connect to each of the two switches. Likewise, on the SAN, split the SAN's interfaces over the two switches. This gives you a fully redundant, dedicated pathway to storage. The server's front-end client traffic won't have to fight with back-end storage traffic and vice versa.

The next thing you want to do is configure the server to connect to the SAN. This is a little less straightforward than it might initially appear. The Microsoft iSCSI initiator makes it fairly easy to configure an iSCSI portal (SAN) and connect to an iSCSI target (volume or set of volumes). However, you'll need to do a bit of research into how MPIO (multi-path I/O) works to get it set up correctly.

Ideally, you'll have at least one connection to each iSCSI target for each NIC you've dedicated to accessing the storage. This gives you path redundancy (in case one of your NICs or switches fails), but also allows the SAN to load balance those two incoming connections onto different SAN interfaces. Not all SAN platforms do this, but for the ones that do, this is a worthwhile exercise.

Many SAN vendors distribute software packages that will install the iSCSI initiator and a path management DSM (device specific module) that will intelligently build MPIO connections in the way that will work best for your SAN (Dell EqualLogic's Host Integration Toolkit is a great example of this). If your vendor offers one of these, make sure to take advantage of it as it will generally result in better performance and is much easier to configure and maintain.

The smooth-running result
After configuring iSCSI networking in this manner, the actual iSCSI interconnect will seldom or never be a factor in performance or reliability problems. In most entry-level SAN environments, this will allow a few servers with two 1Gbps links each to completely saturate the mechanical ability of the SAN's disks to keep up well before the iSCSI interconnect becomes a limiting factor.

If you're running a much larger server or SAN environment, the exact details of the design might change -- perhaps using 10Gbps Ethernet and iSCSI HBA/CNAs in the servers to offload some of the work of sending iSCSI packets. However, the general concept of a dual-cored network comprised of high-quality switches and correctly implemented MPIO will still hold true.

Alhough iSCSI is "easy," it can actually get pretty darn complicated because of the many different ways it can be configured. If you're new to iSCSI, make sure you carefully plan and test your configuration before you push it into production.

1 comment:

  1. Good Article. Thanks for the information it's a great help.