Our Biggest Design Decision – Direct Wired or Backplane?

This is Part 2 in a series of posts from 45 Drives co-founder Doug Milburn, reflecting on the evolution of 45 Drives, where we’re headed and why. Read Part 1 to find out how 45 Drives got its start, with help from Backblaze.

Our Experience with Backplanes 

The original Backblaze Storage Pods were designed around port multiplier backplanes. These are circuit boards that have multiple SATA hard drive connectors (5, in this case), and one SATA cable connector. This allows up to 5 hard drives to be plugged into the board, and multiplexed and sent through 1 cable to a shared SATA port. Therefore, 45 drives require only 9 cables and SATA ports. This is economical as well as practical. Combined with Backblaze’s vertical mounting, this was a brilliant design.

There are a few restrictions and caveats, of course. The SATA adaptors must be chipset compatible with the backplane port multipliers, and drivers are required. Physical dimensions of 5 port boards follow a defacto standard. Also, data for 5 drives shares 1 SATA cable (3Gbit in the original design), limiting performance to 60 MBytes per second per drive (whereas mechanical drives can transfer approximately 150 MBytes per second). But all things considered, limited throughput speed was a reasonable sacrifice for Backblaze, and backplanes were what drove the success of Pods 1.0 to 3.0.

Backblaze has operated their business successfully for years now, mostly with backplane machines. But we experienced a number of challenges when we sold these to the larger storage community.

Conflicts with the Needs of Other Users

At 45 Drives, it is in our nature to get to know our customers and what they are doing (or want to do) with their pod. And we really can’t help thinking about variations to the design, and how it might improve the pod for various applications. Because we are part of Protocase, a mass customizer and the fastest low-volume metal manufacturer on earth, evolving the design was an inevitable step for us and continues to be an ongoing endeavour. It’s what we do.

Over the years, we have listened to feedback from hundreds of very bright people involved in a wide variety of applications. In addition to the standard pod designs, we have built a large number of customized pods (in fact, we end up customizing almost 50 per cent of pods to some degree).

Here’s a summary of feedback we’ve received about the original pod design:

Slide Rails: Almost everyone we deal with considers ball-bearing slide rails to be necessary for convenient access. But the defacto standard pitch of 5-drive port multiplier backplanes makes it impossible to fit a row of 15 drives across a 19-inch rackmount enclosure and still have space for slide rails. This is not an issue for Backblaze, because they don’t access machines to change drives, but rather pull machines out of racks to service them in batches. Therefore, they mount machines on fixed shelves, saving themselves the cost of slide rails in the process. This works fine for them, but pretty much everyone else in the community wants slide rails.

Speed: Most of the storage community is interested in speed. It’s one of those things that make everything easier.  Some users really need speed, such as organizations that stream video on demand. We build machines for several such companies. Their performance target is to stream simultaneously to thousands of connections, with throughput in excess of 1 gigabyte per second. Backplane-based pods 1.0 to 3.0 were only capable of throughput of about 200 megabytes per second, and therefore didn’t make the cut.

Reliability and Compatibility: The model CFI-B53PM (SIL 3726) backplanes and SYPEX 40008 SATA cards of Pod 2.0 (which is when we came on board with Backblaze) were a great place to start, and they were very cost effective. However, failure rates were high enough that we quickly found this to be a strain on the business model. Of course, having 3 SATA cards and 9 backplanes meant each machine would experience 3x and 9x the respective single card failure rates. Some early adopters of storage pods utilized redundancy to compensate for failure rates. But for others, this was just not acceptable, and we ended up taking back or upgrading machines to direct-wired for those customers whose applications were incompatible with this approach. It was clear to us that the general storage community needed a dramatic improvement in this area.

Modularity and Maintainabililty

Another shock came when the CFI-B53PM (SIL 3726) backplanes went end of life. We had very little notice, and the challenge of finding a replacement was compounded for the need to simultaneously meet several criteria: Backblaze’s requirement of absolute lowest cost, and matching SATA card chipsets, backplane and drivers. CFI released an AC-SAN-5PMBP SIL 3826 chipset, which we were ultimately forced to use in our 3.0 pod. I should note here our 3.0 pod (also known as the TeraStor) was slightly different than Backblaze’s 3.0 pod, because they had adequate supply of CFI-B53PM (SIL 3726) backplanes, they wanted to continue using them.

But we soon realized that backplane architecture was restrictive, in terms of what we could do for our many customers who wanted customization. The chip sets of the SATA cards were so old, they were incompatible with any newer motherboards and CPUs, restricting any customizations. (The SYPEX 40008 SATA card was only a 1.0 PCI connection, for example).

Because the SATA cards and the backplanes must be chip set compatible – if one or the other goes end of life, it can really complicate things. When the SIL 3726 went end of live, it required a special modification to the kernel to allow us to utilize the SIL 3826 replacement. This became really problematic for service calls involving failed 3726 backblanes.

Furthermore, we heard from our community loud and clear that they wanted to have a pod with SAS compatibility with Serial Attached SCSI. We quickly realized the backplane versions would not allow us to use SAS drives. Even sourcing SAS backplanes would require extensive modifications and drive up costs dramatically. SAS backplanes were an entirely different design than SATA backplanes, and as a result, it was not feasible to start offering SAS compatibility.

The Inspiration for HBA Cards and Direct Connection 

At Protocase and 45 Drives, we’ve had the opportunity to work with a wide variety of organizations who innovate in computing. We did work for two companies who were building their own storage pods, but utilizing direct cabled connections between their drive arrays and high-port count Host Bus Adaptor cards which provide a large number of high-performance SATA / SAS ports.

We also had interesting exchanges with engineers at Netflix, who, as many know, build their own storage servers to stream video to their customers. They were successfully making direct wired connection architecture work on a large scale.  We and Backblaze agreed that this approach had certain advantages, and began development of the Pod 4.0 architecture (otherwise known as the Storinator), based on an inventive leap of fixing connectors on ‘wired backplanes’ so drives could be mounted in the same configuration as previous pods.  

Here’s how Pod 4.0 addressed the issues:

Slide Rails: Drive spacing is now a design choice, and not restricted by defacto standard for 5-port backplanes. So we were able to redesign the pod to make it slightly narrower, thereby accommodating ball bearing slide rails. Problem solved!

Speed: Each drive speaks directly to the HBA card over SATA cables, so even at 3Gbit speeds, the bandwidth is sufficient to meet the maximum capacity of mechanical drives. And even though there is internal multiplexing on some HBA cards, it was quickly determined that we could achieve speeds well in excess of 1 gigabyte per second from striped arrays of mechanical drives. This was not meaningful to Backblaze, but it was truly enabling for many other users.

Reliability and Compatibility: We instantly dropped board count from 12 (3 SATA adaptors and 9 backplanes) to higher quality HBAs (Host Bus Adaptors) in the form of 2 Highpoint Rocket 750’s or 3 LSI 9201’s. This upped our hardware reliability. However, it wasn’t an instant victory, as we had problems developing a cable with the reliability we needed. We also suffered from a bug in the Highpoint Rocket 750 driver, which was related to adapting to using more than one card in a system.

Once we worked through these problems, we’ve found that when we check back with customers regarding systems we’ve shipped to them, they are finding that they “just work” – which is a good thing. Although the odd system has troubles, we easily work our way through them, and we have a fraction of the warranty returns that we saw in the past. The 4.0 direct-wired pod design also allowed for much easier drive identification, whereas when backplane models failed, it was a maintenance issue to identify drives and backplanes. Not only did backplanes fail more, in our experience, but they took much longer to diagnose issues.

Modularity: The simplistic yet versatile direct wired design consists of a drive array and a collection of wires on one side, and a server grade motherboard on the other. The HBAs (which can be easily swapped out in the 45 Drives production facility, or as a field upgrade) make them work nicely together. This makes many things that were previously a headache – upgrading, dealing with end of life or customizing – a snap.

SAS: The modularity of this architecture and our use of SAS compatible cables mean we can offer SAS capability. We’re still testing, but you should see some updates very soon.  This opens a whole new world of performance and data security.

Our Direction is Clear

Backblaze has announced it is going back to a backplane-based design. We respect their choice, and understand that it makes total sense within their business model and technology context. However, we are an organization that, by our nature, fully supports the machines we sell.

Although the majority of those who bought the earlier backplane-based pods did just fine, we saw too many users struggle to make this class of machine work for them. Once we got through our early cable and driver problems, we saw the load on our support team drop dramatically, and saw the number of users who were delighted by the performance go way up. So we are going to stay with direct wired, because we think it is better for both us and our customers.

Would we ever make backplane machines? Absolutely. But it would likely be custom machines. We regularly get involved in designing and building custom machines for those who are developing their own ‘optimum solution’ to their computing needs, just as Backblaze has. After all, we are part of Protocase, which is the fastest and best mass custom manufacturer on earth.

If backplanes are the way a customer wishes to go, we will build it for them. However, at least for now, we think direct-wired designs will put more smiles on more faces when they fire up their new machine, configure an array, and see how fast and easily they can saturate a 10GbE pipe. We’ve done our own testing on this, and will be releasing that information soon.