Storage Capability that Rivals a Super-Computer
Nebula uses the LUSTRE clustering file system (an open-source project maintained by SUN) to provide highly-scalable storage in the hundreds of thousands of terabytes. This is deployed on a cluster of 64-bit storage nodes, allowing almost unlimited individual file size, and connected to a dedicated 10GigE network. Never before has such research-grade computing been available in a web application platform.
(Additional information about the Lustre file system is available on the Lustre website.)
Data storage has a set of critical characteristics that define the requirements of any storage solution.
- Reliability
- Permanence
- Performance
Evaluating these characteristics in the context of cloud applications, Nebula identifies five classes of storage.
- Operating System
- Temporary Storage
- Computed Data
- Gold Standard Data
- Backups
We fulfill the requirements of these classes of storage in three distinct hardware configurations:
Primary Lustre Nodes
These are rack-dense, 2RU servers with 12 SATA drives in a RAID 6 configuration. They are equipped with 10GigE network ports. Our current configuration uses 1TB drives, for non-blocking access to 10TB of usable storage per server. This provides us with 4 CPU cores, and 5TB of usable storage, per rack unit, at non-blocking network speeds.
(We are currently evaluating the use of even higher-density equipment, such as the SUN xFire servers, which can provide disk density up to 2X that of this configuration. However, network connectivity becomes a bottleneck in these configurations, and they have a much lower ratio of CPU to disk. Additionally, they are substantially more expensive per gigabyte.)
These primary nodes provide large quantities of temporary and computed data storage.
SAN-backed Lustre Nodes
For gold-standard and backup data, we use fiber-channel-connected LUNs on a multiply-redundant and geographically-diverse SAN. This provides us with highly-reliable and permanent storage, and decouples the storage from the connected server as a single point-of-failure. However, this comes with a modest performance penalty (our FC connectivity is at 6 gigabit), and a cost per gigabyte that is more than an order-of-magnitude more expensive.
Critical Lustre Nodes
For those components of the Nebula infrastructure where we cannot accept the performance penalty of a SAN connection, and where RAID 6 is insufficiently reliable, we use pairs of Primary Lustre Nodes, configured with RAID 1+0, and mirrored against each other using DRBD in a Active-Passive pattern. This configuration is used for redundant Master database servers, and Lustre Metadata Servers.
Every Nebula account comes with 100GB of storage in the Primary Lustre filesystem. Additional storage is available. Please contact your NEBULA account manager.