A History of Storage: Block
How drives, protocols, and networks have evolved to indefinite scale
When cloud practitioners talk about cloud storage, they generally refer to three primary categories: block, file, and object storage. These are not comprehensive or fully exclusive categories, but they are consistently decisive ways of referring to the ways in which data may be stored in cloud systems.
Let us explore the history of each of these and what they entail. This post will focus on block storage. We will look at brief pockets in a deep history that could be far more technical and exhaustive than we will choose to be here. This is mainly for the amateur or professional seeking a deeper understanding of the machines we manage.
Block Storage: Origins in the Mainframe
The early days of computing depended upon expensive, bulky mainframes where we find the advent of disk-based technology.
We can start with the IBM System/360, the famous “$5 billion gamble” introduced in 1964 which Jim Collins ranked among the top three business accomplishments of all time, next to the Boeing 707 and Ford Model T.
The System/360 presented a landmark computing architecture which consolidated a number of computing advancements within a single framework. This was where the 8-bit definition of “byte” was born.
Alongside the IBM System/360 emerged the IBM 2311 Disk Storage drive.
This disk storage drive provided direct access to 7.25 million 8-bit bytes per removeable disk pack. Eight drives could be attached to one single control unit for a total of 58MB with a transfer speed of 156 KB/s.
The disk was broken into a Cylinder/Head/Sector (CHS) geometry. Mainframe disks (14 inches in diameter) would rotate platters. Each platter had concentric tracks which in turn were grouped into cylinders, which were stacks of tracks aligned vertically across multiple platters.
Each track in turn was divided into a fixed or variable-sized sectors of data block, including the common pattern of 1024 byte blocks that are familiar to technologists today.
The head was the electromagnetic piece that hovers just about the disk platter and can either read or write data.
This was a mechanical process.
Read/write operations consisted of the following steps:
The control unit would send a “seek” command to the drive. This would trigger the actuator to move the heads.
The head would seek the correct cylinder and the system would wait until it was positioned above the right location. Because of this, the physical distance of cylinder traversal would determine how long the seek took (the IBM 2311 boasted an average seek time of 85ms).
When the correct record physically rotates under the head, the control unit signaled the channel to start reading or writing the raw bitstream. By reading, the data would be assembled into a buffer that was passed into main memory. By writing, the control unit would magnetize the cylinder track surface to etch the correct data bits according to a particular format (e.g. Count-Key-Data, abbreviated as CKD) determined by the OS or access method.
While this physical, magnetic procedure may seem foreign to us today, it is this underlying process which still defines how block storage works today, including modern operating systems.
We can glean these block storage principles from the IBM 2311:
Data is organized into fixed sized blocks (e.g. 1024 bytes)
Data is addressed by a numeric identifier
The OS or hypervisor reads or writes data based on addressing
The storage hardware does not know anything about what is stored (i.e. no metadata)
This is raw data. These fundamentals remain at the deepest layers under the hood of computing today.
SCSI and Open Standard Disk Storage
Block storage had its mainframe successors with IBM System/370 backed by the IBM 3330 and 3350 Winchester disk drives, IBM ESA/390 backed by 3390 as well as competitors such as the Honeywell 6000 or Burroughs B6500/6700.
These primarily operated under proprietary, internal channel interfaces.
This changed with the development of the Shugart Associates System Interface (SASI) in the late 1970s whose primary objective was to attach hard drives to multiple vendors’ computers. It was a move toward vendor neutrality via an open standard.
Much of SASI was co-opted into a more widely known standard Small Computer System Interface (SCSI) in 1986.
SCSI provided generic commands that could be applied across different vendors, such as INQUIRY, READ, WRITE, etc. This provided a higher level of abstraction where the SCSI device could handle seek times and geometry which no longer needed to be considered by the underlying host. The host would provide the commands and the firmware would translate into mechanical execution.
SCSI has since been extended into SAS, iSCSI, and Fibre Channel (FCI), all of which remain dependent upon that original SCSI command set.
SATA and PC Block Storage
By the 1980s, personal computing was on the rise and with it a concrete need to solve for the problem of storage. Integrated Drive Electronics (IDE) came about as a way to simplify PC hard drive connections by integrating the disk controller to the disk itself.
But PC drive storage truly hit its stride with ANSI’s ATA (AT attachment) as the de facto standard for PC consumers to interface with the hard drive.
This early iteration of ATA, now often referred to retroactively as PATA (Parallel ATA) was constrained by issues around signal integrity with the increase of bus speeds, primary/secondary configuration with multiple channels, and bandwidth ceilings generally hitting UDMA/133.
There was a need for better.
SATA (Serial ATA) 1.0 was published in 2003 (downloadable as ZIP here) out of the collaboration of multiple hardware vendors.
SATA 1.0 provided 1.5 GB/s signaling with ~150 MB/s throughput all while maintaining backwards compatibility with ATA.
It was then iterated on with SATA 2.0 (3.0 GB/s) the following year and by SATA 3.0 in 2009 (6.0 GB/s) with minor version increments since then. There has been no movement beyond this 6 GB standard primarily because of the push toward NVMe for high performance in the years to come.
Yet to this day SATA is of crucial importance for HDD storage of massive volumes at relatively low costs and remains a primary archival or backup strategy for cold data storage as the data at rest costs less per GB when compared against SAS or NVMe solutions.
Furthermore, SATA is recognized by nearly all hypervisors and operating systems, requiring almost no specialized hardware while maintaining a strong balance of performance and throughput for standard workloads.
Fibre Channel, iSCSI and Networked Storage
Before the 1980s, compute processes were attached to storage first through mainframe-style channel connections as we have seen above and then through directly attaching disk arrays to the hardware, what we now refer to as DAS (Direct Attached Storage).
However, personal computing meant multiple computers could be at play within a single data ecosystem, computers that were meant to talk to one another.
NAS (Network Attached Storage) solutions such as NFS (Network File System) in 1984 and SMB (Server Message Block) several years later were built to allow for file level sharing over a network.
This works for files but not for block-level access of data in the way that databases and many workloads would require. Organizations needed a way to standardize raw block device access across multiple servers.
Here is where SAN (Storage Access Network) comes into play as a new model for providing distributed block storage access, first through the invention of Fibre Channel in the 1990s.
Fibre Channel SAN works by encapsulating SCSI commands within Fibre channel frames that adhere to the FCP (Fibre Channel Protocol). Servers could perform SCSI read and write operations over a dedicated FC fabric to a centralized storage array for the first time in history.
Now there were new possibilities for block-storage fault tolerance and high availability, but at the same time this new complexity raised new problems to solve for as well such as networking infrastructure, fabric topology design, and building host bus adapters.
The main problem is that Fibre Channel requires specialized hardware. As networking bandwidth capacity soared in the late 1990s, and IP networks took root, a new demand emerged to find both a cheaper and simpler way to handle block storage connectivity.
iSCSI (Internet SCSI) was standardized in 2004 to tunnel those SCSI commands over TCP/IP, codified in RFC 3720.
Now for the first time servers could connect to remote block storage arrays using a standard Ethernet NIC or an iSCSI offload engine rather than needing a Fibre Channel Host Bus Adapter.
iSCSI gained widespread adoption particularly among cost-sensitive businesses, and as Ethernet continued to reach new speed ceilings (and with Offload NICs and Jumbo Frames coming into the picture), iSCSI came to significantly narrow the performance gap with FC.
To this day, FC and iSCSI remain common staples in enterprise data centers for handling storage networking.
NVMe and the Solid State Drive
HDD (Hard Disk Drives) have been all the rage for quite some time. They are one of those vestigial tethers back to those early days of mainframe computing because just like the IBM System/360 we discussed above, they depend upon rotating magnetic platters with a read/write actuator arm to access data. Although they may spin at 7,200 RPM they still have a seek time. They are fundamentally mechanical.
SSD (Solid State Drives) are a fundamental paradigm shift in this regard. They are the first quantum leap beyond magnetic platter drives to what we know as flash-based storage.
Flash memory is not a new thing. It was invented in the 1980s by engineers at Toshiba, deriving ideas and patterns from pre-existing EEPROM technology.
Flash memory can be divided into NOR flash and NAND flash (referring to the the NOR and NAND logic gates). NOR provides writing over an erased location at the granularity of the machine word with powerful direct random access at individual bytes. It has faster read speeds than its cousin NAND.
NAND flash memory can read, write, delete blocks through a serial access approach. While it may be less efficient for random access tasks, it introduces considerable cost-effective power for high-density data storage. NAND is the foundation of many flash-based technologies we are familiar with including USB drives, smart phones, and the SSD.
NAND flash SSDs no longer have a physical seek time where an actuator head spins over a magnetic drive like HDDs. Instead reads and writes occur electronically at microsecond-level latencies with significantly higher random I/O.
However, there remained a problem.
While SATA provides a wonderful level of host abstraction for read and write operations using SCSI commands, the standard was designed exclusively for mechanical drives. Consequently SATA SSDs encountered bottlenecks from the protocols used to access them.
The industry solution to this requires a brief rewind to 2003 when PCIe (Peripheral Component Interconnect Express) was introduced as a bus standard to serve as an alternative to SATA for SSDs, before SSDs were driven by flash technology. PCIe offers direct high speed links between the SSD and the CPU, designed from the bottom to maximize SSD efficiency.
This was then conjoined to NVMe (Non-Volatile Memory Express) in 2013 to complete a computer architecture standard which was built to fully leverage the parallelism possible in modern SSDs through physically storing a NVMe controller chip with the storage medium.
From the start, NVMe feature much higher IOPS and lower latency than SATA SSDs. Today’s performance-critical applications depend upon this kind of hardware.
This has even been extended to NVMe-oF (NVMe over Fabrics) which have attempted to apply the concept of direct-attached flash across entire networks. Just as Fibre Chanenl and iSCSI attempted to expand SCSI commands to the network layer, so too does NVMe-oF attempt to expand NVMe commands to the network layer as well.
PCIe and NVMe remain rapidly improving technologies, and it may not be long until HDD’s will only be considered an archival solution, much like tape storage today. Time shall tell what future horizons there are for block storage.
Block Storage and Cloud Architecture
Several considerations can be drawn from this cursory glance through block storage history within the context of cloud professionals.
First, and this goes for data storage in general, the lifespan of an organization’s data assets can often indicate how many protocols it has jumped through, failed to jump through, or been caught up in some weird in-between state.
This is especially the case with many on-premise data solutions or even earlier implementations of cloud block storage.
History and operations can shape organizational data in a number of ways that may not be obvious at first glance. As Peter Drucker notes, you cannot underestimate how long temporary solutions simply live on by default.
It is always important to ask oneself if the decisions that shaped technology or data solutions a certain way were primarily determined by constraints or requirements that no longer remain relevant, or if there is a true ghost in the past that may emerge if you disrupt the "way things have always been done here”.
Both are possible. This requires case-by-case judgment.
Second, interest in block storage does not seem to be going away anytime soon for IT professionals. If there is one consistency we can trace in this historical thread, it is that there has up until now been an intense interest and demand in optimizing and maximizing low-level data operations using block storage. Even as other data storage technologies wax and wane, block storage has been a consistent forerunner.
We can trace this to the fundamental advantages it has offered throughout history:
It can offer high performance on random reads/writes (e.g. OLTP databases, queues)
Because it is not concerned with metadata, block storage can use any filesystem and offers great flexibility for formatting, encryption, and partitioning.
VM operating systems are strongly integrated with block volumes (e.g. AWS EBS, Azure Disk) to parallel standard computing orchestration. Block storage is a universally accepted, deeply integrated method for storage that can be transferred and migrated across a huge breadth of technology solutions (in contrast to much more opinionated storage methods such as key-value stores).
Third, here are some open-ended use case questions I will leave here that cloud professional could use to judge between various block storage solutions:
Are the workload read/writes random or sequential? Is it more read intensive or write intensive? What are the latency requirements?
Does the workload need direct block device access (transactional databases and hypervisors do)?
What are the consistency model and redundancy expectations?
Is concurrent access a need for volumes, or is it acceptable for only one host to access a volume at a time?
What are the capacity, scaling, and cost needs?
What are the HA and Fault Tolerance requirements?
What layers and forms of encryption are required for data at rest?
How does it integrate with pre-existing hardware, systems, or data? How do your block storage choices align with the trajectory of organizational or industry technologies?
What are the retention and migration SLA expectations?
What levels of IT storage admin experience available and what maintenance overhead is acceptable or preferable for the business line?
Through all of this, it truly is a marvel that through the radical changes that have swept through computing, one thing that remains the same is that we are still reading from and writing to blocks at the end of the day.