There are many changes in the storage industry happening now. Few people could have predicted that EMC, the market leader and storage giant of the industry could be in a position to be acquired, however that’s exactly what happened towards the end of 2015. In a move that shocked the industry, Joe Tucci and Michael Dell stood on the same stage and announced that Dell (the company) would acquire EMC, the pioneer of the modern storage industry.
How could this happen? What had changed to cause EMC, owner of VMware, RSA, Virtustream (and a host of other storage acquisitions) to be vulnerable enough to think that selling the company was their best option? The answer is that the storage industry is changing dramatically. There are three secular shifts that we can highlight from the last 10 years, which show why EMC and the other major storage vendors are finding it hard to survive in an industry that they once dominated.
Shift 1 – The Shift to Commodity Hardware
The storage designs of 10 years ago were based on the idea of custom hardware. Systems had specifically designed back-end components, including shared backplanes, UPS (uninterruptible power supplies) and service controllers. Today only IBM’s DS8000 series retains the legacy of POWER processors, with every other vendor migrating their products to an x86-based architecture. Where previously it was necessary to create custom ASICs, FGPAs and array components, now everything can be based off x86 designs.
This has happened for two main reasons; processing power has increased to a position where features such as compression, de-duplication and other tasks in the data path can be achieved without the need for custom hardware. At the same time, the hardware itself has become infinitely more reliable, with hard drives and interface adaptors (like HBAs) becoming commodity (off the shelf) components.
Even as the industry moves towards NAND flash as the primary storage medium, the innovations in managing NAND has resulted in solid state drives of ever reducing cost, with increased capacity and reliability.
The move to commodity hardware means two things. First, vendors can’t continue to charge excessive mark-ups for their products. There’s now a level of transparency that means customers can more easily see when vendors are taking advantage. The age of overcharging is over. Second, barriers to entry for getting into the storage industry are much reduced; there’s no massive investment in hardware needed.
Shift 2 – Storage Features are being implemented in Software
As storage solutions move to standardized hardware and become more reliable, the features are being implemented within software, rather than in custom chips. Functions that were once embedded in silicon or in firmware are moving to a higher level of code. This delivers a number of benefits:
- Agility – storage software can be developed, tested and released much more quickly than ever before. The platforms needed to achieve this are cheap and can even be done in virtual environments using VSA (Virtual Storage Appliance) versions of the platform.
- Reliability – standardization on components like network drivers, HBAs and disk drives means storage software will work with the majority of hardware. Of course there will still be edge-case problems, but these become fewer as hardware standardizes more. There are now only two main HDD vendors and a handful of SSD vendors. The market in HBAs and controller cards is also shrinking, with a move to create better standards for interoperability (such as NVMe).
- Cost Efficiency – having technology based on x86 means vendors can use the same code base across multiple products; for example, a compression algorithm can be easily ported from one product to another. Storage as software reduces the delivery time and cost of development.
Software-Defined Storage (SDS) is a genuine part of the IT storage market. SDS doesn’t only have to mean running on commodity hardware; it can be on vendor-supplied equipment. However, SDS has introduced new ways of thinking about storage, like quality of service, automation (REST-API driven provisioning) and policy-based management.
Shift 3 – The Rise of Open Source
With standardization and the move of functionality to software, we’ve seen the emergence of open source storage solutions that follow the path of operating systems based on the Linux kernel. There are projects like Ceph, an object store that support file and block protocol access and SwiftStack, an object storage layer that has developed from OpenStack.
Hardware vendors have released community or capacity restricted versions of their products in order to gain market awareness and adoption of their products. It’s now easier than ever to experiment and test storage platforms before deciding to commit to purchasing an expensive hardware solution.
What’s Next?
Although we’ve highlighted only three changes here, the industry is also changing in other ways. The rise of cloud storage has introduced the idea of paying for storage based on capacity and time – otherwise known as consumption-based charging. Within the corporate data center, storage has traditionally been an overhead, been paid for by a project or had costs recouped via chargeback. Consumption-based billing and charge-back options are becoming practical and companies are now able to charge departments based on actual usage, rather than a simple percentage of the CapEx cost.
The future will see storage moving to be a chargeable commodity that is consumed and paid for on demand. The legacy IT vendors are struggling to adapt, as their business model (sales teams, revenue recognition) are all based around hardware and the previously good margins they delivered. However, for new companies that have already adapted, the future of storage still continues to be very good indeed.
To learn more about the shift to software, download our white paper on Software-Defined Storage.
