This piece looks at the history of Lustre, one of the most widely used parallel file systems in high-performance computing.
What makes it relevant today is not just its role in systems like Frontier, but what its evolution reveals about how infrastructure breaks under scale, how the industry responds, and how new architectures emerge to address those limitations. Lustre solved a problem for its time. But like many foundational systems in this industry, it continues to be pushed far beyond the conditions it was designed for. That pattern, holding onto familiar architectures while workloads evolve, shows up repeatedly. We’re seeing it again today as AI systems begin to expose the limits of the infrastructure beneath them. Originally published on LinkedIn, February 3, 2023.Frontier. The fastest computer on the planet.
Oak Ridge National Lab (ORNL) in Tennessee is home to the fastest computer on the planet.
What does it take to build the FASTEST supercomputer in the world???
Back in 2013, while I was employed by Seagate in its software division, the company owned a product called ClusterStor, an asset that came by acquisition of Xyratex in December of 2013. Seagate purchased Xyratex for its storage testing equipment, storage enclosures, and subsystems, probably not fully understanding the software assets they were acquiring with ClusterStor. Xyratex was spun out of IBM some 30 years ago, and the two main product lines were the ones mentioned above. In 2010, Xyratex did what many hardware OEM manufacturers at the time were doing and purchased a software company called ClusterStor. ClusterStor, founded in 2008 by Peter Braam, was a turnkey storage company selling a Lustre-based appliance. Lustre, a parallel file system, Braam developed in the late 90s at the behest of the Department of Energy. After the Xyratex acquisition, one of the things Seagate’s software division was looking into was just how fragile Lustre had become as data scaled from millions of files and directories to billions of files and directories and what, if anything, could be done to make it more stable. Not to mention the mix of both very large and very small files, which really seemed to bring it down, sometimes literally. Back when it was invented, Lustre was designed for data sets that consisted of very large files. In order to facilitate high-performance access through simultaneous or parallel input/output operations (IOPS) between clients and storage nodes, the data was stored across multiple networked servers, some for storage and some for metadata operations. This is essentially how Lustre and subsequently ClusterStor operated, without going into detail. Seagate had a compelling reason to figure out a way to improve Lustre’s performance: the DOE had been pushing for an exascale (10^18) supercomputer to become available by 2018. Up to this point, Lustre had been the parallel file system of choice for many of the supercomputers in the Top 500, but if the company was going to support the unprecedented variety and velocity of data required to feed the cores of an exascale supercomputer, work would have to be done to eliminate the frailty of Lustre at scale. Seagate determined the best approach to eliminate this frailty in the filesystem was to simply eliminate Lustre entirely – that’s when work began on a new project called Mero, this new solution would be an object storage based system that wouldn’t be affected by the scaling issues that impaired Lustre.Time to Reflect
As I proof-read for clarity, I realized that I had just unloaded a great deal of information on you, the reader. Further, I made references to technologies and companies that require a bit more backstory for you to fully appreciate this journey. Not one who likes to make any assumptions, I decided to pull back and provide a little history of this file system called Lustre, how it came about, the companies and organizations that influenced it, and the challenges we faced with it on this road to the fastest supercomputer in the world. This is a recounting of my own personal experience as well as a result of extensive research into, what I like to call, the “bouncing ball of Lustre”. What this research did for me was connect several dots that previously were not and opened my eyes to a great deal more adjacent history that I will write about in future blogs.A Little History
Before we go any further, a little backstory on Lustre. In 1999, Lustre started out as an academic research project led by a man named Peter Braam who was a member of the computer science faculty at Carnegie Mellon University. To commercialize his research, Peter formed a company called Cluster File Systems, Inc. (CFS) in 2001. CFS developed Lustre, an open-source file system under a Path Forward project funded by the Department of Energy called Accelerated Strategic Computing Initiative, which included both HP and Intel at the time. In fact, the only funding CFS used to launch the company was from the money the DOE awarded Braam to build this file system. Note: Some key individuals who joined Braam when he started CFS were Eric Barton and Andreas Dilger. This will be important later. Just six years later in 2007 Sun Microsystems acquired the assets of CFS including all of its “intellectual property”, but in 2008 Braam left Sun to start a new company called ClusterStor, that had a product by the same name mentioned at the outset of this blog. Note: Burton and Dilger, however, remained at Sun Microsystems. Three years later in January of 2010 Oracle ended up with Lustre through its acquisition of Sun Microsystems that owned all of the trademarks and assets of Lustre. But by December of 2010, just 11 months into ownership of Lustre, Oracle announced it would cease development of Lustre 2.x and place the current release, Lustre 1.8, into maintenance mode. Essentially killing Lustre, or at least that’s what we all thought. Following Oracle’s announcement several new organizations sprung into existence to support Lustre customers and provide development efforts to the open source file sytem. Some of those companies included Whamcloud, Open Scalable File Systems, Inc. (OpenSFS), EUROPEAN Open File Systems (EOFS) and others. By the end of 2010, as you can imagine, most Lustre developers had left Oracle, but Eric Burton must have seen the writing on the wall because in July of 2010 left to join startup Whamcloud as its CTO alongside its founder and CEO, Brent Gorda, with Andreas Dilger joining in January of 2011. But before Oracle announced it was going to cease development of Lustre 2.x in December, Braam’s company, ClusterStor, was acquired by Xyratex in November after just two years in business. (see hi-res image of the infographic here)
2010 was a big year in “Lustre-world”, but there’s more.
During 2011 Whamcloud did a great deal of work on Lustre in an attempt to make it less brittle and by 2012 Intel had acquired Whamcloud. You read that right, Intel, the chip manufacturer just bought a software company focused on a filesystem. 2+2=? Note: Intel’s acquisition came after Whamcloud won the FastForward DOE contract to work on “what’s after POSIX”, but according to Brent Gorda, he could not tell the DOE of the pending acquisition and it seems they were not happy about the surprise. Incidentally, the FastForward DOE project issued an RFP essentially saying they wanted to “figure out what comes after POSIX”, which one could infer to mean, what comes after Lustre. That’s when Barton and team at Whamcloud worked on a new architecture they called DAOS (this may sound familiar to some) and presented it to the DOE, winning the contract in June. Two weeks later they announced they were getting acquired by Intel. Sounds as though the DAOS architecture was an effort to replace Lustre, and it seems by virtue of the FastForward RFP, the United States Department of Energy knew it had to be done. Not convinced? Keep reading.Still following? See where this is going?
Remember, Oracle still owns the intellectual property of Lustre from the Sun acquisition of Cluster File Systems (CFS), but shelved it at the end of 2010. Whamcloud with Barton and Dilger was providing support and development for Lustre, “a la RedHat”, since the file system had always been open source. In February of 2013 Xyratex announced it was acquiring Lustre from Oracle. So now, Xyratex who owned ClusterStor also owned Lustre, the trademark, all of the intellectual property. And to top it off, Peter Braam, the one who started it all in 1999 was on staff through acquisition of his company, ClusterStor. Quite the move, right? Well, Braam didn’t stay long after the acquisition because in April of 2013 he left to start yet another company, which doesn’t seem to be focused on Lustre.And now we have come full circle
This is where we started the blog, 2013, when Seagate acquired Xyratex and, again, all the Lustre assets and trademarks, etc. If this is where the story ended, it would most certainly be quite the whirlwind history lesson for sure, but it isn’t over just yet. Next we move to 2014 when Ken Claffey, formerly with Xyratex and now with Seagate announced they were donating, yes, DONATING, the Lustre.org domain back to the user community. The whole idea was, as Ken said, that he believed it was the right place for the platform to reside and said their philosophy is healthy community, healthy market, healthy business. This transfer or transition was completed in March of 2015 and by April of 2015 many at Seagate, including myself, started to see the writing on the wall. Seagate did not want to be in the business of competing with some of its largest customers like EMC, HPE, IBM, and NetApp. The division where we did much of the work for Mero, for lack of a better way of phrasing it, was falling apart. Many executives were let go or left on their own, while many others sought employment elsewhere as the group became something very different from what its former leadership had envisioned. Meanwhile, around the 2014/2015 timeframe, there was development going on with a new technology Intel and Micron had invented called 3DXpoint, or the Intel branded version known as Optane, that has since been discontinued. You’re probably wondering how this even fits into this “ill-Lustre-ous” history lesson. Believe me, it does and this is where some of the dots were connected for me. Remember the reference to DAOS earlier being presented to the DOE by Whamcloud as an alternative or replacement for presumably Lustre as part of the FastForward RFP? Well, development didn’t cease on this concept after the acquisition because the development of DAOS, which stands for Distributed Asynchronous Object Storage, now with Intel is still under contract with the DOE. Just as a side note, what was strangely familiar to me is that the graphics used to depict DAOS in the early days looked shockingly similar to those of the project at Seagate called Mero, mentioned earlier in this blog. In 2017 Seagate made the decision to shift gears with the ClusterStor business and sell it off to Cray. Just four years after acquiring it through Xyratex, and three years after “donation day”. They sold the “business”, but not all the assets, to Cray. Up to this point, this is probably the only thing that makes 100% sense in this entire post. Cray was the largest OEM of ClusterStor anyway, so this just brought it in-house. The following year, 2018, as things seemed to progress handily with DAOS and one can only assume the Optane technology in a persistent memory form-factor at Intel, the company decided to sell off Whamcloud and most of its assets and people to DDN.Could the file system destined to replace Lustre itself run the risk of being marginalized even before its time?
In the 2019-2020 timeframe, Intel’s DAOS was officially adopted by Argonne National Laboratories for its Aurora Supercomputer project which was projected to exceed 2 exaflops. I even wrote a blog about DAOS in 2020 after hearing about it through a Tech Field Day event during the time I was a delegate. But then just two years after I was introduced DAOS, Intel killed off Optane in 2022, including the persistent memory that was foundational to DAOS. For me it is unclear what is next for Aurora, although Argonne did present at the DAOS User Group at SC22, five months after it was announced Intel had killed Optane, where it revealed the node details still included 16 sticks of Optane PMEM, which makes me wonder about the future, support, parts availability, etc.
Back to Lustre, who owns it now?
