Transcript

Murat Demirbas: My name is Murat Demirbas. I have been working on distributed systems for more than 25-plus years. The last five years I have been at AWS and now at MongoDB Research. My research areas are distributed coordination protocols, cloud systems, cloud databases, database systems, and lightweight formal methods. I maintain a very popular blog at muratbuffalo.blogspot. We are going to talk about disaggregated systems, what they mean. Economics drive the decisions for cloud architectures. The cloud economics demands that we decouple compute and storage. The reason is that compute and storage have inherent impedance mismatch. What does that mean? Compute is costly. Storage is cheap. Compute demand fluctuates quickly. Storage demand is stable and growing. Compute can be stateless. Whereas, inherently, storage is stateful. They don't belong together. What we do when we put them in one box and tightly couple them, if we need more storage, we end up having to sell more compute with it, making the customer pay more, or vice versa.

This is against separation of concerns principle. We like compute to scale elastically as storage stays cheap. As Jeff Bezos famously said, your margin is my opportunity. There is a margin here. The cloud is all about cost reduction. Actually, the cloud had embarked on this. Right now, the cloud architectures, data center architectures are disaggregated data centers. This was made possible by improvements in high-speed networks. Right now, hundreds of gigabytes per second bandwidth is the norm. This is up by an order of magnitude from 10 years or 15 years. We have also more advanced networking technologies like RDMA, SmartNICs, and CXL, all working for this.

Benefits of Disaggregation

What are the benefits of disaggregation? With disaggregation, we can make the compute scale up. We can have a better box, which helps for latency. Or after we scale this up, we could also scale out, horizontal scalability, so we can have more bandwidth. We can serve more requests in parallel through sharded processing. As important as that, we can scale down to zero. We scale the compute to zero. This is very important for customers. Why? Because they want to have pay-per-use. The customer value proposition is there, pay-per-use. The rest is just history and details. The technical side is going to work on making this happen. Another cost benefit of the disaggregation is I/O pooling at the storage tier. Now we could have multi-tenant storage and pool them. We say from both storage, make, utilize it, and network better this way. It does not get as much attention coverage as the elastic scalability benefit, but the operational benefits of disaggregation is as important. Disaggregation gives you fault isolation. When a node crashes, you don't lose both compute and storage node. You lose one. Compute nodes are easy to stand up over shared storage. You have fast recovery, simplified operations. Disaggregation turns databases from heavy stateful boxes they were once, into lightweight elastic services running over shared storage.

Context

This is a good point to introduce my high-tech setup, Org Beamer. I'm still the only one left that compiles their presentations. This is compiled by LaTeX. The nice thing I like about this template is that you can follow my progress on the top bar here. Next, we are going to talk about the architecture part. I'm going to introduce the disaggregated architecture, give you a taste of them from three sample applications, architectures. You are going to understand it. Then I'm going to do something very different. I'm going to take five minutes to convince you that Paxos did it before, disaggregation. How is Paxos related to disaggregated database systems? I will convince you, trust me, give me five minutes and come with an open mind. Then, this Paxos connection is also going to make the tradeoffs more clear. Then we are going to discuss the tradeoffs here and the opportunities, the methods to address these limitations in some part, the new challenges here. Then I'm going to talk to you about the future and ongoing work, which are not as mature as the things that I will talk about in the architecture section. These are very exciting opportunities. Finally, we are going to land this plane with the conclusion section.

The Architecture

This is the architecture track and we are discussing architectures. The database architectures went through three big phase shifts. Initially it was monolithic architecture. This is your dad's database. This has been around since 1970s. One process, one disk, local I/O. When I say monolithic, Postgres, MySQL, these are monolithic. Think about these things. Then we have the cloud. To get the databases into cloud, what we did is, for availability and durability, we slapped Paxos, Raft on them. We have Raft replicated groups of this monolithic database. One becomes primary, Raft ensured no split-brain, good coordination. The followers are following, and they are serving the reads. If you look at the architecture, this is the monolithic architecture. The primary, the compute and the storage, they are still in one box. We just had three boxes, and we basically said that this Raft group is a virtually infallible node. For horizontal scalability, we had more Raft groups, and we did two-phase commit on top of them for distributed databases.

It's a hack, it works. What is the problem? We didn't solve the disaggregation problem. We still have the problems here. This is not good for economics, and everything is about economics. As you get old, you understand this better. The thing is the cloud already does storage replication. If your storage here uses EBS, the cloud already replicates it three times. You have nine times replication of storage and three times replication of transactions. This is not a good deal. You could also do NVMes, but those are also costly. The next shift in the architecture is the disaggregated architecture. Here you can see the ephemeral compute now disaggregated from the storage. We even disaggregated storage into Log Stores and Page Store, which I'm going to talk to you about soon. We have another compute. We don't have to have three computes. We can just have primary and a failover secondary that follows closely and ready to take over. We didn't have to automatically go to three-way, because we need three-way durability. We need three-way availability. You can see the first benefit of disaggregation clearly from here.

Before we dive more into disaggregated database architectures, I want to make this point clear. Disaggregated databases are already here. Amazon Aurora, 2015. Alibaba PolarDB, 2018. Microsoft Azure Socrates, 2019. Huawei TaurusDB, 2020. Google AlloyDB, 2022. Disaggregated RocksDB from Meta as well as Rockset, around 2022 again. Neon, which got acquired by Databricks, 2024. Rockset was acquired by OpenAI also. Amazon DSQL, which I also worked on for two years, got general availability 2025. They share domain themes, shared storage, stateless compute, the network replaces the I/O. This is critical right now for the compute to talk to storage, you go through the network. This shapes all the design decisions as we will see next. For durability, we have log replication again. This is maybe short-term durability. For long-term durability, you could have snapshot at S3, or your favorite object store. Finally, for availability, Page Store to serve as buffers from the compute nodes. The idea is that data center is the new computer. It's a giant computer, and the network layer is the backplane of this giant computer.

Aurora. Network is important. We don't want to cross the network too many times. We don't want to bottleneck the network more than it is a bottleneck. Aurora's main innovation was to use the log is the database idea. Later architectures mostly followed this idea. This started with MySQL, the redo log of MySQL from the primary compute, you recognize these boxes now, is sent to storage first. When a write comes, the write is not acknowledged back to the client until the primary, the primary is always a read-write node, and you could have secondaries as many as your heart desires as read-only nodes. The primary before replying the write as acknowledged, it replicates it to a quorum of storage nodes. Aurora selects this quorum as four of six to tolerate f = 2. After four of six replies, the primary replies back to the client saying that it's done.

Reads, the primary can serve it. Replicas can also serve reads because the primary also sends the redo log to the replicas, and replicas, these secondaries, followers, opportunistically materializes things from this redo log and they could serve buffers. If they don't have it, they go to storage. This is great. We are just sending the redo log, not the whole state, and the storage materializes the redo log. Wait a minute, how does the storage materialize the redo log? Does the storage have CPU? Yes, the storage has CPU but it's limited. The compute-heavy nodes, the compute nodes have more CPU. They also have some storage but it is not mounted drives like EBS, you don't need them. It's opportunistic. The storage also has CPUs but not as big as the compute nodes or they don't elastically scale as that. Think of this as more opportunistic. The first challenge is done there.

The second challenge is, how does the storage know how to materialize this? Aurora did a surgery on MySQL, later Postgres, they put half of that materialization logic, the storage logic, they put on the storage nodes. The storage nodes, when they get a redo log, they know how to generate these buffers. This is not ideal because every time a new version of a database comes, you have to tend to those wounds, stitch them up, do some operations. This is not ideal. This lets us to just use redo logs, and it's a good invention to be network cognizant.

My next sample from architecture is PolarDB from Alibaba. They don't go through this surgery. Instead, they have MySQL and InnoDB both at the compute. They still send a redo log, orange is redo log, to ROs, read-only nodes. Since these guys also have the InnoDB, they can materialize state and follow and serve reads, good. Since the storage nodes don't have the InnoDB, the storage engine, now the primary needs to push the entire state to the storage nodes. It not only pushed the WAL logs, the redo logs, it also needs to push the state pages for availability, because this guy, if it doesn't have it in its buffer, it will go read it from the storage node. Alibaba has developed PolarFS, it's a distributed file system. Alibaba also just throws a lot of money into hardware, so they have RDMA, so they push these things through RDMA to the storage nodes.

One invention here is parallel Raft. Raft serializes everything, which is not strictly necessary. These guys are saying that if they are not operating on the same keys, you don't need to serialize that way. You can have a parallel version of Raft. They talked about it. Again, read-only nodes serve reads from memory if available, else they use storage to get the pages. You do WAL log replication for durability, they have a quorum of two over three. Then you also serve pages from storage. The PolarFS layer is oblivious to what it is replicating, it's opaque to database internals.

TaurusDB takes the Aurora lessons a little bit further. This architecture is close to Socrates, but simpler, less moving parts. What they do is, again, we have a primary read-write node, we have follower read-only nodes. The primary sends to Log Store. The innovation in TaurusDB is split storage into Log Store and Page Store, and prioritize the log. This even reflects in how they replicate the log. They have custom replication for Log Stores. This is I/O scattered over storage nodes. The primary says that, ok, three nodes, replicate this, tell me within a short time, if I don't get one, if one of you lags, this is not a quorum, I'm going to invalidate this. I'm going to choose another three nodes to replicate. We are in a hurry not to fix things, and they keep going that way. I'm ok with it, because they have a time budget and they go to the next node, invalidate and fix things.

We use quorums to get rid of tail latency, but this also works. They scatter I/O to other things. After the log is replicated, the write is acknowledged, the log is also sent to Page Store to materialize the pages. They also have this storage engine there to materialize. Log Store also sends the log to the RO, read-only nodes, for them to materialize and serve reads. I'll stop here for the architecture examples, but you get the main idea. Primary compute, that's read and write, followers are read-only. We generally go with log is the database approach. We also need Page Store to be able to serve buffers as needed by the nodes.

Paxos

What does this have to do with Paxos? I have a bold claim, I'm saying, did Lamport invent disaggregation before it was cool? How many people know Lamport? How many people know Paxos? How many people know Raft? When Lamport described Paxos, he described Paxos in terms of roles. The client sends the request to a proposer, proposer talks to acceptors. After a quorum of acceptors say, are you guys seeing similarities? It sends it to the learners for materializing the state. Acceptors for durability, replicas, learners for availability, materializing the state. In Lamport's formulation, the learners send the responses. What happened? Let me first go through this, just to make sure we are on the same page. Proposers generate order values, they are leaders. This corresponds to compute, generate order values. Acceptors persist votes, the log. They persist to log before we go to Page Stores. This is durability, short-term durability, at least.

These correspond to the log servers in the disaggregated architecture. Learners serve state, availability, they serve the reads. These correspond to the page servers. How many people had seen these roles before? What happened is when we go to Multi-Paxos, in that paper, Lamport says that I'm explaining this in terms of these three distinct roles, but if needed for efficiency, you compact all the roles into one node. That's what we did with Multi-Paxos and Raft, because we were going for the shared-nothing architecture. We all put them all, then the primary node sends to two followers, which are also acceptors and learners. It's also a learner, and it's replica. It doesn't have to be that way. Basically, this is the recipe for disaggregated system here. We don't need to go very recent. This has been around.

What do we learn? What are the benefits? There is an analogy, so what? Who cares? What do we learn from this? Do we learn something from this? Yes, in this slide, proposers, leaders just have to be f + 1 to be able to tolerate f failures. In Raft, we are making it 2f + 1, so that's redundant, for example. Just f + 1 leaders is enough. Acceptors, 2f + 1 for quorum and not having split-brain, good. Learners, just f + 1 is enough. There had been two decades of research on Paxos, and I'm also guilty on this, I have been working on these Paxos roles, and using flexible quorums thing to improve the throughput and ways of reading from replicas. We can borrow from that research, is all what I'm saying. The other thing about disaggregated architectures we discussed so far, they always assumed an external configurator, consensus, like primary node.

Then somebody says that now the secondary is the primary node, where does this come? Outside of the system. That's not good. A better design principle is ingrained fault tolerance. Be self-contained, don't depend on another external service, because every dependency is a problem. I'm saying that we could have inbuilt Paxos, the consensus also in the disaggregated database services. This should be happening, if it's not happening. Raft squished them all for shared-nothing architecture. The architecture evolved. We need to revisit these things. There was a branch of Paxos that did not do it, in addition to my branch, what the academics worked on. It's the shared-log abstraction. Mahesh Balakrishnan worked on this at Meta, and also at Kafka, Confluent. The database is maintained over a virtual log composed of loglets. This is a beautiful abstraction. Virtual log maps logical addresses to physical loglets of finite size, via versioned metastore.

There is some dependence on the configurator. Fine, I'll live with it. The benefit is that you could switch on the fly the loglets, the acceptor configuration as well, when you go from one loglet to another. He famously did it for getting meta control planes from ZooKeeper to Raft-based, or Multi-Paxos based consensus on the fly. This is amazing. Again, two things are squished together. The database now squishes the proposers and learners. The acceptors are here, you do I/O scatter over them, and the shared log advances through fixed-size loglets.

Now I'm going to, of course, talk about my branch of work. This is now familiar to you, this is the disaggregated roles of Lamport, but there are extra layers. Now proposers, you don't go to acceptors, there is a proxy leader layer. We published this with Berkeley folks as a collaboration, we called it compartmentalized Paxos. We could have as well called it disaggregated Paxos. We further disaggregated the leader into two parts, proposers and proxy leaders. Now the leader just orders, proxies handle the I/O, because I/O fan-out and fan-in, it kills the leader. We don't want that. We could even disaggregate further to have proxy leaders. The acceptors, we disaggregated them. We could do, not through a metastore, we could do through the protocol itself, through flexible quorum result. I'm not going to go into the detail of the flexible quorum result, but we could do write quorum and read quorum for leader maintenance, not split-brain purposes.

We could effectively go from each decision to the next, not from each loglet to the next, more fine granularity, scatter I/O over acceptors. This is phenomenal. We could show just over the same codebase, multi-Paxos throughput go eight times. This is the power of specialization. Initially, leaders were the bottleneck. We have proxy leaders, that's solved. If you need more reads, add more learners to your architecture, that's solved, eight times throughput. What gives, what do you notice? Too many hops. What about latency? This also brings us closer to the tradeoff discussion.

Tradeoffs

We are going into the tradeoff section. We are losing something, the network. We cross the network. If we further disaggregate, we are crossing the network even more. The network is the new bottleneck. Disaggregation shifts bottleneck from CPU disk to the network. There are inherent limitations here. I'm going to talk about ways we address them, the methods. Remote I/O is three times slower latency and four times less bandwidth than local SSDs. In the next two slides, I'm going to show you methods to address this. You should do your own cost-benefit analysis. The costs are clear, network latency, some bandwidth cost, I/O amplification. The benefits we discussed, to remind you, elastic scalability, now you can pool compute, storage. Memory, which I will talk in the future part. We could also do disaggregation at the memory side. Fault isolation and failover. Based on this, you could do it.

Now let's look at the latency challenge and the mitigations. For log commit on the storage side, shared storage, we need to cross the network, so this is there. For page fetches, if I don't have it locally, I need to cross the network and get it. There is some synchronization and cache coherence costs that are also prone to this latency thing. What are the methods? I'm not going to explain them, but buffers help. We have seen read-only nodes. If they have big buffers, they also materialize pages, they could serve. When you don't have the page, you need to go and get it. Prefetching helps. Before, you're looking at the access pattern and prefetching these things so that you don't have to go fetch it, that reduces latency. Pipeline, I think I/O OS-level things, the thread-level things you could do, tricks you could pull here helps. Use faster fabrics.

RDMA is now more commonplace. CXL is becoming a thing. Use custom distributed coordination protocols. One nice example is with two-phase commit, it has two phases, two-phase commit, you have a prepare and commit. You could do the prepare going one way, sequentially, and you could do the commit coming that way. In one round trip, you could finish. There are tricks where we can change the underlying communication layout, topology to improve things. Throughput challenges and mitigations. Log contention, log and data pages contention is a thing. Log to log contention is a thing. We have several techniques already in these disaggregated architectures. Also, if you go back, the challenges are not new. The Paxos analogy also shows that. We have principles, the same principles apply. Log-as-database cut data movement by 2.5x. Aurora did a really nice thing by doing that, cut the traffic. The TaurusDB lesson is that prioritize log traffic over page I/O that could come asynchronously. Even in log traffic, you could do smart filter replay, discuss things further. We could do scatter I/O so that we don't have head-of-line blocking on the same storage nodes, acceptors, and we could do pushdown compute to data.

Future

That brings us to the future section. Pushdown computation to data, in this future section, this doesn't mean these are not there. This means that these are not as mature or we don't have as many examples. Taurus Near Data Processing already shows an example of pushdown computation to data. Amazon Redshift also has examples of this. Snowflake have examples of this. We need more of this. The Taurus NDP case, to compute nodes for a scanned query, it converts pushdown predicates, filter, projection, aggregates into LLVM bytecode and sends them to Page Stores. This is on a best effort basis on the Page Store. Nodes may skip this under load and send the whole pages if they don't have enough CPU to process this. If they do, they JIT compile this, execute this code to drop rows, trim columns, and emit partial aggregates early, so instead of sending whole scanned range of pages, you send a lot less pages. Instead of the whole range, you already do the predicate computation at storage node because they had CPU, and send less network. Win-win. Utilize things. Jeff Bezos loves this.

Second thing, we talked about storage compute disaggregation, and said that, you don't have to tie storage to the compute, but the compute is still tied to the memory. What if we didn't tie memory to the compute? Because right now, if we need more memory, we get more compute, we buy more compute. If we untie them, we could also provision them separately. This is a surprising statistic, 50% of DRAM in data centers is wasted by the static provisioning. If I'm not using my RAM, if you could use it to access some of my pages, maybe if we are working on the same database worker nodes, win-win. Luckily, RDMA, and there is an example of this, PolarDB Serverless says they did memory disaggregation, shared memory, but this is not commonplace yet. People are still debating the cost-benefit analysis here, so that's why it's in the future work.

Remote memory, because RDMA is not as mature, there are problems. Remote memory via RDMA enables compute to access shared elastic buffer code. Why is this important? This is important for cutting latency and improving throughput. Instead of going to the storage, I go to another compute node's RAM, and I get what I needed there. I'm elastically increasing the size of my memory pool. This is utilization, the same ideas. Recently, we have CXL. It's a newer networking technology. CXL fabric basically makes the tricks to CPU to treat remote memory as local memory. CPU thinks this is local memory. The Log Store semantics work there. This achieves six times lower latency than RDMA. This is phenomenal. What is the catch? The catch is now you are limited to a rack. With RDMA, you could do data center-level fetches, but with this, you have to be staying in the same rack. There is also the specialization allows different ways of doing things. We could do the log, we prioritize them. Oracle Exadata uses the more expensive persistent memory to ensure durability of redo logs. That's another smart idea.

We talked about disaggregating things, but this disaggregation also enables us to unify things where they count. There is this code, if you are disciplined, it enabled you to be more flexible. If you are more disciplined and have this shared storage layer, on top of this, you can unify things. Google AlloyDB shows a nice example of this. It makes sense to unify OLAP and OLTP to be able to serve the analytics on the most recent, most fresh data. Instead of having a separate OLAP cluster, if both OLAP and the OLTP work on the shared storage node, this is easier to do. Again, you recognize this architecture by now, primary read-write node. Read pool nodes, these are read only. There are now both columnar engine that could even use vectorized processing, and row engine on the same compute node, so it can serve hybrid transaction analysis and processing. Again, database storage engine on top of a distributed file system like PolarFS. In this case, it's Google's Colossus.

Another interesting point, this is disaggregated RocksDB through Rockset architecture example. Here, this shows a more ecosystem-y disaggregation. This is the ALT architecture, Aggregators, Leaf nodes, Tailers. The disaggregated RocksDB lives here, both compute nodes and overshared storage nodes. The writes and reads boundary, the writes come from this side, reads come from this side. Tailers tail Kafka logs and get the real-time logs. Leaves build columnar inverted indexes from tailers. Aggregators run read-only queries, distributed SQL over indexed shards. Disaggregation beautifully isolates bursting ingest from indexing and querying, bursting querying. We have a disaggregated architecture. The heavy lifting here is done by LSM trees. So far, we talked about B-trees. LSM trees are beautiful things because they send logs as immutable SSTs. This immutability enables them to share safely so that compaction can be done on any node, whereas in B-tree, it required random remote writes and page coupling.

This is not a problem with LSM tree architectures. We need more work on disaggregation on LSM tree. There is ongoing work. People are using LSM trees more and more for disaggregation. Since I have talked about, it's easier for any node to be able to compact, this paves the way to serverless. For a truly serverless database, you need a disaggregated database first. Disaggregation, because stateless compute, the serverless idea, stateless compute depends on shared storage. Remote pages and logs let workers appear and vanish safely. Any worker can serve any shard, fast failover, low cold-start.

Conclusion

Open questions. What is the trend? The trend is we disaggregate. Can we take this further? Can we have self-assembling database management systems? Can a database management system auto-configure compute, memory, and storage as workloads shift. Can we have auto-tuning systems? We are going to need them for dealing with agentic workloads. They have bursts that are 30 times worse than the bursts we see from normal traffic. When an army of agents start hitting your databases, how are you going to deal with them? This is an open question. The second thing, a benefit from all the work on AI is new hardware is coming. With disaggregated architecture, we have the gift of specialization. How do the databases exploit this new hardware? I recently have seen a GPU-based version of DuckDB. That's, for example, an interesting approach. As a distributed systems person, this disaggregation shapes how we do coordination.

It also shapes failure modes. In the shared-nothing, we had private logs. Most of distributed protocols are designed for that. With shared storage, it replaces private logs. The decisions are not necessarily done by messages or messages talking to other nodes. Now, they could be log-mediated agreement. For example, we could have a commit or consensus by just compare and swap in a shared log. MemoryDB gives an example on this. Leader election, just make sure you know the last number, switch it by writing it to the log, and then you also fence off the other leaders. You could have leader election by just writing to the log this way, for example. You could also stand new nodes by replaying for learning and recovery. How does this shape? What are the new distributed coordination protocols for this new model, new architecture? I lied to you a little bit. I said that disaggregation makes fault isolation because you don't lose both the compute and storage.

Those are true, but this works for maybe master single file spell, but this also opens up another front. Only the paranoid survive in this distributed systems business. We should be thinking about the downside. The downside is, we don't lose both the compute and the storage, but we lose some fate-sharing. The more components, the more moving parts, the more we are trailing the information, front from behind. There are delays in learning things. These delays, they accept maybe failures to propagate across layers. This may open you to metastable failures, which is a new area of research. Especially when we talk about this dynamic composition of systems, how can we make sure they are correct? How can we vet them for metastability? Simulations seem to be a good approach here, but we are still doing the science of metastability.

Learnings

The takeaway. The future is this fabric-aware databases that self-assemble microservices. We specialized the database, disaggregated into services, and we assembled them into a database which treats the data center as a giant computer. The challenges are not new. Through the Paxos examples, I hope I conveyed to you that these are the classical distributed systems challenges. We have new opportunities through this hardware, through this log shared storage architecture. As the famous industrialist Emerson said, "There are many methods but few principles. If you master the principles, you can pick and choose your methods, but if you try to do the methods without principles, you are going to have trouble." The idea is, first get the principles right and then choose the methods. I'm excited about this new architecture. We know the challenges and we have new opportunities to address these challenges.

See more presentations with transcripts