Transcript

David Chisnall: I'm going to say a lot of words very quickly. If you remember almost none of them, I want you to remember this one thing, that isolation is something that we know how to do. It's easy. We've had MMUs isolating processes, VMs isolating, again, with MMUs, for many decades now. CPUs are really cheap. If you want to have two fully isolated workloads, the cheapest way to do it is you just run them on different computers. The place where this starts to get really difficult is when you have two workloads that are not fully isolated, where they need to communicate. That impacts how you design the hardware, how you design the programming model, and everything that follows from that is the difficult bit. This is a message that I've been trying to drum into CPU vendors for the last decade or so. Every time they come along and say, we have this new isolation technique, and you say, great, what is the programming model for sharing? They say, we can put pages in both trust domains. You say, that's actually not a programming model.

CHERI: What It Is, and What It Isn't

I'm here to talk about CHERI. Of course, the first thing you need to do when you're going to explain what something is, is start by saying what it isn't. You'll hear a lot of people talk about the CHERI ISA. There is no such thing as the CHERI ISA. Think of CHERI in the same way you'd think of something like SIMD. There's no such thing as the SIMD ISA. There's NEON or SVE on ARM. There's SSE or AVX on Intel. These are all localizations to a particular ISA of the same abstract set of ideas that you can operate on vectors of data rather than on scalars. CHERI is exactly the same. The original research prototypes we built were extending MIPS, because MIPS was conveniently 20 years old and out of patent. ARM produced a CHERI extension to Aarch64, called Morello. The one that I will probably talk about too much, because I'm largely to blame for it, is CHERIoT.

We're currently in the final stages, hopefully, of standardizing a RISC-V base called RVY, which is another CHERI localization to a particular ISA. For a lot of the CHERI work, on the software side, it's very much focused on these two extremes. The original research was done on CheriBSD, which is a fork of FreeBSD. A lot of those ideas are gradually being ported back to Linux. This was really focused on incremental adoption. It's looking at application cores, things from mobile phone scale-up to data center scale, things that need to be able to run existing binaries, because people have gigabytes of existing binaries on their machines. If you say, yes, we have a new architecture, it's great. It's secure. All you need to do is throw away all your software and start again. People look at you like you're a crazy person. At the opposite extreme, we have CHERIoT, which is designed for microcontrollers.

If you say to embedded systems people, you need to recompile all your code, they say, ok, it's Tuesday. CHERIoT was designed, in part, to demonstrate that we could scale the ideas down from that mobile phone class right to microcontroller class, but also to show what you could do if, for the entire design of the hardware, software stack, you were able to assume you had CHERI and what impact that made on system design. This is source compatible. Because again, if you say to embedded systems people, "I know you spent the last 20 years working on this C codebase. Trust me, if you rewrite it from scratch, everything will be awesome." Again, your salespeople are very rapidly shown the door.

I'm actually going to start explaining what CHERI is and not what it isn't now. CHERI started with unifying a set of ideas from two quite disparate bits of research literature. On the one hand, capability systems go right back to the '60s as a way of expressing security concepts. A capability is an unforgeable token of authority that you can delegate to other people. Whoever holds it, when they present it, they're authorized to perform some action. They typically have permissions, which you can reduce. You can take a capability and create a simpler one and delegate that further on. On the other side, there's the whole fat pointer idea, where you say, traditionally, pointers are just an address. What if we make them more than an address? What if we add some other metadata on to them? That might be bounds. It might be types. They're still passed around like pointers.

They can be embedded in other data structures. Again, they must be explicitly presented to be dereferenced. CHERI unifies this set of ideas. The very high-level view of CHERI, the 30-second overview, is CHERI teaches the hardware that pointers are actually a thing. In conventional architectures, a pointer is a language-level abstraction. C has some rules for pointers. Rust has some much stricter rules for pointers. Java has some slightly differently strict rules for pointers, and it calls them references, specifically to confuse undergraduates. CHERI says, pointers can't be created from thin air. They carry bounds. They carry permissions. Any instruction that, in a traditional ISA, would have taken an address as a base operand, so loads and stores and jumps, now takes one of these CHERI capabilities, a hardware-enforced and a hardware-protected pointer type. In your conventional system, a pointer is just a number. It identifies a point in memory. You do integer manipulation on it. Now you've created a new pointer. In a CHERI system, it's very similar. You still have that address. It's also carrying around the bounds. You can only use it while it's in bounds.

CHERI Capabilities

Let's look inside one of these things. It's probably not surprising that half of this is the address. On a 32-bit system, the capability is 64 bits, half of which is the address. On a 64-bit system, you still have a 64-bit address, and then 64 bits for all of the metadata. They have some bounds. You might notice that you have an address of the top of an object, the address of the bottom, and the address that you're actually pointing to, and that's three times the size of the address. We don't have space for that. We have to compress it. It turns out there's a lot of redundancy in these things. The top bits of the top and the bottom and somewhere in the middle of an object are all almost always the same. The way we store the bounds is to have a floating-point representation where the displacement to the top and the displacement to the bottom each have their own mantissa, but a shared exponent.

Practically speaking, what that means is the larger an object is, the more strongly aligned its top and its bottom have to be. Practically, you just don't have to care about that most of the time because your memory allocator is already aligning large objects more strongly. It typically has a representation that has like a 3-bit mantissa, and we're giving you a much larger mantissa in hardware. There are also some permissions. Some of them are very simple things like load and store permissions. Some of them are a little bit more complicated. We separate out a permission that says, if you can load or if you can store, you're also allowed to load and store pointers. I can give you a buffer that if I happen to have left any pointers in it, yes, you can't load those. You can just load data. There are also some richer ones.

We can enforce a deep immutability property with these things with a permission with a slightly confusing name, which is permit load mutable. If you have that permission, when you load a capability, you just load it. If you don't have that, when you load a pointer via the pointer that doesn't have this permission, that permission and store permission are stripped off. This means I can hand you a pointer to the root node of a tree with just a couple of permissions removed. Now you can follow all of the pointers within that tree. You can do a complete traversal of that tree. No pointer that you load will have store permission. This means I've given you now a read-only view of that entire tree. In CHERIoT, we also enforce a deep and shallow no-capture property in roughly the same way. Really critically, there's a tag bit which says, this thing really is a pointer.

Or if it's not set, then this thing actually is not a pointer. I'll come back to that. There's also an object type which is used to say, is this a sealed capability? Again, I'll come back to that to explain what that means. Critically, all of these are monotonic. You can take the bounds and you can reduce them. You can take the permissions and remove some of them. You can't go the other way. If you want to go from a pointer to a single field in an array that's had its bounds restricted to that field, and you want to reconstruct a point to the containing array, you have to find a pointer to the containing array from somewhere else and then do a re-derivation.

What does this look like in memory? You have some data and you have some capabilities. Alongside them, and non-addressably, you have one additional bit in physical memory, which is where we store that tag. That's also duplicated in the register file. This is a very simple machine with six registers. Register 6 is a capability that points to that second from the bottom memory location. Let's imagine a very simple CHERI instruction set where we have load and store instructions that can operate on either capabilities or data. Usually, the data ones would be subdivided more, so you'd have byte and half word and word and so on. This is a slide, so we can't have a full ISA up here. That first instruction is loading a capability from offset 0 relative to register 6 into register 2. When we execute that, it's copying the whole capability and the tag bit.

This is really important because this property is the thing that lets us implement things like memcpy that want to do type-oblivious copying. You need to be able to copy some data without knowing whether it's a pointer or not. If you read research literature on systems for retrofitting memory safety to C, a huge number of them say, and yes, we have a compiler pass that goes through and instruments which things contain pointers and where all the pointers are. You read those and you think, you've not compiled any non-trivial C with your tool, have you? Now if we do the opposite operation and we store some data from register 1 over the same location, we're only storing half the width of the register, but because we've overwritten any part of that capability size word in memory, we've implicitly cleared the tag. If you're on a big application set core, this will be an in-cache operation.

If you're on a microcontroller, then that's just another bit in the bus message saying, please clear the tag associated with this. Again, we can load that back with a load capability instruction, and that again propagates that zero bit. These are the two halves of what you need to be able to do memcpy. If we then try and dereference that, even though it's in a capability register, when we try and dereference it, the tag isn't set. The very first check that the hardware is going to do will fail, and it will say, you are trying to use a thing that is not a pointer as a pointer. This is really a thing that prevents a lot of pointer injection. A lot of first or second steps in attack primitives involve overwriting a pointer with something attacker controlled. If that's something you've read from the network or from a file, now it's not a pointer, so you can't dereference it. This gives us a 1-bit dynamic type system where everything is either a pointer or not a pointer. If it's not a pointer, go ahead, use all the remaining bits, do whatever you want with it. If it is a pointer, the hardware is guaranteeing those monotonicity properties.

I said I'd come back to the sealing mechanism. Sealing is a really powerful thing in CHERI. I'll show you some code explaining how you use this from C later on. If your capability in its permission field has permit seal and permit unseal, it's not referring to that linear space of addresses. It's referring to an abstract space of types. That means you can use it as a sealing token alongside this seal operation. Again, this is a register-register CPU operation. All that does is copy the address from the sealing token into the object type field of the thing you're using as a pointer. The thing you get out of that is something that hardware will not let you modify in any way. You try and modify it, the tag bit is cleared. Will not let you dereference in any way. You try and use it for a jump or a load or a store and the hardware will trap.

You can still store it in memory. You can pass it around. When some untrusted code that you've handed this token to gives it back, you can do the converse unsealing operation and turn it back into the original pointer. If they give you absolutely anything else, even if it's a valid pointer or even a valid sealed pointer with a different type, that unseal operation will fail and you'll get an untagged value. This means you can have type safe opaque pointers. All the things where you'd use file descriptors or handles with an indirection table, in CHERI, those can just be direct pointers to the thing. This is really useful when you want to start moving from a world where you have a kernel which manages that table to a fully privilege-separated, least-privilege design where any compartment can vend opaque handles to things, you have socket handles, message queue handles, whatever, to any other compartment.

What does this look like in your memory hierarchy? In the sorts of things that we build, the microcontrollers, it's actually very simple. You have a capability. Your load-store unit will do the bounds of permission checks. They're very fast. They're just single cycle operations that happen in parallel with the rest of the pipeline. If those pass, you just operate on the physical address and it's just like on a non-CHERI system from that point down. If you're on an application core system, though, there's an MMU here and it's doing one or more phases of address translation to take you from that virtual address down to a physical address. These happen after the CHERI checks, and are largely orthogonal and composed very cleanly with CHERI. Similarly, the program counter on a CHERI system is extended to be a full capability. If you're running an unmodified legacy binary, it might have some instructions that will jump to a particular address.

When that happens, you treat the metadata half of the capability as implicit. Those jumps will just be setting the address of the program counter. All of the bounds checks still apply. If you try and jump out of the bounds of the region that you've been told you're allowed to execute code in, that'll be a trap, but otherwise it just looks like legacy code. Similarly, for data, there's an implicit default data capability that loads and stores can be indirected against. On the larger systems, the combination of these two lets you very easily build a programming model that looks very much like WebAssembly, where you say, this is a region of my address space. It has some code and some data in it. You can run arbitrary code in there. You just can't access memory outside of it. Maybe you have a little bit of CHERI-aware code inside there for copying things if you've given it explicit access to them, but otherwise it's just a sandbox.

Adopting CHERI Doesn't Change C

As a C programmer, the most important thing about adopting CHERI is you don't really have to do anything. Here is a traditional buffer overflow where you forget that C arrays index from 0, and so element 14 in a 14-element array is actually one past the end. When you do that, the C standard says that's undefined behavior. The implementation is free to do whatever it wants. On most C implementations, it rubs its hands with glee and says, I'm going to insert arbitrary code execution there. On a CHERI system, that's going to trap. Let's take a look at how that works. If we're on a non-CHERI RISC-V system, that first stack allocation will be lowered to a couple of instructions. The first one is part of your Prolog, and that's just reserve some space on the stack, and it's more than the space we asked for because it also will put the return address and the frame pointer in there.

Then, to take the address of the buffer, we just do some arithmetic on that. Slightly confusingly, it happens that this is adding 14, but it's because of some subtraction and commutativity and things. Now register a0 contains the address of the start of that buffer. It's just an address. When you try and do a store at offset 0 from that somewhere else, maybe in a function that you've passed this pointer to, great, now it's overwriting your return address on the stack. That was definitely what you meant in your C code. In a CHERI system, and this assembly I'm going to show is from CHERIoT, but other CHERI systems have roughly analogous sets of instructions. The first two are exactly the same, except that now we're doing instructions that know that they're manipulating pointers, and so they'll preserve the bounds. If you just do a normal add instruction, typically that will only operate on the low half of the register, and the result will be something that doesn't have a tag and has 0 for the metadata.

We're still reserving some space. We're still doing an add to give us a pointer that's derived now from the stack pointer. It will definitely be within bounds of the stack pointer if you expect to be able to use it. Now there's one more instruction which says, the compiler knows that you wanted this to be a 14-byte thing, so now we're going to set the bounds on this to be 14 bytes. Now the bounds of the capability in register a0 go from that start address that you set it to, up to that plus 14. That means when you try and do a store byte instruction and it's at offset 14, now we're in that one past the end thing that C says is totally defined to have a pointer to, but don't dereference it. Again, it's very similar to the RISC-V code except rather than overwriting your stack pointer or your return address on the stack, now you get a trap deterministically.

This lack of change is how we can do things like running a FreeBSD kernel that's compiled in CHERI mode, is fully memory safe with a Wayland compositor, with KDE and a suite of apps that are all memory safe. I think just this screenshot has about 100 million lines of C and C++ running memory safe to make it work. That was the work of a very small team at a university to do all of that porting. If you imagine the equivalent effort to rewrite all of that code in a memory safe language, that's just at least tens and maybe hundreds of billions of dollars, if not trillions to get to that point.

CHERIoT Temporal Safety

Most of what I've talked about so far is spatial memory safety. Spatial memory safety is very nice to have, but it's also ignoring the elephant in the room, which is temporal memory safety, which is the hard problem. I'm going to talk you through how CHERIoT does this for two reasons. One, I'm incredibly biased. Two, it's actually much simpler than how this works on the bigger CHERI systems. I don't have enough time to fully explain that because it has to deal with more concurrency and with MMU interaction and all sorts of fun things. On the bigger systems, the approach is very similar. The abstract machine that you get as a programmer is roughly the same. Temporal safety. You have some memory and your memory allocator says, this bit of it is an object and so are these bits. Then you get the annoying bit. Now you have pointers between objects, and there's no easy way of saying, I have an object, where are all the pointers to it?

Those pointers can be anywhere in memory, and that's what makes temporal safety annoying. The traditional solution to this problem is garbage collection. You say, I have a few things that I know don't get deallocated, like my stack and my globals. Now I'm going to do a graph walk through all of the pointers, and I'm going to mark all of the ones that I've seen and I'm going to deallocate everything I haven't seen. This means you need some additional state for holding all of that mark state, so tracking all the objects you've seen because, otherwise, if you've got a cycle, you'll just keep going round and round and round and never finish. You also need some other way of saying, where are all the objects I haven't seen? We're not actually trying to change the C abstract machine here. We still have malloc and free. For CHERI revocation, what we're trying to do is say, when you free an object, rather than saying, as with garbage collection, the object doesn't go away until the last pointer goes away.

We're doing the logical dual of that and saying, when the object goes away, all pointers to that also go away. The first building block we have for this is those tag bits. They say, we know where all the pointers are in memory. We can just look at those tag bits, and if it's one, this thing is a pointer. How do we use that to build temporal safety? Say we free that middle object. On CHERIoT in hardware and in software on the bigger systems, we have a little shadow memory that has 1 bit per allocation granule, which for us is per 8 bytes. I think on the larger systems, it's typically 16. We paint all of that with 1's. On CHERIoT, there's a little bit of hardware called the load filter that when you load a capability into a register, calculates the base address and then goes and looks up the corresponding bit in the shadow bitmap.

If it's 1, it clears the tag. It's safe to use the base address for that because if the memory allocator hands you a capability to this chunk of memory, that monotonicity property I talked about earlier means you can move the top in and the base up a little bit, but you can't move them outside. The base will always be within the original bounds. Actually, with the corner case, if you have a length 0 thing, but no one cares about length 0 capabilities leaking because they have length 0, you can't dereference them. That gives you deterministic use-after-free. What it doesn't give you is the ability to reuse memory. That's the important bit. We also have a little hardware state machine that in the background periodically goes and scans all of memory. This starts at the beginning of your memory, scans through, finds a pointer, and then looks up the value in the shadow bitmap, does the same check the load filter does.

If this points to a deallocated object, it clears the tag. Clearing the tag means this isn't a pointer anymore. It can keep scanning through, gets the next one, does that same operation, skips any pointers that point to objects that are still live. Now you have the guarantee that if you freed an object before one of these revocation scans, by the end of the scan it's safe to reuse that memory. If you freed in the middle of the scan, maybe it's safe, maybe it's not. You can't guarantee it, so you defer it to the next one. Again, as a programmer, you malloc some memory, you print the pointer, you free it, you print the pointer again. On a CHERIoT system, we print pointers with all of the metadata fully expanded and actually the bounds expressed twice, because sometimes it's useful to know where the top and the bottom is.

Sometimes it's useful to know what the length is. The only difference between these two is the tag is cleared on the second one. If you try and use that pointer after calling free, now you get a trap. That could be in a different thread or a different copy of that pointer embedded deeply in some other data structure. It's not just the fact that they happen to be right next to each other here in the source code. Those are the building blocks that we have for spatial and temporal memory safety.

How Did We Get Here?

I think it's worth just taking a step back and asking us how we got here. When we started the CHERI project 15 years ago, the real goal was to provide fine-grained compartmentalization. Looking at things like web browsers and seeing they were using process-based isolation, and that doesn't really scale very well because of the MMU and the associative lookups everywhere and needing page tables. It's also not a great programmer model because now you need to build RPC frameworks, you need to serialize data and copy it around everywhere. It scales ok up to a few tens of compartments. Web browsers want to decode every image in a separate isolated sandbox. They want to decode every audio sample. They want to have a privilege-separated JavaScript JIT, where the thing that can write to the memory that the JIT runs and the thing that executes it are separate. What they actually want is thousands to tens of thousands of sandboxes, and just none of the existing techniques work there.

We rapidly discovered that to make any of this usable, we needed to be able to share not just things that are not pages, not just objects even, but object graphs, full data structures. We needed to be able to say this bit of code actually is able to access this structure and all of the things reachable from this structure. That meant we needed to be able to have memory safety for shared objects. We initially went down a little path of a C dialect where you could annotate some pointers and say, ok, this one will be represented as a capability. Now I can share a data structure that embeds these, but most of my code doesn't need to be memory safe. We rapidly discovered that that involved enormous changes to the source code. If we just made everything a capability, the cognitive overhead for the programmer was much lower because now every pointer is the same.

We're not back in that near and far world that anyone who has post-traumatic stress from 8086 programming might have blanked out in their memory. We did a tcpdump port where we just protected the packet buffer and we compartmentalized all of the things that operated on that. That was 1,700 lines of diff. We then rewrote that to be fully memory safe for everything, and that turned out to be a four-line diff. This was much simpler. Now we have memory safety for everything. Then everyone said, memory safety bugs, they're about 70% of security vulnerabilities. You've solved that. Let's get very excited about that, and sort of forgot a little bit about compartmentalization. That was what we were doing it for to start with.

From Memory Safety to Compartmentalization

I just want to talk a little bit about how we go from memory safety as a foundation layer up to compartmentalization, which is how we go from 70% to 80%, 90%, close to 100%. In a CHERI system, we have a very simple property, which is that the set of reachable memory is the transitive closure of the memory reachable from the capabilities in your register file. Some of those might point to PLT or GOT entries from libraries. Some might point to your stack. Some might point to heap objects. You implicitly have the notion that everything reachable from your register file is a protection domain. Within an address space, if you have two threads with different register files, you have two different protection domains. They might be overlapping protection domains. There might be things that are in both. As long as they're not fully containing the same set of routes for that graph of pointers, they remain isolated.

On the big systems on CheriBSD and hopefully soon on CHERI Linux, there's a very simple incremental deployment path for this, where in addition to the traditional Unix execve system call, we have a coexecve, which doesn't create a new address space. It creates a new OS process. Things like file descriptors remain vfork created new ones. execve inherits all of the kernel state and says, now I'm a completely new process. It takes a set of root capabilities to install. Now you can say, this new thread that's running, it's going to have access to maybe even this new binary, and that will be mapped into my address space and will run through the runtime linker again. It also has access to these pointers that I want to share with it. Once you can share one pointer, you can then share any arbitrary complex data structure. This lets you build a little tiny executive in your address space that can implement things like a shared object abstraction, where you can effectively malloc memory that is now shareable between two processes or message passing things, or something that looks like a pipe, but is roughly an order of magnitude or two faster, because it's not doing page pinning or bounce buffering into the kernel, or pivoting MMU translations, or all of the things that add overhead for traditional pipes.

In CHERIoT, though, we took this really to extremes. We have an RTOS, which is really aggressively privilege-separated. It starts with a loader. The loader is the only thing that is omnipotent in the system. It's responsible for the initial setup. If you've got tag-carrying non-volatile memory, which actually we do have on our chips, this is actually much simpler. It's basically just copy some things that you want to mutate out of the non-volatile memory and into SRAM. The really critical bit is the switcher. This is roughly the equivalent of a kernel in a conventional microkernel system. This is the thing that handles context switching between threads and cross-compartment calls. This is about 350 instructions. To put this in perspective, how many people here have heard of the seL4 verified microkernel, and want to guess how many unverified instructions they say they have in their microkernel? They advertise that they have about 200 unverified instructions, which when you consider that their entire microkernel is about 10,000 lines of code, that's a really small amount.

That's probably fine. It's about half as many as we have, and we're not claiming a verified kernel. We actually are working with a couple of groups to verify those 350 instructions. The fact that they're memory safe and they don't have escape hatches that are, go access any arbitrary bit of memory, makes that verification much easier than it would be on a non-CHERI system. We have a scheduler, and the scheduler is built using that sealing mechanism. When we take an interrupt, the switcher will serialize the register file, and it will then hand a sealed capability to that register save area into the scheduler. The switcher is entirely stateless aside from having access to that one additional privileged register. The scheduler has to be trusted for availability by design. It's the thing that chooses which thread runs next, so it can choose to never run your thread if it hates you.

It doesn't see any state of the interrupted threads. It can't see their stack. It can't see their register file. It just hands back a sealed capability to the next thread to run, and then the switcher installs that. Then we've got a heap allocator, and this looks a lot like malloc and free, except it has a quota mechanism, and we can use that for allocating shared objects or objects that are just private to your compartment.

I think at this point, I've probably said the word compartment about 200 times. Maybe now's a good time to say what a compartment is. In the CHERIoT, very high-level view of the world, it's really just two registers. There's the program counter, which gives you access to your code, and the global pointer, which gives you access to all of your mutable globals. If we look inside that code region, we also put read-only data there. That's fine. Crucially, we also have a region that we refer to as the import table. This is the only thing that, at system start, is allowed to contain pointers that aren't derived from that program counter capability or that global pointer. This is really important, and I'll come back to this repeatedly in the next 5 minutes for being able to build auditable systems. Because to initialize that bit of memory, the loader needs to have some metadata saying, you may provision capabilities to these other things into this compartment's import table.

That means the linker has to populate that metadata section, which means the linker knows exactly what write every compartment has that refer to something other than that compartment's private state. That's something we use to build auditing tooling later on. There's also one other memory region which the compartment doesn't have direct access to, which we'll use for the cross-compartment mechanism called the export table. If you look inside the export table, there's the program counter and the global pointer for the compartment, so the two capabilities that refer to those other regions of memory. Then there are a load of entries describing all of the entry points to that compartment, so the offset within the program counter that it should be set to when you do a cross-compartment call to that entry point, whether it runs with interrupt enabled or disabled, how many argument registers it uses so the switcher can zero all the others, and a few other bits of metadata.

Import tables contain pointers to these where the bounds are the full bounds of the table, which means we can look at the base and we can find that program counter and global pointer. The address points to one of those entries, so we're effectively stashing three words worth of state in one pointer. These are sealed capabilities, so the only thing that has access to those, the only thing that's able to unseal them is the switcher because the switcher is the only thing that holds that capability. These aren't the only things that you find in the import tables, you also have just normal pointers that allow you to access MMIO regions. You have those imported functions both from other compartments and also from shared libraries, which are stateless and act as if they've been copied into the compartment that calls them. Also, pre-shared objects, and these can have restricted permissions, so you can say, this is the only compartment that's allowed to write to this shared object, but all of these can read it, so you can do fan-out style communication very easily.

You can also build on top of the sealing mechanism to have a software-defined capability layer for things like, you are authorized to allocate up to 4K of RAM, or you are authorized to hand this to the network stack and it will give you back a socket to this specific remote endpoint. There's also the hash of every section that goes into each of these compartments so that you can then tie that to SBOM infrastructure and say, this compartment has access to this quite sensitive thing and it must be exactly this binary which came from this auditable chain of reproducible build.

What does this look like inside the system? You have those two registers I mentioned, the program counter and the global pointer, and you also have a stack pointer which gives you access to the current thread stack. You have some heap objects maybe, some of them are reachable from global, some from the stack, some only from other heap objects. What you want to be able to do is call in to another entry point in compartment B. Before you invoke the switcher, you maybe decide you want to pass a pointer as an argument to this function. Just like on any other system, you put a pointer to that object in argument register 0. When you invoke the switcher, it's going to use that sealed capability you got from your import table, unseal it, get the new program counter and global pointer and the correct offset from the program counter from the export table entry that you've handed it.

It's going to zero all the other argument registers, which aren't shown on this slide. It's going to truncate the stack to only contain the region that you haven't used yet. It's going to zero that both on call and return. This means now anything that was left on the stack, you don't need to worry about, it zeroed. Implicitly, you've lost access to that previous region of the stack, to the previous compartment's code and globals, and to any other heap objects. That's really important because all sharing is explicit. All non-sharing is implicit. That means there's no other lookaside policy you have to look at and go, how can these two share access to this memory region via some MPU or MMU policy? It's just, no, I passed you a pointer to this object. Now you have access to this object and anything reachable from it. What does this look like in C?

Imagine we've got a very simple key-value store API, takes maybe a null terminated string as a key and some data that it's going to insert into this store. All you need to do to expose this as an entry point to your compartment is annotate it with the compartment name that it's exported from. When you look at the call site, the call site is completely unmodified. When you compile that call site, you'll be passing a cheri-compartment = whatever the name is to each compilation unit. Each compilation unit ends up in exactly one compartment. If this matches the name that you've decorated the prototype with, then it just becomes a direct call. If it doesn't match the name, then we know this is something we need to generate an import table entry for. The compartment will do all of the interworking things. This is roughly analogous to the Windows DLL import, DLL export attributes, except you're saying which DLL you're exporting it from.

You don't need that horrible preprocessor hackery that every single Windows header ends up with. I mentioned sealing. Actually, you can use from C and it's quite nice. We've added a qualifier on pointer types that says this is a sealed thing. If you try and dereference this or do any arithmetic on it, you'll get a compile failure. It's C, so it's not type safe. You can cast this away. If you do, you won't get a compile failure. You'll get a runtime trap. Don't do that. You can also then just pass this to one of the RTOS APIs to say, please unseal it with this key that I have. If it is a thing sealed with that particular sealing key, you get back an unsealed thing of the same pointer type. Now you can operate on it just as if it were any other pointer. If that fails, it'll return null. Then you can handle that just like you'd handle any other kind of error. Building your compartmentalized interfaces around existing software engineering notions of opaque types just works, except now with dynamic enforcement.

Auditing - Who Can Make Network Connections, and Where?

I want to just highlight some of the things we can do with the auditing framework that's built on top of this notion that everything outside your compartment has to come via the import table. You can say, which compartments are allowed to make network connections? We have some tooling that uses Rego. Has anyone used Rego? Imagine you take Prolog and Python and JSON, you put them in a blender, and you forget to put the lid on top, and you run it, Rego is basically what's left in the bottom of the blender. It's a language designed for expressing policies. It's quite useful for this. When you ask, what are all the connection capabilities? It will say the SNTP compartment has the ability to create a UDP socket that it's then allowed to send and receive to pool.ntp.org on the well-known NTP port. Similarly, this came from our Hue the light bulb demo.

The Hue compartment, which controls a colored LED, is allowed to talk to the Mosquitto test server on the well-known TLS MQTT port. You can also start building up higher level things like, I want to say, is unencrypted data allowed to leave the device? The starting point for that is, who is allowed to call the send function in the TCP/IP stack? Hopefully the answer is just the TLS compartment. You can use it in this introspective way to query the shape of the compartment graph. You can then just write policies that are pass-fail things and tie those into CI or into code signing flows.

Summary

To start summing up, porting code to a CHERI system is usually just a recompile, if all you want is the memory safety. When I say, all you want, 70% of security vulnerabilities becoming non-exploitable is quite a big only. Then, to go beyond that to build rich compartmentalized interfaces is usually a very small amount of additional code, 1% to 3% to fully defend those interfaces and be able to do things like say, this compartment crashed, I'm going to restart it from scratch. Then the auditing tooling and what we have on CHERIoT lets you not look at all of the code.

The nice thing about this platform as a programmer is all of your security policy, everything that your code can do is visible in your source code. The annoying thing as a security auditor is I don't want to read all of your source code. I want to be able to say, assume this library that I got from this totally trustworthy North Korean person on GitHub is maybe malicious. What's the worst that can happen? Write policies that say, I am ok with that outcome. We give you memory safety for C and C++ that can be used to build lightweight compartmentalization. Crucially, that's actually with a programmer model that humans can understand because we can share objects. You can then audit over that compartment structure.

Questions and Answers

Participant 1: Given that it's just recompilation of existing code, this question is hopefully fairly interesting. What would the performance impact be of running this? Because I assume you would need a bit more memory and you have a bit more computational overhead. Do you have any insights into that?

David Chisnall: It is the worst question to try and answer because you're never able to do an apples-to-apples comparison. No one is going to design a processor optimized for CHERI, and put the same amount of effort into designing a processor that is not optimized for CHERI, and let you do that side by side comparison. In the limit case, all those bounds checks are fully in parallel with other operations on the load-store pipeline, so they don't really add anything. The overhead from doubling the size of pointers is really workload dependent on some pointers that are less than a percent of total memory. On the other extreme, for desktop workloads, they may be as much as 10% or 20% of total memory. That then impacts cache usage, impacts store buffering, and you want to make sure that those are adequately sized. The only other overhead you get is now you need explicit instructions for setting the bounds.

You can also turn that question around. If I can share memory between two trust domains by handing you a pointer rather than by serializing a data structure, copying it via the kernel, copying it out, deserializing it. How many people here have written code on macOS? macOS has this XPC framework that's used for building RPC between mutually distrusting processors. It copies the data that you're sending seven times, including the kernel copy in and out.

On a CHERI system, you don't need any of those copies. Maybe you need one if you don't trust the sender. You can send an object graph, and you can just walk it and maybe defensively copying it if you need it. We're talking about things that on an unoptimized implementation may be a few percent performance cost in microbenchmarks, leading to a factor of 1,000 speedups on system-level things. One thing I'd love to do, actually, is take some of the CHERIoT stuff and do the same benchmarks on the Ibex core, but running something like FreeRTOS with protected tasks and with defensive copying at all the boundaries. Because I know that if you actually got the same level of security, the overhead would be a factor of 10 higher on the non-CHERI hardware. It would also be way more code to write, because it's just so much harder, and so no one does that.

See more presentations with transcripts