Memory safety is table stakes

The wrapper function print() polls the async_print() function until it returns a value other than PENDING, and then returns the AsyncResult return value alongside the msg parameter in a new enum called PrintResult. This outer enum has two variants: it represents the result of an asynchronously printed a byte array, or synchronously printed null-terminated CString value.

Again, the snippet above seems unproblematic at first. Unfortunately, when paired with the violation of Rust’s invariants on valid values above, it will perform an out-of-bounds memory access. This is due to an optimization called niche filling. When considering the underlying memory representation of the enum PrintResult type (assuming a 32 bit system), it holds the following components:

• a discriminant value indicating the enum’s active variant (4 bytes),
• in case of PrintResult::Async being active, the enum AsyncRes value and a &[u8] slice pointer and length (12 bytes),
• and, in case of PrintResult::Sync being active, a pointer to a null-terminated CString allocated on the heap (4 bytes).

This means that the size of the PrintResult type should be 16 bytes. However, Rust knows that the only values an enum AsyncRes can assume are 0, 1, or 2. Therefore, it can combine this field together with the outer enum’s discriminant value and reduce the overall size of this type to 12 bytes.

Yet, when we break the assumption that the AsyncRes type will only contain values from 0 to 2, the above optimization can cause the program to misbehave: for instance, assuming the call to rand() within the async_print function returned 3, then Rust would store this value as the PrintResult type’s discriminant. However, reading this value back, it would incorrectly assume that the PrintResult::Sync variant is active, and interpret the stored slice pointer as a pointer to a null-terminated C string, potentially reading other out-of-bounds data or experiencing a segmentation fault.

Notably, many existing approaches to safely interact with foreign or untrusted libraries would not prevent the above soundness violation: the out-of-bounds memory accesses occur from within the Rust domain itself! Even if a foreign library was prevented from accessing any of Rust’s memory, the soundness violation could still occur. This is a violation of type safety, which has escalated to a violation of memory safety.

Next to valid values there are many more safety-critical invariants that Rust requires a developer to uphold: for instance, a defining feature of Rust is its concept of aliasing XOR mutability, which disallows any aliased references from being mutated. Similar to the invariants around valid values, these safety properties are difficult to reason about and maintain across complex interactions with foreign code. Thus, instead of considering all individual invariants at any point where Rust interacts with foreign code, we need a systematic approach to reason and maintain them.