Rust in embedded – should we try this?

…or indeed in any software project?

I got a lot of positive feedback for my talk at last year’s ESE congress, where I basically talked about the state of affairs regarding Rusts current suitability for embedded projects. Given the limited time and the rather broad scope of the talk there was not all that much room for a deep dive.

Earlier this year I was invited to a larger German company to give that talk again. Since there was no tight schedule to follow we had a very interesting discussion afterwards. What kind of struck me was, that company’s lead engineer saying “well, I’m a bit disillusioned now”. How did that happen? Most people would say I’m an avid supporter of Rust in most spaces and yet I failed to convince a leading engineer of a billion euro company, that Rust would be a good choice.

The legacy burden

As it turns out I did not so much as fail to convince people. I actually succeeded in showing them the complexity of their current solution. Thinking about that (in their case: A large codebase that freely mixes C and C++ coupled with a proprietary IDE), they were struggling with the idea of adding the complexity of another language into the mix. Add that their staff is mostly made up of electronics engineers with no formal computer science training. These people already have trouble with modern C++ and modernish approaches to embedded software development. All this doesn’t exactly help the case for Rust – we know it is hard to learn (although one could argue, that modern C++’s intricacies are harder to grasp), even for people with formal training. Forcing Rust’s concepts onto people without formal training will probably cause a lot of pain in the respective organization. Add to this that most companies usually already have substantial codebases that encode a lot of knowledge of their domain and the perspective gets a lot less rosy. Even more so, if one considers the API surface that any Rust code will probably have to interface with. It’s one thing to slap isolated functionality into a static library, tightly integrating it with the whole system is a whole other story. For systems that run a full OS (i.e. embedded Linux) this is probably less of a problem, since the abstractions provided by the OS go some way to enable the use of Rust side-by-side with other languages, hell, one even has the option to isolate the Rust code into it’s own process if necessary. On bare metal systems, the story is different. Usually we only have a single application. The operating system often is an RTOS and is added as a library. The API surface by an RTOS is not trivial, even for more basic RTOS like FreeRTOS. Interfacing with these libraries will require lots of glue code, that needs to be written and maintained, as there are very few offical bindings available (or even inofficial ones, that are acutally maintained). Interestingly at this point, there seems to be no consulting company willing to provide support for these libraries.

And we have not talked about the actual application yet – even an only moderately complex embedded device will run tens of thousands of lines of already existing code, that is – quite often – not written with easy interop in mind. Worse: Depending on the age of the codebase, the amount of cruft and hacks it has accumulated will often interfere with interop, making an already hard task even harder.

Not a true Scotsman

Another thing, that doesn’t help to archieve wider adoption is the outright hostility Rust projects receive on sites like Slashdot. Even with the evidence we have that tells us Rust design really does provide a significant improvement most Rust related posts are showered with an array of negative comments about the language. These range from “any competent programmer is able to write safe code in C, only incompetent ones need a crutch like Rust” style comments to “don’t believe the hype, Rust is just another fad and will go away.”.

I’ve always thought software developers by and large would be reasonable people. However the comments section on any Rust article on Slashdot has proven me wrong on a scale I wouldn’t have imagined possible. We know memory safety is an issue with C, we also know that the skull of your average programmer is not large enough to ever hope to archieve constantly writing safe code, especially when presented with a codebase he/she doesn’t know inside-out. That last bit is noteworthy – a lot of memory related bugs will not be brought into codebases during the initial design, but later when the original designers have moved on, by less experienced people. No matter what people say – the data, in the form of CVE’s keeps proving them wrong. And no amount of moaning about incompetence will help here – we have to deal with the fact that the distribution of most things follow a gauss curve, this counts for technical skills as well, and we should never write code that is only maintainable safely, if we have it done by a “top-5%” developer.

People will usually point out, that there’s an ample source of coding standards and static analysis tools available for C and C++, and thus no need for Rusts rigidness, as we can archieve all that by being disciplined and using the available tools. Spoiler: These tools are usually on the expensive side, and (experience shows!) companies will skimp on the licences, leading to limited use of said tools. Further, these tools are opt-in – you’re free to not use them at all and just code away. What some might consider a feature here, I’d consider a bug. These tools should universally applied, when C/C++ code is developed, but reality tells a different story. At last, these tools are often pretty hard to apply correctly. Anyone who has ever tried to get PC-Lint to work with a multi-plattform build will agree with me here. Add to that the funky licensing models the vendors of static analysis tools often use and you can be all but sure, that these tools will never be as pervasive as they should be. Further: Even when using recent C++ standards to their fullest effect the language will never be able to escape its roots, due to its commitment to interoperability. This means, that some concepts that would allow the kind of memory safety Rust provides out of the box, are just not possible to express in C++, even if we wanted to. And as for “we only need to be disciplined enough” – I’d like to be able to confidently say, that programmers are always disciplined, but alas, they’re human and bound to cut corners when projects start get rough. This ranges from the outright scary “no time for testing, it’s enough if it compiles” to the less scary but still seriously unprofessional “we don’t have time to write tests/do reviews, we’ll do that later” (where “later” is an approximation of “never”). So: I wouldn’t expect discipline to work.

Moving on the the “don’t believe the hype” arguments: Luckily Rust seems to have passed the test of time, with a lot of good news in the past year:

• Rust support in the Linux kernel
• Microsoft rewriting core parts of windows in Rust
• Meta having teams that use Rust exclusively
• First Linux tools being rewritten in Rust (e.g. su/sudo)
• Google has been busy writing Android code since at least 2019, with the number of yearly vulnerabilities reelated to memory safety dropping by more than 60%)

While there was some hype surround Rust in the past, it was – IMHO – an order of magnitude smaller than the hype surrounding Node.js in the early 2010s. The tech itself can’t do much about what people make of it, and some of Rust’s more passionate supporters calls to basically “rewrite everything in Rust!!!11!!” were bound to trigger a lot of people. However: We are starting to see more and more projects implemented in Rust, where the involved companies dedicate significang amounts of money and effort. Rust is here to stay and developers that have not yet reached the very last stretch of their career would be wise to take it seriously.

Problems, problems, problems…

The first parts of this post mostly discussed acceptance problems that limit the appeal of Rust to a wider audience: There are actual technical risks involved as well as “humans being humans”. To answer the question that started this post we’ll somehow have to deal with both issues, as only then there’ll be any hope of succeeding.

We can easily argue the merits of the language, but that alone will not convince people, that have a large codebase to keep up and running to introduce Rust – greenfield projects are a different story and should be much easier in 2023 and onwards. The tooling has grown fairly mature and doesn’t need to hide from what chip vendors put out (coughcough looking at you, TI!). So, how do we get Rust into existing codebases?

I’d argue, that codebases, that have not yet been touched by C++, are probably an easier sell. Rust’s semantics, as they are exposed towards C, are not too far off, of what should seem familiar to C programmers. There’s little magic involved in the interop and a lot fewer footguns than with C++. The learning curve is obviously still there and programmers will have to deal with two languages afterwards, instead of one (but this is something that can be said about mixed C/C++ codebases to some extend). Arguments that can be used to convince stakeholders here would be:

• Large crate eco system, where getting a library to work with a project is as easy as adding it with cargo (compare that to convoluted builds some C libraries have).
• Less need for static analysis in new code, as a lot of what is checked by common checkers, is already part of the compiler and enforced by it.
• Vastly more expressive language that will – in the medium term – increase programmer productivity.
• Comparably large standard library (a lot larger than C’s!)
• Resource use and performance is at-least on par with C.
• Studies by Microsoft have shown, that the number of bugs dramatically decreases (my own anecdotal evidence here is: When a Rust program compiles, there is a realistic chance that it actually does, what it’s supposed to do).

I wouldn’t talk about Rusts guarantees with respect to concurrency. Mostly because I’d expect these to be of limited impact for existing codebases.

For codebases that are already mixed (C and C++) things are a bit different. Rust’s C++ interop is somewhat clumsy (as is that of most languages) so adding Rust to a complex C++ architecture, where calls have to go back and forth across the language boundary (along with objects!), adds a lot of complexity that somehow has to counterbalanced by the benefits of Rust. I’m having a very hard time making a general recommendation here, since the effort required to get this to run will vary wildly, depending on the present architecture. Again, arguments could be:

• Crate eco system
• Static Analysis less important
• Memory safety

Since we’ll potentially have to create glue for a large part of the software architecture to interface with Rust a recommendation here would be to start with an isolated feature that does not need to interface much with the rest of the code. Also use an approach similar to what I described here to get Rust code to play somewhat nicely with the existing class hierarchy.

Also – one of the most important things – don’t start alone. There are a couple of companies out there, that already have a good idea about this issue.

TL;DR

By all means, give it a shot, if you have the chance. Manage the risk involved responsibly and get a development partner aboard, at least for the first couple of months, to help get you started.

Image Credit: Unsplash/CDC