Hi everyone, it's Calandiel.

I'd like to talk about some of the work we've done this month. As some of you may know, we decided to push back the release of 0.3 and deal with some long standing issues with the codebase that we originally wanted to tackle only after the update. Since that work consists largely of bug-fixing, optimizing data structures and other things that don't lend themselves to pretty images, I thought it'd be good to explain why we're doing this and what exactly we're changing.

First, though, a short history lesson. When SotE originally started, I was still in college and Demian was still working on M&T (a mod for Europa Universalis. There was also a third person making significant contributions to the project and a few other people helping with things like art, game design or scripting. At that point, nobody knew if SotE was going to get anywhere so getting *anything* up and running was the priority. This lead to some decisions that, in hindsight, were suboptimal. Unity3D is great if you absolutely need to get a product out. It's also great if not everyone on the team knows what they're doing quite yet. Unfortunately, it also has severe scaling issues, which only got worse over the years. I wasn't aware of them, but today, in 2022, it takes a whooping 5 seconds for the engine to update its asset databases whenever I make any change to it, even if it's as small as moving a button 10 pixels to the right.

That alone would be a strong incentive to change things up, but it's not even the only issue. Some antiviruses give SotE as a false positive, due to the way our loader works (which itself exists only to accommodate the way memory is laid out in the tectonic code), we can't support platforms other than Windows (which is due to Unity crashing when we try to build for other platforms), we can't use optimal sorting algorithms (due to Unity's Burst compiler being too immature to correctly do branch elimination in some situations) and there's all sorts of glue code tying our tectonic code, Unity, and the rest of the simulation together. It's not *all* bad and I'm overall happy with quality of our code but it's definitely enough to make me stop and deal with these problems first.

So, how far along is this whole bug-fixing process? I'd say we're about halfway done.  We started by fixing rivers and moved onto removing Unity as a dependency. Over the years, its role became less and less important and now it's  not much more than a glorified build system with an asset manager and a visual UI designer. Definitely *not* enough to justify the amount of wrenches it throws down our feet. The replacement engine is hand crafted and has support for 64-bit entity positions (removing the need for floating origin points in the simulation). It also supports exporting worlds to human readable file formats (.ron) and to file format compatible with other languages (Python, through .pickle). That alone will be a great help as previously we had to add exporters ad hoc, whenever we wanted to do some complex plotting or statistical analysis of programs output.

When it comes to code bloat from gluing everything together, things are much better too. Here's a small part of the file that was responsible for generating a lot of that glue code:

Note the line number at the bottom. It was a pain to work with and required modifications by hand whenever we wanted to expose a new collection  type to it. Now, those 12 hundred lines are replaced with a simple 

#[derive(Debug, Serialize, Deserialize, Hash)]

above entity type declarations. And that's before taking into account all of the code that these 12 hundred lines generated. C#,  especially the HPC# dialect that the Burst compiler exposes, has very poor language support for the kinds of problems we're solving. While the language is pleasant to use by itself, it really falls short when you don't have access to objects, reflection, most of the standard library and to top all of that, you need to interop with C++ while also dealing with Unity's little quirks.

And speaking of which, SotE doesn't require its awkward loader anymore. This should stop the game from showing up as a false positive in peoples antivirus software. It also makes it possible to run SotE on Linux, which I immediately used to get 2 gigabytes of free RAM when working on the game (it always blows my mind how bloated Windows' development environment is). I'll be testing Mac support some time next month as well and I'm rather optimistic about it working out. I'll keep you guys updated in the next dev diary.

Lastly, the language to which we're porting tectonic (and engine) code has built in support for detecting many kinds of errors at compile time. For those interested, it's Rust, though, I imagine the previous statement was enough to infer that. The bugs in tectonic code were of the use after free and dangling pointer kinds, both of which are checked for by Rust's borrow checker, so going forward we're less likely to run into another such roadblock.

Anyhow, that's it for today. Next month, I'll be finishing up the rewrites started this month and then get back on track with the gameplay code for 0.3.

See you guys next time,

Cal