Rewrite

The v0.7.0 version is a complete rebuild and a partial rewrite of the engine, on both the core and modded sides.

Let’s begin with a bit of a background on why this change seemed necessary. As I was closing in on releasing the previous version, I thought that it would make sense to test whether things really worked in a networked environment. While I already knew that things worked in a purely technical sense, I soon realized that the protocol of port usage is unpractical in real-world scenarios.

More specifically, every time a client has authenticated, the server would tell the client which port it will start listening on for a connection from that client. This port could be any from the ephemeral port range of 1024-65535. That means whatever environment a server administrator is working with, they have to be able to open all of these ports, and many real-world scenarios don’t really allow for that. For me, the reason was my router only being able to add a maximum port range of 100 in a port forward rule, and managing ~640 rules through the buggy abomination that is its web interface strikes a few points too high on my pain scale.

The problem I ran into was that I couldn’t figure out how to create the usual star topology on a single port using the abstractions of nanomsg-next-gen, the networking library I used at the time. The documentation was too terse, the interface seemed like it was working against my use case (even though it was supposed to be generic), and after finding an open issue where even the creators acknowledge this topology with free pairwise communication is a bit difficult, I knew it was time to move on to greener pastures. It would have been possible to tack on some solution around pooling a specific port range, but at that point I was already carried away by all the benefits that a rewrite usually brings.

In the end, I rebuilt the engine from the ground up around a networking library called enet. It’s a completely different beast aimed specifically at games, being built around UDP transport with optional reliability, with connections, sequencing and fragmentation all handled by the library. Using it ended up being fairly simple and comfortable, and it was easy to build my own generic packet handling interface on top of it. The first improvement of this rewrite is that I can put more faith in the security of the packet processing functions.

Rewrites are also a good opportunity to shed dependencies, to replace them with leaner and/or custom-made ones specifically suited to your needs. I’ve replaced ncurses with termbox2, which ended up being a great single-header alternative to a dependency with a lot of historical cruft. Also, I got rid of Cello in favor of my own type-erased data structures, such as unordered vectors, hashmaps, and strings.

This is where perhaps the second greatest change comes in. Because my ECS was written on top of the data structures of Cello, I had to rewrite it to use those of my own. The current ECS pretends very convincingly that it works right now, but it was very humbling to find a long-standing bug in the implementation after all this time (deleted entities were not removed from the empty archetype). On a related note, the static ECS that spontaneously emerged around entity modules is no more. It’s all part of the dynamic ECS now. Same goes for the Memory component and the segmented serialization scheme. In the end, I just wrote serialization and deserialization functions for all components by hand, and it ended up being simpler and less error-prone than the previous serialization scheme.

The last sweeping change is that a lot of core functionality was moved out into the default mod implementation. This makes the core less rigid, allows greater freedom for modders, and it simplifies the modded interface down to just a few functions. This means the game world, the ECS, serialization, windowing, input handling and a few other things are mostly up to modders now.

To summarize, a lot of core functionality was rewritten (and some of it became safer for it), a lot of it was moved out to the modded side, multiple dependencies were replaced (the core should build from scratch faster), redundant mechanisms were merged into more general ones (such as the static ECS being moved into the true, dynamic ECS), and the uxn core was updated to the current one (which should be a bit faster, while still remaining compact), and servers should be easy to set up to run on a single port. The interface towards uxn emulation is much better formed now (thanks to its creators), so this means that it should be much easier to swap in new cores in the future.

Rewrites are a strangely powerful thing, even if they usually take painful exertions of willpower to execute on moderately-sized personal projects. In retrospect, the old implementation usually looks like a bunch of shoddy abstractions piled on top of each other, even if you genuinely tried to build it to your highest standards. You can only get things so right on the first time around (or, as it seems, the second time around).