Daemon, the Unvanquished engine is a descendant of Quake3 (1998) through ioQuake3, Xreal, ET:Xreal and OpenWolf. That's 15 years of tweaks, fixes and features added at random places in the code.
What used to be the “final evolution” of John Carmack's C style has grown into a mess of interdependent modules and special case handling. As a result, Daemon is less maintainable and the apparent lack of clear separation in the codebase scares new developers. Because Unvanquished is a long-term project we need to make our engine friendlier for new developers while becoming competitive with other modern engines.
Modernizing & cleaning the codebase
The Quake 3 family of engines is single-threaded, although it can optionally use a second thread for the renderer. The multithreading that would be the least intrusive to implement would be to use worker threads like Doom3 BFG does. However we eventually want to get close to TTimo's experimental engine in which the inputs, the client side gamelogic and the renderer are in separate threads. Yet, this is not possible without extensive changes, with most modules working like state machines and exposing inner data.
This is why we decided to incrementally rewrite the engine starting from the base framework, decoupling modules and taking multithreading into consideration. In addition, we are switching from C89 to C++11, or more like C with classes and some templates. Our hope is that by rewriting systems one at a time and keeping compatibility with the older engine, we can easily test changes and eventually switch engines seamlessly. I have already been playing all my games using the engine-upgrade branch.
Changing the virtual machines
The virtual machines containing the gamelogic still use QVMs compiled with a LCC (Light C Compiler) that hasn't been updated since the release of Quake 3. The gamelogic is written in C89, comes with its own standard C library, is under-optimized, and can barely do dynamic allocations. Experiments have been made on using LLVM bytecode compiled at runtime but they have security issues and many limitations.
Most new games and engines would use some scripting language like Lua or AngelScript, but we cannot afford to rewrite all of our gamelogic. Instead, we are looking at PNaCl (Portable Native Client) that is used to sandbox native applications in Google Chrome. It solves all of our problems, and it is a proven technology that will hopefully be maintained by Google for a long time. PNaCl basically compiles C/C++ to an instrumented LLVM bytecode that is in turn compiled at runtime into a sandboxed executable. Another blog post will describe in more details the implementation of PNaCl.
Progress and prospects
Work has started in the engine-upgrade branch, first by making all the engine compile and work in C++. We have replaced some C modules with empty shells that translate and proxy calls to their new C++ equivalent. The command system, base commands, the console data-structures and cvars are ported, soon to be followed by the filesystem API and the logging API. We chose these modules because they are at the basis of the engine and should only depend on each other. In its current state, the engine-upgrade branch is fully functional with minor differences in comparison to the older engine. The wiki page tracks these changes. In addition, we implemented PNaCl sandboxes for the gamelogic and successfully played on a PNaCl server with QVM clients.
The next step will be to stabilize the engine-upgrade branch and merge it into master, along with experimental support for PNaCl sandboxes. Overall I'm very hopeful about this project, making incremental changes to allow the engine upgrade to have visible results very quickly, and PNaCl can make both the engine architecture better and gameplay developers more productive.
As a final note, we are not professional engine developers so comments and criticism are very much welcomed.