Lua Integration

This week I worked on the integration of the Lua JIT. This is one of the more critical steps towards completion of the engine as it is the main way that developers will interact with the engine once they begin work on the game which will be built on top of the Monochrome engine.

Lua

Lua is a dynamic, embedded programming language with significant features for interoperability with C. It’s also considered an extensible language because features for the language can be developed in C and exposed to Lua through library functions not available in the core Lua runtime. We chose to use Lua as the scripting runtime for our engine because it is an easy language to learn, is fast to implement, has good performance, and provides great interoperability with C. We are using the Lua JIT implementation of Lua 5.1, which provides a tracing just in time compiler for the Lua language, as well as a few extensions to allow easier integration with C.

Lua cFFI

The main extension provided by the Lua JIT is the foreign function interface. In conventional C code interaction with Lua, a virtual program stack is stored in the Lua state and is interacted with in C in order to allow the transferal of data from Lua to C and vice-versa. This provides good and fast interoperability between Lua and C, but it provides an issue where significant work is required to expose C functionality to Lua. This is where the foreign function interface comes in. This extension allows you to enter simple function and type declarations in strings and pass them to the FFI, and then use those structures as they are passed back and forth to and from C functions, meaning that as long as Lua has flow control, very little work has to be done to allow interoperability.

Engine Split

One disadvantage of the FFI which the Lua JIT integrates is that the functions which it exposes must be loaded via a shared object or dynamically linked library. For more conventional libraries this usually proves no issues as those libraries are often on the system path, however in this case we need to ensure that Lua can call functions from within our own engine. This means that our engine needs to expose a large amount of functionality via a shared object, which obviously doesn’t work out well if the engine is left as a binary. As a result, we spent some time splitting out the codebase of the engine into a library and a core runtime. The library is loaded as a shared object by both the core runtime and the Lua JIT for the sake of the FFI and all the functionality needed is now able to be accessed when needed, without any duplication.

Lua Components

Components are able to be defined in Lua with the integration of this because of our design of the component data tables (see the last post in the Engine Development category) which do not assume any particular data format, and in fact do not need any information about the components besides the byte width of a single component. A folder is stored in the game’s runtime resources with Lua files which define the components as C structures using the JIT’s FFI and then register the component with the Lua runtime. During the initialization of the scene the Lua runtime checks to see if any components defined in Lua do not have data tables within the scene, and if any do not it creates them with default component counts from the component. This ensures that if any components are added at runtime by various systems it doesn’t catch the engine off-guard.

Lua System Runtime

Systems are split into two halves on both the C and Lua sides, with some systems being run at physics time at a constant sixty frames per second, while there are also render systems which are run as quickly as the computer running it can handle (at time of writing with minimal scenes on the test systems, this is in excess of 2.5k frames per second). When scenes are loaded they contain lists of names of systems, one for the physics frame systems and one for the render frame systems. The Lua runtime then takes these names and requires the relevant files from the systems directory in our runtime resources and then initializes each system. Each frame the C physics frame systems run before the Lua physics frame systems run, followed by the Lua render frame systems and then the C render frame systems. This allows physics to be run at the beginning of every frame in the C runtime to utilize the full amount of speed we can provide, then each of the gameplay systems implemented in Lua, then each of the visuals-based Lua systems, followed finally by actually rendering the scene and applying skeletal animation, all in C.

Integration

At this point the core engine features are nearly completed, with only the integration of the JSON parser and serialization library needed to allow saving and loading of scenes, input management, sound, and the physics engine. After each of those modules is integrated into the engine it will be fully capable of producing any single player game we might desire, and adding networking (while not planned for this project) would not be a difficult task. After that, the core engine team will be moving on to creating a more powerful rendering engine supporting many lights, full animation, UI, and eventually PBR and Voxel-Based Dynamic Global Illumination.