Lua Integration

This week I worked on the integration of the Lua JIT. This is one of the more critical steps towards completion of the engine as it is the main way that developers will interact with the engine once they begin work on the game which will be built on top of the Monochrome engine.

Lua

Lua is a dynamic, embedded programming language with significant features for interoperability with C. It’s also considered an extensible language because features for the language can be developed in C and exposed to Lua through library functions not available in the core Lua runtime. We chose to use Lua as the scripting runtime for our engine because it is an easy language to learn, is fast to implement, has good performance, and provides great interoperability with C. We are using the Lua JIT implementation of Lua 5.1, which provides a tracing just in time compiler for the Lua language, as well as a few extensions to allow easier integration with C.

Lua cFFI

The main extension provided by the Lua JIT is the foreign function interface. In conventional C code interaction with Lua, a virtual program stack is stored in the Lua state and is interacted with in C in order to allow the transferal of data from Lua to C and vice-versa. This provides good and fast interoperability between Lua and C, but it provides an issue where significant work is required to expose C functionality to Lua. This is where the foreign function interface comes in. This extension allows you to enter simple function and type declarations in strings and pass them to the FFI, and then use those structures as they are passed back and forth to and from C functions, meaning that as long as Lua has flow control, very little work has to be done to allow interoperability.

Engine Split

One disadvantage of the FFI which the Lua JIT integrates is that the functions which it exposes must be loaded via a shared object or dynamically linked library. For more conventional libraries this usually proves no issues as those libraries are often on the system path, however in this case we need to ensure that Lua can call functions from within our own engine. This means that our engine needs to expose a large amount of functionality via a shared object, which obviously doesn’t work out well if the engine is left as a binary. As a result, we spent some time splitting out the codebase of the engine into a library and a core runtime. The library is loaded as a shared object by both the core runtime and the Lua JIT for the sake of the FFI and all the functionality needed is now able to be accessed when needed, without any duplication.

Lua Components

Components are able to be defined in Lua with the integration of this because of our design of the component data tables (see the last post in the Engine Development category) which do not assume any particular data format, and in fact do not need any information about the components besides the byte width of a single component. A folder is stored in the game’s runtime resources with Lua files which define the components as C structures using the JIT’s FFI and then register the component with the Lua runtime. During the initialization of the scene the Lua runtime checks to see if any components defined in Lua do not have data tables within the scene, and if any do not it creates them with default component counts from the component. This ensures that if any components are added at runtime by various systems it doesn’t catch the engine off-guard.

Lua System Runtime

Systems are split into two halves on both the C and Lua sides, with some systems being run at physics time at a constant sixty frames per second, while there are also render systems which are run as quickly as the computer running it can handle (at time of writing with minimal scenes on the test systems, this is in excess of 2.5k frames per second). When scenes are loaded they contain lists of names of systems, one for the physics frame systems and one for the render frame systems. The Lua runtime then takes these names and requires the relevant files from the systems directory in our runtime resources and then initializes each system. Each frame the C physics frame systems run before the Lua physics frame systems run, followed by the Lua render frame systems and then the C render frame systems. This allows physics to be run at the beginning of every frame in the C runtime to utilize the full amount of speed we can provide, then each of the gameplay systems implemented in Lua, then each of the visuals-based Lua systems, followed finally by actually rendering the scene and applying skeletal animation, all in C.

Integration

At this point the core engine features are nearly completed, with only the integration of the JSON parser and serialization library needed to allow saving and loading of scenes, input management, sound, and the physics engine. After each of those modules is integrated into the engine it will be fully capable of producing any single player game we might desire, and adding networking (while not planned for this project) would not be a difficult task. After that, the core engine team will be moving on to creating a more powerful rendering engine supporting many lights, full animation, UI, and eventually PBR and Voxel-Based Dynamic Global Illumination.

 

Entity Component Systems in the Monochrome Engine

Introduction

This week I’ve been tasked with creating an Entity Component System (or ECS) for the Project Monochrome engine. This is the main “runtime” on which all the rest of the game will sit upon, since it is what drives the state and updates in the game. Any given ECS has three parts to it: entities, components, and systems, each of which are outlined below. A fourth necessary piece to the resulting architecture, resources, was also designed this week, and is also outlined.

For those of you familiar with Unity, the idea of an ECS probably sounds pretty familiar. The initial mapping you might come up with mentally would likely look something like this: entities would map to GameObjects, components would map directly to Unity components, and systems would be a nebulous concept of the kinds of functionality built into the components. While on a surface level this might seem pretty close, an ECS in the way that it is being designed for Project Monochrome is very different from the component system found in Unity.

Entities

In a full on ECS like is being implemented for Project Monochrome, entities are one of the most bare-bones types of functionality needing to be implemented. Entities are represented internally by simply a UUID generated and saved in a simple byte format. This allows us to reference a particular entity specifically as if by name, without having to worry about collisions when generating new ones. This internally is represented by a simple union like so:

typedef union uuid_t
{
    uint8 bytes[64];
    char string[64];
} UUID;

This simple representation also allows us to compare two UUIDs by using the C function, strcmp.

Components

Components are one of the more interesting parts of the Monochrome ECS. The reason is that it diverges greatly from the object-oriented idea of components in other engines like Unity, instead favoring a much more data- oriented approach. In the Monochrome ECS, Components are simply data. They have no associated functionality, and are only directly associated with entities because entities are the way that you access component data.

Components are simply a form of data, and the shape and content of that data is entirely determined by the gameplay designers, very few of them will be defined by the engine itself. Because of this, we need a form of generic storage which will function for any component type, of any size, without wasting memory.

The solution to this problem that we came up with was to use one of the less- commonly used pieces of functionality in C, that being using malloc with a size only loosely correlated with the structure being allocated, in conjunction with zero-element arrays in trailing positions in structures. This allows us to allocate more memory than required to store the structure itself, and access the zero-element array as a pointer to the rest of the data of an unknown type. Our component data is stored in a table which uses this functionality.

typedef struct component_data_table_t
{
  // Maximum number of active components
  uint32 numEntries;
  // Byte size of the component structure
  uint32 componentSize;
  // Maps UUID to uint32 index of component
  HashMap idToIndex;
  // Data table
  uint8 data[];
} ComponentDataTable;

This structure is the one which stores the data inside of components, in the form of an array of pairs of UUIDs and component data interleaved together, as well as a hash map of UUIDs to indices into said array to allow faster lookup into the array without requiring a linear search. The size of individual component structures is also stored inside this table to allow functions which add and remove components to know how much data to copy into said structures in a generic way.

To store multiple kinds of components in the system, we have the concept of a “scene,” which stores two hash maps, one which maps from component type IDs, stored as UUIDs, to pointers to these component data tables, and one hash map from entity UUIDs to lists of components that entity has associated with it, allowing us to quickly discern what set of components an entity has based solely on its UUID.

Systems

In the Monochrome engine’s ECS, systems are small functions which run over sets of entities which have a given set of components associated with them. They are then allowed to make any changes to the world’s state that they see fit. This allows us to create generic functionality to run during a game loop without directly tying it to a particular entity in the world. Since frequently features don’t map cleanly to objects in the world, this also allows a conceptually simpler way to deal with abstract concepts like UI managers or score and achievement functionality.

Each system when run is given a set of components that need to be associated with any entity on which it is called. The runtime which calls these systems then determines which entities in the first necessary component’s table have all the needed components, before calling the system with the UUID of that component.

Resources

While this ECS concept gives us a clean framework to think about entities, their associated data, and how they change over time, it doesn’t however give us a way of dealing with assets which have to be managed as a part of a game’s runtime.

A few options present themselves for creating a resource management system within the Monochrome engine. First, we could represent resources as entities with associated components in a given scene, allowing us to have asset management fall cleanly within the ECS model and run along with everything else. Second, we could also have resources be their own separate module, unassociated with the base ECS implementation.

Each of these options gives us some advantages and disadvantages. Having resources be entities gives us a clean and unambiguous way of representing them within our current framework for the engine, but it also forces resources to be scene-specific, meaning that if multiple scenes load the same texture, for example, that texture would then be represented in GPU memory twice. Since our engine has some plans for the rendering engine which will be pushing GPU memory to its limits anyways, we opted for the second option which requires more design, but allows us more flexibility.

The design which was devised is a global reference counted system where by a given scene can have references to specific resources by filename, and when a scene is loaded, any resources which have yet to be loaded will begin an asynchronous load process, and whenever a scene is unloaded any resources whose reference count drops to zero will be unloaded. If a resource that has not been fully loaded yet is accessed, it will return an error and the system attempting to use it will simply fail, e.g. a model will simply not be rendered, or a texture will reset to some default value.

Rendering as a System

Taking all this together allows us to create a concept for what our rendering system might look like, once it is integrated with the entire ECS architecture.

A simple static mesh rendering system would iterate over all entities with a static mesh component, which would contain the name of the model and textures that it would render with, the shader pipeline that it will use, as well as any other data required to render a static mesh, such as a transform. An optimized rendering function would then be called, honed in specifically for drawing static geometry and avoiding unnecessary changes of graphics state.

Each other kind of rendering required for the entire engine can then be implemented with its own system, allowing a set of rendering commands which are much more conceptually simple to be used over a more conventional polymorphic approach.

Inception

Welcome to the engine development blog for this project which is code-named “monochrome”. We will be posting here weekly about the failures, successes, and everything in between that we will inevitably face over the coming months. We look forward to sharing technical documentation about the solutions we come up with to implement the features we will need in the engine.

DevOps

Given that our engine is planned to be cross platform, targeting Mac, Windows, and Linux, the first order of business was deciding on a language, compiler, build tool, and cross-compilation wrapper for the engine. Although C++ tends to be the de-facto language of choice for game engines nowadays, we opted to use C. C allows us to focus more easily on a procedural paradigm, and combines well with the Entity Component System which will be implemented later. It also allows us to put our focus on cache efficiency. We decided to go with clang as our compiler since it promises expressive diagnostics, is very standards compliant, and allows for simple cross compilation. GNU Make was the build tool of our choice for simplicity’s sake. CMake was put aside onto the backlog as a tool we may switch to in the future if ever needed. We used the mingw-w64-clang package from the arch user repository as a wrapper for clang during the cross-compilation process. All compilation was done from Linux build machines, so we required the appropriate libraries for our target machines to be installed on our build systems. This necessitated the use of mingw-w64-glfw and -glew. With these tools, we were able to setup a build environment that could compile and build Linux and Windows binaries on a Linux system prior to the first day of real development.

Day 1

For our first day, we surprised ourselves with our quicker than expected pace of work. We found that iterative development was extremely useful in forcing us to simply implement working builds as quickly as possible instead of worrying about the robustness of the implementation beforehand. This style of development has, so far, allowed us to work spontaneously first while allowing us extra time to focus on cleaning up and improving code afterwards.

Getting a simple game loop along with a blank window using GLFW was trivial. We decided to use the COLLADA schema to store and load mesh data, as it makes it really easy and simple to be able to import mesh geometry, skeletal animations, and physics geometry and properties. Instead of using COLLADA to store our scene graph, we decided that our scene graph will be stored in JSON files which will be exported from the scene editor so that we can easily construct a scene representation using only named references to mesh data. Instead of writing our own mesh importer from scratch, we opted to use the Assimp library to greatly reduce the time spent on that aspect of the engine. With Assimp being used to import mesh data, our first day of development came to a close.

Basic Rendering

With mesh data loading properly into our engine, we moved on to some basic rendering. In order to represent vertices in our engine, we needed to either write our own math library with vectors and matrices or use an existing library. We opted for the latter for the time being until we get around to implementing our own SIMD math library designed for performance. Sticking with what gets us a working build fastest will continue to be a recurring theme throughout our development. We were able to find kazmath, a very simple math library written in C, so we integrated it into the engine along with some very simple representations of mesh geometry and materials. We used the Utah Teapot as well as a single directional light to test our renderer, which worked out well enough for a simple example.

Cross Compilation Fun!

As it turns out, cross compilation isn’t an easy task for non-trivial projects. Our project is written in pure C which should make it easier to cross-compile, and we are using clang which makes it easier still since it is designed for cross compilation. However, clang does not help with ensuring the correct build environment is set up. To this end we used several packages from the Arch User Repository (AUR). Among the packages used were a part of the MinGW ecosystem, including mingw-w64-clang, mingw-w64-glfw, and mingw-w64-glew. These packages create the windows compilation environment necessary by wrapping the clang command for ease of use (mingw-w64-clang) and providing windows-specific dynamic and static libraries (mingw-w64-glfw, and mingw-w64-glew).