GSoC Week 11 and 12: Finishing up

This blog post is related to my GSoC 2022 project.

In my last two weeks of GSoC I worked on finishing up the serialization and deserialization routines for the various object types and implemented the game_save_buffer and game_load_buffer functions. I would’ve posted this post earlier, but unfortunately academic work got in the way and I fell quite ill on the 8th of September.

I got rid of the ENIGMA_INTERNAL_OBJECT_* macros and replaced them with enigma_internal_serialize and enigma_internal_deserialize functions, along with making variadic templates enigma_internal_serialize_many and enigma_internal_deserialize_many to reduce the verbosity of the code using them.

To make life easier I made the serialization and deserialization routines (serialize and deserialize_self) virtual so that they could be called with only a base pointer to any object in the inheritance heirarchy. This also made it so that I would not have to generate a game-specific function to call the correct instance of deserialize on the object returned by instance_create_id.

The most confusing part of the process came when implementing the symbol table for binary compatibility of the saved game state across multiple revisions of a game. User-defined objects cannot simply be serialized and deserialized byte-by-byte as the layout may change across versions so there is a need for finding the intersection of data available in the saved game state and the object itself.

To do this, the compiler emits a table for each object which maps the name of each variable it contains to a function which is responsible for deserializing the variable from a byte stream. When saving a game to a buffer, the names in this table are written out at the start of the data block of each object along with the offset from the end of the header where the data itself is actually stored. This makes it so that when loading a game from a buffer, the intersection of the contents of the header and the contents of the map can be computed to get the list of offset-deserialization routine pairs for each variable in an object.

Aside from this the game save routine also emits a single-byte object tag at the start of each object and sub-object. While this is not strictly necessary, it is useful as a small and simple check for the integrity of the following data. Finally at the end of the serialized state the SHA-1 checksum of the contents is emitted as an actual integrity check of the data. More in-depth documentation of each function is available in the docs commit itself.