Cap’N’Proto Serialization in C++

As I was building some game packet structs I finally came to the realization that, wait a second I don’t need to be doing this… at all. The plan all along was to serialize all the game netcode data anyways, so I might as well adopt a serialization package now?

The three major contenders were:

Honestly, one look at the FlatBuffer’s page and I got some bad JavaDocs vibes. That being said, the API seems OK, and I may actually give it a shot later. Second was Cap’N’Proto, I honestly can’t say why, maybe it was the scary JavaDoc but I ended up choosing Cap’N’Proto to try out. Third (FBE), I immediately threw out as I have concerns about portability and some scary issues in their backlog (also lack of a union type).

Anyways, so let’s dive in and see how we can serialize messages with Cap’N’Proto. First off, the documentation is pretty bad. If you’re looking to actually use it in C++. How do you create a message to be serialized with a union object? No documentation. How do you serialize the message to bytes? Also no documentation. How do deserialize from a blob of bytes to your objects? No… documentation…

Anyways, lets look at a sample schema file, which we have to create before we can generate the C++ header/code files:

@0xf0c855d2ab834bc2;

using Cxx = import "/capnp/c++.capnp";
$Cxx.namespace("game::packets");

enum MessageType {
    pubKeyRequest @0;
    pubKeyResponse @1;
    gameMessage @2;
    outboundKeyMessage @3;
}

struct Message {
    messageType @0 :MessageType;

    messageData :union {
        pubkey @1 :PubKeyRequest;
        pubkeyResponse @2 :PubKeyResponse;
    }
}

struct PubKeyRequest {
    userId @0 :UInt32;
    timeStamp @1 :UInt64;
    pubKeySize @2 :UInt8 = 32;
    pubKey @3 :Data;
}

struct PubKeyResponse {
    userId @0 :UInt32;
    timeStamp @1 :UInt64;
    pubKeySize @3 :UInt8 = 32;
    signatureSize @5 :UInt8 = 32;
    pubKey @2 :Data;
    signature @4 :Data;
}

Pretty basic, I have an enum for message types, a message that has the type, then a union that determines what type of message it will be. We want unions to save on space and reduce the schema complexity.

So, how the heck do I work with this? After running capnp compile -oc++ messages.capnp I have my c++ code ready to go, but no idea how to use it!

I knew I’d need some sort of input buffer/reader so I looked over the code until I found PackedMessageReader. Luckily, I have GitHub Code Search bookmarked for these situations, and started to search around for some of these keywords. I also found some references to a function called writePackedMessage, this looks good! Finally, I was able to find an example to figure out how to serialize the message:

	// Message malloc builder
	::capnp::MallocMessageBuilder Message;
	// Initialize the builder for a game packet message
	game::packets::Message::Builder Msg = Message.initRoot<game::packets::Message>();
	// Guess I needed this?
    kj::VectorOutputStream OutputStream;
    // And write our mesage to the output stream
    ::capnp::writePackedMessage(OutputStream, Message);
    // Now we need the bytes!
    auto OutputData = outputStream.getArray();
    // ...

So that’s how we can serialize our message, and then fire it off in our socket send() call. There’s a catch though. How the hell do you initialize a union type of a complex object? Again, no docs! The example they show is a union with simple types:

 employment :union {
    unemployed @4 :Void;
    employer @5 :Text;
    school @6 :Text;
    selfEmployed @7 :Void;
    # We assume that a person is only one of these.
  }

Their example C++ code is also not helpful:

bob.getEmployment().setUnemployed();

OK, how the hell do I make my PubKeyRequest struct? If you try to call a constructor you get:

game::packets::PubKeyRequest req;

../tests/../src/schemas/messages.capnp.h:76:3: note: 'PubKeyRequest' has been explicitly marked deleted here

[build]   PubKeyRequest() = delete;

Turns out you need to call, init<Type> which they definitely don’t explain anywhere:

	// build our msg
    game::packets::Message::Builder Msg = Message.initRoot<game::packets::Message>();
	// Set the type
    Msg.setMessageType(game::packets::MessageType::PUB_KEY_REQUEST);
	// call init!
    auto PubKeyBuilder = Msg.getMessageData().initPubkey();
    // Then work with it.
    PubKeyBuilder.setUserId(0xd34db33f);
    // ... serialize it ...

Cool so now we can serialize our full objects. How to deserialize? Again, no docs! So back to GitHub Code Search. After a while I found out you need a kj::ArrayInputStream. This is what that looks like:

	// get our array of bytes, this will come from a socket recvfrom. 
	 auto OutputData = outputStream.getArray();
	// create an input stream of the array bytes
    kj::ArrayInputStream in(OutputData);
    // create a reader
    ::capnp::PackedMessageReader reader(in);
    // read away!
    auto PubReader = reader.getRoot<game::packets::Message>();
    // real API says to use which() to figure out the type, but just an example here
    if (PubReader.getMessageType() == game::packets::MessageType::PUB_KEY_REQUEST)
    {
	    // we know what we got, work with it
	    PubReader.getMessageData().getPubkey().getUserId(); // == 0xd34db33f or whatever
    }

And that’s it so far! So I think I’ll stick with Cap’N’Proto, but maybe FlatBuffers would be a better choice :shrug:.

Cap’N’Proto Serialization in C++

Share this: