a place to jot

The past day or two I’ve been struggling to update my crypto code to use Cap’N’Proto (capnp) and I’ve decided I don’t like the API at all. You have to translate your messages using streams and writers and a bunch of other janky stuff. I need to do something a bit unique in that I want to encapsulate my messages with encrypted messages that can then be deserialized aftrer decryption, so we have something like:

struct Msg {
  UserId
  Nonce
  union {
	  EncryptedData -> SomeStructEncrypted, SomeStructEncrypted2 etc
	  SomeOtherStructNotEncrypted
  }
}

The union bit is some what of a headache because I needed to use capnp’s streams and types to handle nested structures. Then I need to figure out what the hell type matches what I need to pass to my crypto code:

EncryptMessageBuilder.setUserId(0xd34db33f);
EncryptMessageBuilder.setXXX(kj::arrayPtr(Buf.data, SomeSize));
kj::VectorOutputStream outputStream;
::capnp::writePackedMessage(outputStream, EncryptMessage);
auto PlainText = outputStream.getArray();
crypto_aead_xchacha20poly1305_ietf_encrypt(Output, ..., PlainText)
MsgBuilder.setEncryptedData(::capnp::arrayPtr(Output.data, OutputSize));

kj::VectorOutputStream MsgOutputStream;
::capnp::writePackedMessage(MsgOutputStream, MsgBuilder);
kj::ArrayPtr<kj::byte> OutputData = MsgOutputStream.getArray();

What the hell is a ArrayPtr<kj::byte>? Why not just return a unsigned char *? Keep in mind I need to do the reverse of the above too, but using InputStreams and writers. Another problem is I have no idea how ArrayPtr stores the data, does it make a copy? If I use unique_ptrs and they go out of scope are they going to destroy it? If I pass data from the stack is it going to go out of scope and become invalid? I’m sure a lot of this would be more clear if I had more C++ experience but for now, I’m just not comfortable using this API.

Not having good documentation, no real sample code to look at, and the code itself being difficult to grok, I decided to drop it in favor of FlatBuffers. Besides some weird build issue I was able to get up and running with FlatBuffers pretty quickly. Their documentation is pretty good and they have samples/test code that is readable. I also found during debugging they have very helpful asserts and documentation as to why the assert occurred, A++.

An added bonus is the generated code is actually quite small and readable! And finally another benefit is that you don’t have to worry about not using fields on your tables/structs because FlatBuffers will automatically remove them and it won’t take up space if unused. This alone is a pretty big win I think.

The API is much easier to reason about too, here’s an example test of creating a game message, setting some raw bytes, creating a union type of a public key with it’s own data, and assigning that, then translating it to a straight up uint8_t ptr! Easy peasy:

    flatbuffers::FlatBufferBuilder FBBuilder(1024);
    
    std::vector<uint8_t> Bytez{1,2,3,4};
    std::vector<uint8_t> Nonce{4,5,6,7};
    uint32_t UserId = 0xd34db33f;
    auto PubKey = Game::CreatePubKeyRequest(FBBuilder, UserId, 0, 32, FBBuilder.CreateVector(Bytez));
    
    auto GameMessage = Game::CreateMessage(
            FBBuilder, Game::MessageType::MessageType_pubKeyRequest,
            UserId, FBBuilder.CreateVector(Nonce), 
            Game::MessageData::MessageData_PubKeyRequest, PubKey.Union()
    );

    Game::FinishMessageBuffer(FBBuilder, GameMessage);
    
    uint8_t *Buf = FBBuilder.GetBufferPointer();
    int Size = FBBuilder.GetSize();
    REQUIRE( Size == 80 );

    auto Deserialized = Game::GetMessage(Buf);
    REQUIRE(Deserialized->user_id() == UserId);
    auto Ret = std::equal(Nonce.begin(), Nonce.end(), Deserialized->nonce()->begin(), Deserialized->nonce()->end());
    REQUIRE( Ret == true );

I’m now in the process of wiring up the messages. During this rewiring, I’ve realized I need to add another queue to the network thread, one for inbound messages from clients and one for outbound messages directly from the crypto related code. This is because I am going to use ECS for my reliability layer but the network thread is where the crypto routines are being called (such as generating a symmetrical encryption key). My messages now look something like:

namespace Game.Message;

enum MessageType:byte {
    SignKeyRequest,
    SignKeyResponse,
    GenKeyRequest,
    GenKeyResponse,
    GameMessage
}

// Sent to auth server
table SignKeyRequest {
    user_id:uint;
    time_stamp:uint64;
    pubkey_size:uint8 = 32;
    pubkey:[ubyte];
}

// Response from auth server
table SignKeyResponse {
    time_stamp:uint64;
    pubkey_size:uint8 = 32;
    signature_size:uint8 = 32;
    pubkey:[ubyte];
    signature:[ubyte];
}

// Sent to game server for generating temporary symmetrical encryption key
table GenKeyRequest {
    time_stamp:uint64;
    pubkey_size:uint8 = 32;
    signature_size:uint8 = 32;
    pubkey:[ubyte];
    signature:[ubyte];
}

table GenKeyResponse {
    time_stamp:uint64;
    encryption_key:[ubyte];
}

table EncryptedData {
    data:[ubyte];
}

union MessageData {
    SignKeyRequest:SignKeyRequest,
    SignKeyResponse:SignKeyResponse,
    GenKeyRequest:GenKeyRequest,
    GenKeyResponse:GenKeyResponse,
    EncryptedData:EncryptedData
}

table Message {
    message_type:MessageType;
    user_id:uint;
    nonce:[ubyte];
    message_data:MessageData;
}

root_type Message;

The GenKeyRequest message requires generating a GenKeyReponse in the network thread. This then gets passed down to the game loop so we can notify ECS we have a new player and we need to send them their encryption key, along with tracking reliability of the message to make sure the client actually gets the message. Once they receive it we can start passing encrypted Message packets.

Anyways, it’s been a wild few days of rewriting code and I’m looking forward to solidifying the messaging layer so I can actually work on the stupid logic!