Finally, encrypted communications!

Published by

on

Phew, it’s been quite a week trying to get server/client communications working. I now have a much better understanding of FlatBuffers. An added bonus is that certain aspects of modern C++ are starting to make sense now.

Before we dive into how I’m structuring my messages, let’s look at the current flow:

For now I just hard code the auth service inside the client, where it signs the clients user id and public key with the auth servers private key and hands the signed message back to the client. The game server has the auth servers public key to verify. So the client then sends this same exact message from the auth server directly to our game server.

Our game server can verify the signature by using lib sodiums crypto_sign_verify_detached with the auth servers public key, if everything checks out we can then generate a symmetrical key for encrypting individual messages between the client/server. Note these keys are only for the lifetime of the session.

Our key generation is pretty simple:

    /**
     * @brief Generates a xchacha20poly1305 key
     * 
     * @return std::shared_ptr<std::array<uint8_t, PMO_KEY_SIZE>>
     */
    SharedUint8ArrayPtr Cryptor::GenerateKey() const
    {
        auto Key = std::make_shared<std::array<uint8_t, PMO_KEY_SIZE>>();
        crypto_aead_xchacha20poly1305_ietf_keygen(Key->data());
        return Key;
    }

Now with a key to use, we can encrypt this key using the users public key and our game servers private key:

    /**
     * @brief Encrypts a generated key using a public key of one user, and a secret key of another.
     * 
     * @param Nonce 
     * @param UsersPublicKey 
     * @param GeneratedKey 
     * @param OutputLen 
     * @return UniqueUCharPtr 
     */
    UniqueUCharPtr Cryptor::EncryptKey(std::array<unsigned char, crypto_box_NONCEBYTES> &Nonce, const unsigned char *UsersPublicKey, const SharedUint8ArrayPtr GeneratedKey, size_t &OutputLen) const
    {
        randombytes_buf(Nonce.data(), sizeof(Nonce));
        
        OutputLen = crypto_box_MACBYTES + KeySize();
        auto CipherText = std::make_unique<std::vector<unsigned char>>(OutputLen);

        if (crypto_box_easy(CipherText->data(), GeneratedKey->data(), KeySize(), Nonce.data(),
                            UsersPublicKey, TestGameServerPrivateKey.data()) != 0) 
        {
            return nullptr;
        }
        return CipherText;
    }

This is where the messaging fun begins, we need to be able to wrap up encrypted data inside our messages, but after decrypting, we want to be able to deserialize it using the normal FlatBuffers deserialization classes.

I opted to go for a nested object system. We have a top level GameMessage that contains details we’ll need, such as the user id and nonce value. Then we’ll have a union type that could be an encrypted message or an unencrypted message. By having an additional message_type field, we’ll know how to deserialize the object after decryption. Before serialization we insert the encrypted object as a simple uint8 array. Once the encrypted object is inserted, we serialize the whole GameMessage.

The serialization process of an encrypted ClientCommand message looks like this:

// Create our nested message type
auto Output = net::MessageBuilder::BuildClientCommand(1, 1, Commands, Orientations);
// Encrypt the nested message type details by getting the Output->data() buffer
std::array<unsigned char, NONCE_BYTES> Nonce{0};
auto Encrypted = Crypt.Encrypt(ExpectedUserId, Nonce, Output->data(), Output->size());

// Now we create our top level message
flatbuffers::FlatBufferBuilder Builder(1024);
auto EncryptedBuffer = Builder.CreateVector(std::move(Encrypted->data()), Encrypted->size());
auto NonceValue = Builder.CreateVector(Nonce.data(), NONCE_BYTES);
auto EncryptedData = CreateEncryptedData(Builder, EncryptedBuffer);
// Build the Message Header and apply our encrypted message type
Game::Message::MessageBuilder GameMessage(Builder);
GameMessage.add_user_id(ExpectedUserId);
GameMessage.add_message_type(Game::Message::MessageType_ClientCommand);
GameMessage.add_nonce(NonceValue);
GameMessage.add_message_data(EncryptedData.Union()); // note the union call!
GameMessage.add_message_data_type(Game::Message::MessageData_EncryptedData); // <-- DO NOT FORGET THIS

Builder.Finish(GameMessage.Finish());
// We can now access Builder.GetBufferPointer() and Builder.GetSize() to send over Sockets

It took me quite some trial and error to figure out how do this, mainly because there’s certain orders/peculiarities in using FlatBuffers. You have to create complex/vector types before building a message, otherwise you get assertion failures. You also have to call add_message_data_type with the union type or it just won’t insert it, even if you call add_message_data. This was frustrating as I just had an empty field and no error.

I’m so thankful I went with FlatBuffers because they really make identifying these errors way easier than what I saw in cap’n’proto.

The deserialization/decryption process is as follows:

// Get Packet from the Socket
auto Deserialized = GetMessage(Packet->data());
// Can access these fields directly as they are not encrypted
auto UserId = Deserialized->user_id();
// note the call here to treat the message as EncryptedData
auto DeserializedData = Deserialized->message_data_as_EncryptedData();

// Now we can decrypt the data from inside the Deserialized message
unsigned long long OutputLen{0};
auto Decrypted = Crypt.Decrypt(UserId, Deserialized->nonce()->data(), DeserializedData->encrypted()->data(), DeserializedData->encrypted()->size(), OutputLen);

// Since we have access to the message type, we can deserialize the encrypted data as a Flatbuffers message now.
auto Command = Game::Message::GetClientCommand(Decrypted->data());
// ... process commands

We are safe to rely on the unencrypted UserId because we use that value as the AD in the Decrypt routine. Meaning if someone tried to forge the UserId, we’d get the wrong key and not be able to decrypt the message.

/**
     * @brief Decrypts a message intended for UserId, validates the UserId as part of the Authenticated Data (AD).
     * 
     * @param UserId 
     * @param Nonce 
     * @param CipherText 
     * @param CipherTextLen 
     * @param MessageLen 
     * @param OutputLen 
     * @return UniqueUCharPtr 
     */
    UniqueUCharPtr Cryptor::Decrypt(const uint32_t UserId, const unsigned char *Nonce, const unsigned char *CipherText, const unsigned long long CipherTextLen, unsigned long long &OutputLen)
    {
        auto UserKey = GetKey(UserId);
        if (!UserKey)
        {
            return nullptr;
        }

        auto AD = serialization::UInt32tToUChar(UserId);
        auto OutputBuffer = std::make_unique<std::vector<unsigned char>>(1024);
        auto ret = crypto_aead_xchacha20poly1305_ietf_decrypt(OutputBuffer->data(), &OutputLen,
                                               NULL,
                                               CipherText, CipherTextLen,
                                               *AD, sizeof(UserId),
                                               Nonce, UserKey->data());
        if (ret != 0) 
        {
            /* message forged! */
            return nullptr;
        }
        return OutputBuffer;
    }

OK so we have the server sending back to the client a GenKeyResponse message that contains our encrypted key. I actually had a really stupid bug where I directly added the GenKeyResponse encrypted_key field to my key store. Meaning I didn’t actually decrypt/verify it first. I noticed that the client thought the Key was 48 bytes, but I know the key should be 32 bytes. After some head scratching I realized I just blindly added the “encrypted encryption key” instead of decrypting the key, and then adding it to the keystore.

The final issues I ran into were how to tie this into my ECS system. Since I don’t have a reliability layer yet, I opted to just use flecs interval directive for my system:

GameWorld.system<NetworkComponent, NetworkClientComponent, AuthComponent>("Authenticate")
	.interval(.150)
	.each([&](flecs::iter& It, size_t Index, NetworkComponent &Network, NetworkClientComponent &Client, AuthComponent &Auth) {
		auto CurrentPlayer = It.entity(Index);
		if (Auth.RequestSent > 50)
		{
			// TODO handle properly
			fmt::print("Auth failed too many times\n");
			GameWorld.quit();
			return;
		}       
		auto Now = std::chrono::system_clock::now();
		fmt::print("Entity {} requires auth\n", CurrentPlayer);
		Client.Client->Authenticate();
		Auth.RequestSent++;
		Auth.TimeSent = Now;
	});

The client will just send every 150ms an auth request to the server, and wait for a GenKeyResponse, giving up after 50 retries. Once the client gets the GenKeyResponse we remove the AuthComponent to signal that the system should no longer be called (since the player will no longer match having those components).

We handle this in the packet processing portion of the game loop:

for (auto Packet = InQueue->Pop(); Packet != std::nullopt; Packet = InQueue->Pop())
{
	switch(Packet->get()->MessageType)
	{
		// Got key to encrypt or ClientCommands, remove the AuthComponent set Connected
		case Game::Message::MessageType::MessageType_GenKeyResponse:
		{
			newPlayer.remove<player::AuthComponent>();
			newPlayer.add<player::Connected>();
			break;
		}
		...
	}
}

The server is a little different, we send the GenKeyResponse but then wait for the client to start sending ClientCommands. The first time we see a client command means they are connected and we can drop the AuthComponent from the player on the server side.

// Create a cached query to match players who are trying to authenticate
auto AuthQuery = GameWorld.query<player::NetworkComponent, player::NetworkServerComponent, player::AuthComponent>();

// ... game loop ...
for (auto Packet = InQueue->Pop(); Packet != std::nullopt; Packet = InQueue->Pop())
{
	switch (Packet->get()->MessageType)
	{
		case Game::Message::MessageType_ClientCommand:
		{
			// This query matches all players who had an AuthComponent 
			// but now sent us their first ClientCommand
			// We can remove the AuthComponent and set them to Connected
			GameWorld.defer_begin();
			// Call auth query and set them as connected
			AuthQuery.each([](flecs::entity Player, 
			player::NetworkComponent&, player::NetworkServerComponent &, player::AuthComponent &) 
			{
				Player.remove<player::AuthComponent>();
				Player.add<player::Connected>();
			});

			GameWorld.defer_end();

			auto Command = Game::Message::GetClientCommand(Packet->get()->Message->data());
			if (!Command) { ... }

That’s pretty much it, now I have working client communications. I tested with 10 clients:

recv 596 bytes from client 746571137
Deserializing/Decrypting packet took: 0.029948ms
recv 596 bytes from client 895195883
Deserializing/Decrypting packet took: 0.053068ms
recv 596 bytes from client 2111722076
Deserializing/Decrypting packet took: 0.020592ms
recv 596 bytes from client 270303950
Deserializing/Decrypting packet took: 0.027815ms
recv 596 bytes from client 1210144944
Deserializing/Decrypting packet took: 0.03103ms
recv 596 bytes from client 428800927
Deserializing/Decrypting packet took: 0.039512ms
recv 596 bytes from client 576537365
Deserializing/Decrypting packet took: 0.031732ms
recv 596 bytes from client 286914240
Deserializing/Decrypting packet took: 0.063374ms
recv 596 bytes from client 881098529
Deserializing/Decrypting packet took: 0.027832ms

I am pretty impressed with how fast deserializing and decrypting the packets takes! Here’s hoping it can scale to 1000 clients :>.

Final thoughts, I will need to slim down my client messages (they’re 500 bytes now for the last 20 commands/orientations). This needs to be reduced significantly as some math shows with a tick rate of 30hz per client (1 tick is 33ms):
500 bytes * (1000ms/33ms) = 15151 bytes per second, per client.

But for now, I’ll probably just stick with this just so I can start getting something working in Unreal Engine to see some actual network/gameplay.

Stay tuned!