a place to jot

Groking epoll

Published by

wirepair

on

Well, things aren’t going as planned, while I’ve implemented about 90% of the rollback code, I’m finding that the servers and clients are getting desync’d way more than they should be. Looking over the logs I’m seeing the server process client packets at strange times. This made me go back over my netcode and give it a more in depth review. To be fair to myself, I pretty hastily threw it together and kind of never looked back.

What I was absolutely doing wrong

Looking over the net::Listener code I immediately realized two things: 1. My network loop was inefficient, and 2. I didn’t really fully understand how epoll worked.

So like any good investigator I broke it down into smaller grokable parts and lifted the architecture into it’s own project so I could re-implement and make sure I understood everything.

My network loop looked like this:

  1. Create an epoll instance, socket fd, and an epoll fd.
  2. INCORRECTLY set the event.fd = epollfd when creating an epoll_event struct (should have been socket fd!)
  3. while running
    1. process any messages from main thread to Send(…) to clients
    2. listen for any socket events via epoll_wait
      1. Check if the event.fd == epollfd and call ProcessPacket(…) (WRONG)
      2. if no events after 1ms timeout and go back to step 3 (while loop).

This is not how to use epoll. I need to be using a second file descriptor that can signal the epoll_wait when the main thread is ready to send out packets. For that you can use eventfd and eventfd_write. epoll has an instance, and an interest list. This interest list should have two descriptors, one for my main thread notifications and one for the socket to recv() from. If you want a really detailed breakdown of how epoll and the epoll API works, I highly recommend this blog post I found: the method to the epoll madness. It has pretty pictures and everything!

Working example code!

You can find the full repository here, it’s a simple server/client architecture with a lock free queue for thread safe message passing between a main thread and a network thread. It uses eventfd_write to notify when the network thread should read the queue and send the out going messages. This is what I should have been doing all along. While the code is here, let’s pull out the interesting bits. We’ll just look at the server.

The server

This may look familiar, it’s kind of how pmoserver is setup, but simplified. We create our socket, we create two queues that hold net::Message. Create a server, go into a loop popping off any messages the network thread processed and sent back to us, then every 100th iteration we send packets to all clients.

net::Socket Sock("127.0.0.1", 4042);
auto InQueue = std::make_shared<LockFreeQueue<std::unique_ptr<net::Message>, 65536>>();
auto OutQueue = std::make_shared<LockFreeQueue<std::unique_ptr<net::Message>, 65536>>();
net::Server Serv(Sock, InQueue, OutQueue);
Serv.Start();
std::cout << "Server started" << std::endl;
int i = 0;
while(1)
{
	std::this_thread::sleep_for(10ms);
	for (auto Packet = InQueue->Pop(); Packet != std::nullopt; Packet = InQueue->Pop())
	{
		std::cout << "Message from client: " << Packet->get()->PacketFrame << std::endl;
	}
	// signal our network thread we have "Outbound" messages to send to clients
	if (i % 100 == 0)
	{
		OutQueue->Push(std::make_unique<net::Message>(std::to_string(i)));
		Serv.NotifyWrite();
	}
	i++;
}

Note we have a Serv.NotifyWrite(). This is the epoll event we use to signal the network thread. Let’s look at the Server’s base class constructor, net::Listener:

class Listener 
{
public:
	explicit Listener(Socket &ServerSocket, std::shared_ptr<LockFreeQueue<std::unique_ptr<Message>, 65536>> In, std::shared_ptr<LockFreeQueue<std::unique_ptr<Message>, 65536>> Out) : 
		ListenerSocket(ServerSocket), InQueue(In), OutQueue(Out) 
	{
		// Create our epoll fd which will be used in Listen()
		EpollFileDescriptor = epoll_create(2);

		// Listen for NotifyWrite() events
		WriteEventFD = eventfd(0, EFD_NONBLOCK);
		struct epoll_event WriteEvent{};
		WriteEvent.events = EPOLLET | EPOLLIN; // make it edge triggered (e.g. only set when we call NotifyWrite())
		WriteEvent.data.fd = WriteEventFD;
		
		epoll_ctl(EpollFileDescriptor, EPOLL_CTL_ADD, WriteEventFD, &WriteEvent);
	

		Running.store(true);
	};
// ... other methods ...
}

Here’s where the interesting “event” bits get created.

  1. We create our EpollFD using epoll_create(...) (we say size of two but apparently linux just ignores this)
  2. We create a new event using eventfd and set it to EFD_NONBLOCK for non blocking
  3. We create our WriteEvent epoll_event and make it “edge triggered”. We do this by OR’ing EPOLLET | EPOLLIN together.
    1. This basically means epoll_wait will ONLY set ready state (return) if we wrote to WriteEventFD using eventfd_write(WriteEventFD, XXX);
  4. We set the WriteEvent.data.fd to this eventfd() created fd. We use this to compare epoll_wait‘s event arrays to see if the event epoll_wait saw was this WriteEventFD or something else (e.g. the socket fd)
  5. We then call epoll_ctl with EPOLL_CTL_ADD to add it to the interest list.

Both the client and server share the same Listen() method. So we only really need to look over this loop:

void Listener::Listen()
{
	struct epoll_event Events[MaxEventSize];

	// Listen for Socket events Socket.GetFD() should now be connect()ed or bind().
	struct epoll_event SocketEvent{};
	SocketEvent.events = EPOLLET | EPOLLIN;
	SocketEvent.data.fd = ListenerSocket.GetFD(); 
	epoll_ctl(EpollFileDescriptor, EPOLL_CTL_ADD, ListenerSocket.GetFD(), &SocketEvent);

	while (Running.load()) 
	{            
		auto Ready = epoll_wait(EpollFileDescriptor, Events, MaxEventSize, -1);
		if (Ready < 0)
		{
			std::error_code ec(errno, std::system_category());
			std::cout << "Server::Listen exiting" << ec.message() << std::endl;
			return;
		} 
		else
		{
			for (int i = 0; i < Ready; i++)
			{
				if (Events[i].data.fd == ListenerSocket.GetFD())
				{
					std::cout << "time to read socket" << std::endl;
					struct sockaddr_in ClientAddr;
					std::memset((char *)&ClientAddr, 0, sizeof(ClientAddr));	
					std::array<unsigned char, 1024> PacketBuffer{0};
					
					auto PacketLen = ListenerSocket.Receive(ClientAddr, PacketBuffer.data(), 1024);

					if (PacketLen <= 0)	{ continue;	}

					ProcessPacket(PacketBuffer, PacketLen, ClientAddr);
				}
				else if (Events[i].data.fd == WriteEventFD)
				{
					for (auto Packet = OutQueue->Pop(); Packet != std::nullopt; Packet = OutQueue->Pop())
					{
						 std::cout << "Main thread sent: " << Packet->get()->PacketFrame << std::endl;
						 SendPacket(*Packet.value());
					}
				}
			}
		}
	}
}

Before we start our listen loop, we create our SocketEvent epoll_event. This is almost exactly the same as the WriteEventFD except we are using the Socket created from either Connect() (client) or Bind() (server). Again, we add this SocketEvent to our EpollFileDescriptor interest list using EPOLL_CTL_ADD.

From here we enter our loop, and wait.

Before (in pmo code) I was setting a timeout of 1ms (the 4th argument to epoll_wait). Now I set it to -1 which says just return whenever there is an event, so basically blocking. Once something writes to our FDs in our interest loop, Ready is now set. If it’s less than 0, we got an error.

Otherwise… we could have multiple events so lets iterate over them. Here’s where setting Event.data.fd = {some fd} becomes important! We need to compare these fields to see which event we got and handle it appropriately.

If our fd == ListenerSocket fd, then we call Receive and fill our PacketBuffer with the data.

If our fd == WriteEventFD we send out packets using SendPacket.

So how do we make WriteEventFD trigger? We do that in Listener.NotifyWrite().

void Listener::NotifyWrite()
{
	eventfd_write(WriteEventFD, 1);
}

That’s it really, since this is thread safe, we call this method from our Main thread after we push data into our Lock Free Queue (LFQ).

The rest of the code isn’t that all interesting, it just handles serializing the string message and deserializes it. If server, it sets a list of peers in an unordered_set but doesn’t really do anything else. This really was an exercise for me to make sure I understood the netcode I had.

With that, I’d say mission successful.

Previous Post
Next Post