Session outgoing queue full

gruset · July 12, 2024, 4:32pm

Session outgoing queue full.

This is the error that is thrown once 30+ people joins my game.

Im building an mmorpg, once match one server. We are 3 years into development, and running our 2nd beta test right now.

Im using go runtime framework. Everything works great until with 10-20 people are playing. But once we get to 30 players online at the same time (somewhere around that number) the error occur, and all users get disconnected. After that i have to restart the server. I must say, i can assure you that im not sending 64 messages each tick to a player…

I’ve done so many optimizations to my custom go code, that i cannot figure out how to handle this in any way. Can someone explain to me, why the queue is filling up? Does the queue have to wait response from the client to free up its message? I have set the queue size to 64.

How about a virtual machine? Im using a vmware partition, to run test this game now. Could that be the reason that the game logic is too demanding for the vm?

Looking forward for response.

Best regards,
Gillberg

sesposito · July 22, 2024, 1:34pm

@gruset looks like the client connection is not able to keep up with the number of messages sent:

github.com

heroiclabs/nakama/blob/master/server/session_ws.go#L409


      
          
          	return s.SendBytes(payload, reliable)
          }
          
          func (s *sessionWS) SendBytes(payload []byte, reliable bool) error {
          	// Attempt to queue messages and observe failures.
          	select {
          	case s.outgoingCh <- payload:
          		return nil
          	default:
          		// The outgoing queue is full, likely because the remote client can't keep up.
          		// Terminate the connection immediately because the only alternative that doesn't block the server is
          		// to start dropping messages, which might cause unexpected behaviour.
          		s.logger.Warn("Could not write message, session outgoing queue full")
          		// Close in a goroutine as the method can block
          		go s.Close(ErrSessionQueueFull.Error(), runtime.PresenceReasonDisconnect)
          		return ErrSessionQueueFull
          	}
          }
          
          func (s *sessionWS) CloseLock() {

Increasing the buffer size will likely not help in this scenario.

If the client bandwidth is insufficient, there’s a few things you can do:

Use Area of Interest tracking so that the clients only receive messages that are relevant to them
Reduce the bandwidth usage of the messages by using something like protobuf to serialize/deserialise them in binary format.
Try to reduce the number of sent messages further, if possible.

Hope this helps

gruset · July 22, 2024, 2:09pm

I’ve already done everything i can for bullet point 1 and 3. Sending messages to players within the same area and reduced the number of messages by batching it into bigger messages.

I will look into protobuf, currently everything is in json!

Eatos · July 22, 2024, 2:11pm

well that is a bit of problem, have you used JSON Size Analyzer | DebugBear?
Also remember that json gets converted to base64 which will increase size too.

Topic		Replies	Views
Outgoing queue full Runtime Framework	2	395	February 9, 2023
All players freeze on session outgoing queue full Runtime Framework	3	41	July 22, 2024
Knowing the message queue size on server side Game Design	5	516	July 12, 2024
Simulate multiple concurrent players Runtime Framework	4	107	March 26, 2024
Created party with max size 5 but 6 players can join Runtime Framework	3	325	December 29, 2022

Session outgoing queue full

Related Topics