@novabyte, it has been running for about 24hrs and I could not see any improvement. I can see the memory usage is increasing over time and same as the CPU usage.
@briar This is very very weird. Weâve not been able to reproduce at all and my theories on what cause it also appear to fall short. Can you give exact details on the type of hardware and platform level details you use with your deployment? Things like OS versions, database version, cloud platform services, etc.
It would be also good if you can share any unusual output you see in your server logs.
@novabyte I also think that this is weird. My setup consists of a cloud server (used for development purposes) with 1 vCPU and 1 GB of RAM. I use Nakama:2.11.1 for this setup. I have seen 100% CPU usage spikes, but other than that, I would say that the CPU usage is not increasing over time.
this is my code and it is an over-simplified test case. The âminimal configâ branch can be used to test whether the newer versions of Nakama are causing the CPU overload, or the code itself is causing the problem
@Mahdad-Baghani This CPU usage doesnât look like the problem reported by @briar.
Do you run the game server and the database server on the same hardware? Do you use the socket API in the game clients? If so, what CCU numbers do you see in terms of session counts on the server?
What other non-socket features do you use from the server API?
I probably should have elaborated more on what I meant by development server. I use this server for checking my pvp game in a more realistic way compared to low-latency local setup. The highest CCU that could have contributed to this diagram is 4. I run Nakama and cockroachDB instance on the same server. And yes, I use sockets in client to connect to the server. On the server, I currently use matchmaking+authoritative multiplayer features, nothing fancy.
@Mahdad-Baghani The problem with your CPU spikes will be caused by the fact that youâre running both the game server and database server on the same hardware. The database itself can be CPU hungry especially when it performs garbage collection to reap dead rows from the file system. I donât recommend you run the two on the same server hardware.
1 Like
Yep, that would be imperfect to run both database and game sarver on the same server instance, especially at scale when you want to manage clusters of your game backend, but do not need to have one instance of db per cluster node. But I didnât know the source of the cpu spikes and that perfectly explains it. Thanks. I donât know if I could help to shine some light on the main issue of this thread, though.
1 Like
@briar Any news on the CPU issue you noted in the 2.11.1 release of the server?
hi @novabyte, unfortunately it is still same on my side, CPU and memory is increasing over time.
I checked my logs, could not see any unusual messages in the logs. As I only use the multiplayer feature, almost all of my logs are:
New WebSocket session connected
Closed client connection
On very rare cases (approx. 10 in a day), I do also see error logs related with SSL like:
{âcallerâ:âhttp/server.go:3095â,âmsgâ:âhttp: TLS handshake error : read tcp : i/o timeoutâ}
@briar Would you be interested to run your project on our Heroic Cloud?
Weâd provide a free account for you for 3 months due to the special circumstances with the memory leak and it would enable us to take a look at the IO patterns, and other low level investigation paths to determine the root cause of the strange CPU patterns?
hi @novabyte, I completely missed this message. I think it will require quite bit of work for us to move to Heroic Cloud unfortunately. Is there anything we can do to help you diagnose the problem?
@briar Nothing more that I can do at this point. Yours is the only game project to exhibit these issues which make me think its related to how you run the infrastructure. I have no other ideas on how to investigate it. Sorry 