How to identify all room closed automatically

Local Setup
1

daily 5-10 time all rooms are closed suddenly. i can’t able to identify why all rooms closed in same time. my game developed in unity3d , server Authoritative Multiplayer code written in typescript. i installed nakama using docker. docker close all connection? anyone have idea about this issue? any facing similar issue

these log i found on server side

{“level”:“debug”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:198”,“msg”:“Error reading message from client”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”,“error”:“read tcp 172.20.0.3:7350->106.208.60.113:50957: i/o timeout”}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:432”,“msg”:“Cleaning up closed client connection”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:440”,“msg”:“Cleaned up closed connection matchmaker”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:444”,“msg”:“Cleaned up closed connection tracker”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:448”,“msg”:“Cleaned up closed connection status registry”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:452”,“msg”:“Cleaned up closed connection session registry”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}
{“level”:“debug”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/tracker.go:834”,“msg”:“Processing presence event”,“joins”:0,“leaves”:1}
{“level”:“info”,“ts”:“2021-07-03T06:53:38.621Z”,“caller”:“server/session_ws.go:469”,“msg”:“Closed client connection”,“uid”:“87da5177-81fc-4310-8a22-04220635e708”,“sid”:“8bc99ec9-dbca-11eb-8032-688e25b5a5ca”}

2

@srirampandiyan115 The logs you show don’t indicate anything unexpected just the usual socket cleanup which we log out at DEBUG level when a game client disconnects. If the server suddenly appears to close all connections its possible one of these situations occurs:

  • You lack the server hardware resources and some kind of out of memory issue occurs. What kind of hardware set up do you run the server on at the moment?

  • You run both the database and the game server on the same hardware which could cause the game server to run out of filesystem handles. What are the ulimits configured?

  • There’s a bug somewhere in your server logic which appears to trigger the server to shutdown that causes all game clients to be disconnected. This can be difficult to find if you don’t make sure that your game logic correctly stops running matches when all players leave a game.

    You should check the Nakama Console to determine if you see a continuous increase in goroutines that are active over time or the number of active matches never goes down. This could be a sign of “zombie matches” where your game logic does not terminate the matches where it should (due to a logic bug).

3

  • i’m using aws t2.medium [2core, 4gb ram ]

  • game server and cockroach db in same server i hosted.

  • My Ulimit config
    core file size (blocks, -c) 0
    data seg size (kbytes, -d) unlimited
    scheduling priority (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) 15689
    max locked memory (kbytes, -l) 65536
    max memory size (kbytes, -m) unlimited
    open files (-n) 1024
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) 819200
    real-time priority (-r) 0
    stack size (kbytes, -s) 8192
    cpu time (seconds, -t) unlimited
    max user processes (-u) 15689
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited

  • currently i written logic of all player left room match will end after 2minutes. anyway i’ll check match logics once again.

  • can you confirm docker will restart server when player usage increase?

  • my current DAU 300-400, 30-50CCU
    when i monitor server it never went above 50%. my admin panel hosted in separate server. nakama files and cockroach db only hosted in this server

4

@srirampandiyan115 Thanks for the information. All that looks fine.

can you confirm docker will restart server when player usage increase?

It depends on how you’ve configured Docker engine. By default it should not stop the server if it has no resource limits imposed on the container. Though that could lock up other system resources in general if you run at close to top end RAM usage. I do think you have plenty of RAM on the hardware instance at the usage level you’re at right now.

Can you look at the Docker logs for the container and have a look at whether there’s any logs that occur just before the reboot that might indicate a segmentation fault or similar? Maybe it would be covered as a stacktrace with a nil dereference?