One node crashed when restarting instance

We restarted our instance, to perform script updates, and one of our node after succesfull start crashed with error

2024-04-04T08:33:25.367Z panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1ddbd8d] goroutine 220 [running]: github.com/heroiclabs/nakama-enterprise/cluster.(*rpcServer).PresenceBroadcastDelta.func1() github.com/heroiclabs/nakama-enterprise/cluster/rpc_server.go:111 +0x2d created by github.com/heroiclabs/nakama-enterprise/cluster.(*rpcServer).PresenceBroadcastDelta in goroutine 219 github.com/heroiclabs/nakama-enterprise/cluster/rpc_server.go:101 +0x67

After that node successfully wakes up, but all matches assigned to that node disappeared.

This log error doesn’t look like code on our side, and we’ve lost a few thousand of matches.
How to prevent such a problem in the future?

How do you work with nodes restarting and keep matches safe?

Hello @jaggred, I suggest that going forward, for any Heroic Cloud related questions or issues you reach out at support@heroiclabs.com.

This bug should be fixed in the nakama-enterprise v3.21.1-r3 release.

On a multi-node cluster you can use match migration to migrate the matches to another node during a rolling reboot.