Why does % CPU usage on an idle barebones server (with no custom runtime plugin) increase with higher available RAM?

I’ve identified that idle CPU usage for a Nakama server with no custom runtime plugin varies with the specifications of the machine it’s running on. In this case, I’m focusing on the amount of available RAM.

By idle, I mean with no clients connected, no sessions and no network traffic.

In the image below, I’m showing some Prometheus metrics for 3 Nakama instances: A, B and C. In this example image, B and C are running on a machine with 1Gb of RAM, while A is on a machine with 4GB of RAM. All other specifications are identical.

The metrics are shown per instance, and each time point represents the per-second average rate over the previous 30 seconds, except for the number of sessions. They are, from top to bottom:

  • CPU Usage: process_cpu_seconds_total
  • Number of Memory Allocations: go_memstats_mallocs_total
  • Number of Goroutines: go_goroutines
  • Number of Sessions: nakama_sessions

While not shown, the amount of bytes received (nakama_overall_message_recv_bytes) and sent (nakama_overall_sent_bytes) was at 0 B when there were no sessions. I can provide more metrics and/or information if necessary.

In case it might also be relevant, these are the options we’re customizing, with secret or unimportant information redacted:

config.yml
name: <redacted>

logger:
  level: <redacted>
  file: <redacted>
  rotation: true

console:
  port: <redacted>
  username: <redacted>
  password: <redacted>

metrics:
  prometheus_port: <redacted>
  prefix: <redacted>

runtime:
  http_key: <redacted>
  path: <redacted>
  env: <redacted>

socket:
  server_key: <redacted>
  idle_timeout_ms: 3600000
  max_request_size_bytes: 10000000
  max_message_size_bytes: 10000000

session:
  encryption_key: <redacted>
  refresh_encryption_key: <redacted>
  token_expiry_sec: 3600
  refresh_token_expiry_sec: 604800
  single_socket: true
  single_match: true

As you can see, the % CPU usage is much higher on A compared to the others. This is certainly related to the number of memory allocations per second. Why does this happen? And, if possible, how can this be fine-tuned?

  1. Versions: Nakama 3.11, Docker 20.10.15
  2. Server Framework Runtime language: no custom runtime