Automated scaling and blue green deployment in heroic cloud

Runtime Framework
1

Questions about scaling and blue green deployment in heroic cloud

After going through heroic cloud UI interface, i didn’t find any interface to make automated scaling possible.
So my question is

  1. Is it possible to do automated scaling in heroic cloud?

  2. If it is not possible to do automated scaling, how is scaling supposed to be handled ? is it manual ? if so can you tell us how other game studios do it and any best practices.

I see an option for rolling deployment in the heroic cloud interface, is there an option to do blue green deployment ?

Thanks and Regards
Harish Kumar Datla

2

@harishkumar These are good questions. I’ll answer each inline.

Is it possible to do automated scaling in heroic cloud?

It depends on what level of automation you mean. We can configure the autoscaler to adjust Nakama Enterprise deployment capacity for the game backend to fit the peak and trough player activity patterns across the timezones over 24 hours. This can be done based on generic metrics like CPU usage or game specific metrics that are defined through our server framework as custom metrics.

At the moment these rules are applied manually to a game title on Heroic Cloud. We do plan to expose UI for the configuration of these options in the future.

NOTE: This will not automate the scale in/out of database resources defined for the game title. It’s very complex to “scale in” databases because it affects the data storage on disk, available IOPS, and memory resources - all can affect overall system performance.

If it is not possible to do automated scaling, how is scaling supposed to be handled ? is it manual ? if so can you tell us how other game studios do it and any best practices.

I think most of this question I’ve answered above but will try to give a better example of scale out strategies we help game studios with. The process typically works as follows:

  1. The game studio builds up some custom cloud functions on the Nakama server framework which covers their game logic.
  2. They load test their RPCs - we usually recommend Artillery but some game teams use K6 or similar tools. This is done on prod type hardware for the game title over a few days.
  3. The target scale is reached as part of (2) which prepares the studio for their launch.
  4. The hardware is sized appropriately for launch date and then over a follow on window of time (depends on the game type, marketing plans, and distribution platform) we adjust the sizing of the hardware for peaks and troughs and any autoscaler rules.

You’ve probably already seen by our website but we’re very experienced in these game launches and have achieved these outcomes together with game teams over many years.

I see an option for rolling deployment in the heroic cloud interface, is there an option to do blue green deployment ?

This option is also locked behind a manual change which can be applied by our cloud team to the game project. We plan to expose UI for it in the future.

I should also add another point to consider with autoscaler rules. They cannot predict what traffic changes will look like; only react to what has changed in the metrics used to trigger the autoscale. This makes them less useful when games like the ones we typically work on are in Live Service because a Live Event can cause an immediate 3-4x in traffic change which no autoscaler rule would be prepared for in advance.

We have strategies to help with this kind of rapid traffic change in the way we handle games on the Heroic Cloud.

Hope this helps.

1 Like