Leaderboards API Improvements

Runtime Framework
1

In my opinion, the design of the leaderboard API is overfitted towards this ‘reset schedule’ design that heroic labs has concieved of. Unfortunately, this design does not work for us so we have had to jump through many hoops to use the leaderboard system at all.

There are 2 core problems with the leaderboard system:

  • CRON format is insufficient

We want leaderboards that last a certain number of weeks, not ‘first day of the month’ or whatever limited set of options the CRON format allows.

The API shouldnt expose ‘cron format’ to the user. It should allow us to create a leaderboard with a precise expiry unix time. We can run the cron format function ourselves if we want to use that design.

  • Rank Cache is unwieldy

Because we couldnt use the leaderboard reset system because CRON format was insufficient for us, we have to use leaderboards without expiry times. We lazily create a leaderboard for each season, ie: global.points.19. After 2.5 years our nakama instance grew to several gigabytes of ram as it computed the rank cache for hundreds of leaderboards. We blacklisted each leaderboard ID manually. Note that the open source version of nakama doesnt even fully implement the leaderboard black list. The code is simply missing. We fixed this in our fork but didnt submit a PR. In our second game, we realized this flaw and instead use ‘tournaments’ instead of leaderboards for each leaderboard. The tournament API allows us to specify an exact end time. Note that its pretty confusing to have ‘start’, ‘end’ and ‘duration’. The documentation doesnt make clear why this time range is specified with 3 values. Rather then specifying ‘blacklist IDs’ in a pre-baked config, we should be able to create the leaderboard with a bool that excludes it from the rank cache. Note that we then wrote a custom SQL query to get leaderboard ranks for expired leaderboards. Nakama should default to using a full query if the rank cache is unpopulated, rather than returning ‘0’ for rank. Perhaps this should be a flag provided to the query API or it should be a seperate function call. Note that if the blacklist was part of the leaderboard creation system, it could be part of the DB, and the SELECT query the rank cache runs to get all the leaderboards could filter out blacklisted leaderboards. This would lower the cost of having many leaderboards.

Note that I am currently designing a system to have a leaderboard for every single user created level. These leaderboards should never expire, should delete when the level is deleted and shouldn’t be in the rank cache. We have over 20,000 user created levels. I am worried nakama’s systems wont be able to handle 20,000 leaderboards. I’m looking at modifying the rank cache code to allow a ‘pattern’ for the blacklisted IDs. Or I’m going to design a leaderboard schema for the DB from scratch.

[0] commit that finishes the blacklist system: actually fill the blacklist · winterpixelgames/nakama@c602e18 · GitHub

2

Hi @00jknight - I think the feedback about the CRON format is somewhat valid and we can take it into consideration for general improvements.

I can also see some value for a very minor case where the leaderboard/tournament entries should not have ranks associated with them; maybe we can make it as part of a flag on leaderboard/tournament creation to always skip rank calculations.

However your other issues with the rank cache and how you’d like to use it is a bit more complicated; I’d highly suggest that we speak on a call to try and help you. If you’d like to have a call, please contact either alim@ or the team on support@heroiclabs.com

3

My feedback is simple:

  • Leaderboards should be creatable with start and end time, basically just like tournaments
  • Leaderboards should be blacklisted from the rank cache in the create API, not in the config for nakama
  • Leaderboard haystack query should fallback to a full DB query if the rank is not in the cache

I think the rank cache would be more usable if it didnt grow indefinetly, if it had a cache eviction policy and only lazily added stuff to the cache…Currently, the stock, unconfigured rank cache is a liability as it can grow to exceed the maximum memory of the pod. Developers must be aware of the risk of this rank cache memory consumption.

I tweaked nakama to allow us to specify a basic wildcard pattern in the blacklisted leaderboard ID, and we have a custom SQL query to get the current rank of any user on any leaderboard, expired or not. These things seem like they would be useful to the community and to the heroic labs devs, which is why I am bringing it up here.

4

To summarize how we use nakama leaderboards:

  • for leaderboards that we want in the rank cache (ie: heavily used global leaderboards), we only create tournaments, not leaderboards, because we want to specify exact start and end times
  • for expired leaderboards (ie: past seasons) we have a custom SQL query to do a haystack lookup with rank, the query is probably expensive-ish but its never been a problem
  • for leaderboards we want not to be in the rank cache (ie: user created levels) we create them with a blacklisted prefix, and our fork of nakama respects wildcard patterns in the rank cache blacklist

note that in our first game, we werent aware of the rank cache growth issue, so due to our naive usage of the leaderboard API, we have to every 6 months or so go in and manually blacklist old leaderboards

and note that blacklisting leaderboards doesnt work in stock open source nakama: actually fill the blacklist · winterpixelgames/nakama@c602e18 · GitHub

5

Will you be willing to open PRs (separately) for each of the improvements you’d like to see in main line Nakama so we can begin to review (and ensure it’s backwards compatible) and land them?