Storage Value Versioning - Tips

Hello everyone,
Recently we started to do some changes in our game, that made some of the saved collections structures obosolete. Not fully obsolete but some of the fields had change type and structure.
For example Before structure looked like this:

{ "test":[ { "param_a":"value1", "param_b":"value2", "param_c":"value3" }] }

Now we want to change structure to something like this

{ "test":[ ["value1","value2","value3"]] }

So basically instead to have an array of anonimous object, we want to switch to have an array of an array.

Looking at storage system since this is as value stored, writing a postgres query to update it sounds impossible not very efficient to run it through on whole DB.
Next to it was to try to update the data when user authenticates and for a single collection it could work.
And if update is easy, but over the time making function for every change, every version could buckleup.

So instead of doing all of these manually i tried looking up on JSON Schema validation and maybe using that as an upgrade mechanic. I do remember but my memory may be faint, is that with avro schemas, I could upgrade JSON structure to new one just by defining schema mitigration for any data.

Could something like that be used here or what is the best way to handle it?
I could possible dump for every collection that is related to user, its structure, and then introduce somewhere for each file “version” but not 100% sure on which way should we go.
Any tips are greatly appreciated :slight_smile:
Thanks

Nakama: 3.16.0
Runtime: Lua
SDK: C#, Unity

Hi @Eatos,

Please can you explain what data it is that you are storing and what decisions you have made that cause you to need to change the structure of the data?

If there really is a need to change the structure, I would go down the route you already mentioned which would be to migrate the data on a per-user basis when the user next authenticates.

Can you please explain what you mean by:

over the time making function for every change, every version could buckleup

Is the plan to change the structure of the data many times in the future? If so, for what reason? I would advise against a constantly shifting data structure where possible.

Thank you @tom for replying :slight_smile:
Sure i can elaborate a bit more, we are still in semi dev version but we do tend to make saves of the users backward compatible or even though being able to upgrade it to new version without need to delete account :confused:
So for instance i have a collection called cards where we store what card user has unlocked and its level.

{ "cards":[ { "cardId":"Rocket", "cardLevel":1 }, { "cardId":"ReRoll", "cardLevel":1 }] }

So basically for each card we have stored unlocked cardId, and cardLevel, but due to recent changes in gameplay mechanics, i would need to extend that serialized data to include additional fields.
But then we have also learned that we dont need to have here data stored as objects, as we don’t need to know keys, rather just values so therefor we could completly switch to array based access and structure.

v1 change - includes new value and removes keys

{ "cards":[ ["Rocket",1,"*"] , ["ReRoll",1,""]] }

v2 change - switches all data to int based

{ "cards":[ [10,1,1] ,[8,1,0]] }

And this is only related changes to this collection. Next what could happen additional changes to account.metadata. We do tend to store there data for quick access rather than to query storage if there is no need for it and list can go on.

So potentially we would end up with having an upgrade controller which would need to have for every collection a function to mitigate data to newer version.

Do we expect in the future data to change many times? I hope not, although with ever shifting changes to gameplay and addition of newer mechanics, the data may require an update or two. I hope again nothing drastical as whole structure change, but rather addition or removal or keys and values.

Thanks for explaining.

Can I ask what the perceived benefit of moving from a keyed map to a simple array is here? Is it simply to reduce the size of the stored objects or is there some other benefit you are looking to achieve?

With a keyed data structure the order of properties is irrelevant and you can add new properties in a backwards compatible way. With an array based approach you will be locked into that structure much more rigidly.

Let’s say for example index 0 in the array defines your cardLevel and you later add a cardExp as index 1 in the array. You then realise later that you can infer cardLevel from cardExp and so cardLevel becomes redundant and needs to be removed, but you do need to add another property for cardRank. This becomes trickier to achieve with the array based approach. How would you know if User A has a data structure that refers to [cardLevel, cardExp] and needs migrated or [cardExp, cardRank] and is ok?

Regardless of how you choose to re-structure your data I would still recommend migrating it when the user next authenticates if possible to avoid having to do a large batch migration which could be impacted by user data being modified during the process or potentially involve taking the game down for maintenance.

The end goal would be to reach that this structure can become an array of ints for example.
Why an array of ints?
In int taking 32 bits, we have figured out we can represent cards easily, with imposing some gameplay limits that are currently set into the stone for example:
Max cards in the game 255 → so 8 bits
Max card level 12
and so on…

In our game, during one turn player can draw multiple cards and at first we were sending for every card draw an rpc, but that was a bit overkill. So instead where we could we converted data that is being sent to array based sending for example:

On Client side:

We have method down the line that has something like this
_RPC_On_Cards_Drawn(params object cardsData)

On Server side:

local cards_packed = {}
for i=1,#cards_drawn do
if cards_drawn[i] ~= nil then
local packed_num_str = game_helpers.get_binary_card_representation_str(cards_drawn[i])
if packed_num_str ~= “” then
local raw_num = utils_bin.bin2dec(packed_num_str);
if raw_num ~= -1 then
table.insert(param[message_key], raw_num)
end
end
end
end
local message = nk.json_encode(param)
dispatcher.broadcast_message(opcode_handler.opcode_name_to_id[“_RPC_On_Cards_Drawn”], message, nil)

So the message that gets sent, is pretty much just an array of ints.

In your example currently we had that case with opcodes being called, that is why for every opcode we have a packet structure/definition how to read it, and it needs to map on it, else it is being discarded.
But we didnt had until now for player data structure.

For now we have came up with 2 possible solutions but we need a little bit more info before deciding to proceed on any.

  1. Introduce to storage system tables a new colum (game_client) version. This was only so we could better filter /access collection state/structure before parsing it completly.
  2. Add to each value inside collection a key “version” and which structure it is using it, and through JSON Schemas let data be controlled and migrated properly.

Idea is to:

  1. Define schemas for each collection and version

  2. In let say Upgrade Manager define a list of all possible user collections names like: [“cards”,“rewards”,“quests”,“deck”]

  3. On Before hook, check for each user collection what version it have and then if it needs updating it would start the upgrade process. This step can also be mitigated after authentication and before our internal loading process and fetching of the data.

  4. The Upgrade Manager when it receives oudated data structure, it looks for its version, it would check how far it needs to go or how many updates would it need to apply and then look for mitigatation schemas or functions. Not sure if Nakama runtime allowes dynamic function calling through string concatanation need to test.
    Example would be maybe something like this:

    Outdated {“version”:“1”, “cards”:[{“cardId”:“Rocket”, “cardLevel”:1}, {“cardId”:“ReRoll”, “cardLevel”:2}]}

    Current Server/Client required version: 5

    It would then run through a for loop going from 1 until 5 and look for mitigation functions or scripts.

 local old_data = current_data
 local collection = "cards"
 local from_version = start_version
 local next_version = from_version + 1
 for i = next_version, required_version do
      old_data = upgrade_manager.mitigate(from_version, next_version, collection, old_data)
      from_version = next_version
 end 
 -- Save to storage

And mitigate function it self, could based on collection and from, next version to what is required with that thus:
version:1 => [cardLevel, cardExp]
version:2 => transform function would be
→ Requires that version1 data has structure based on schema
→ Do manually steps to upgrading by reading data that is in currently and applying well code
→ So then it becomes very easy to know how to mitigate it and if values are correct.

Thanks for the detailed response @Eatos.

It seems you may be conflating data sent across the wire with data that is stored in the Storage Engine to represent what cards a player has.

The idea of optimising the packing of data sent across the wire for RPCs is a sound one, but this packed representation does not necessarily need to be carried into the persisted data structure inside the Storage Engine, where you could use a much more verbose and change tolerant data structure (such as a JSON map). Ideally the request/response model should be separate from the data storage representation.

Can I ask, is this a live game and if so how much data are we talking about when we say it would need migrated?

Currently the game is not openly live anywhere.
We had a internal pre alpha within our country where we tested how game idea performs, how did server performed and how game peformed at all. Interest of the players and working towards next milestones.
From that testing we have received a much needed feedback and in which areas we need to improve.

I can post link of the video but everything else is atm off.
Game Video.

I am well aware that data in Storage Engine doesn’t need to be tightly compressed, or to follow same structure which is later used for match logic and processing.
We intent to have it, as it simplyfies later data pulling and transformation of it in underlying system, thus having data be in the same structure across all the clients/servers/match logics is something we aim for.

As for how much data atm there are for migration not much less then 10GB and this was all from testing we had. I mean i can ditch the data and change collections to new structure, and well give some compensation to those players but that doesn’t solve the issue, as the issue may arrise again and i would be in the same situation right now.
Not maybe fully structure change, but possible of addition of new fields, keys.
Forgot to tell you: thanks for spending your time in this discussion @tom.

My suggestion would be that implementing a full data migration system may be heading down the premature optimisation route at this time, especially since the game is still in pre-alpha stage and that this time may be better spent working on solidifying a flexible domain model that can support your game long term.

I understand your concern regarding simplifying data transformation, but consider the possibilities that using the same data representation over the wire vs in storage engine would lead to your clients needing a new build every time you change the server side data representation which is not ideal.

You mentioned that in the future the structure may not fully change but that new fields or keys may be added, this in my opinion would be another reason to stick to a more verbose and change tolerant data structure in the storage engine.

We thought about that, and atm clients only need new build if new mechanic has been introduced or feature more related for displaying or something like that.
Other than that, if there are some underlying server changes if they are not mandatory, it wont be requested from clients to require new version. Even current example could be achieved not to update structure but upon match creation while decks are being fetched to do necesserly changes.
Although this case is a bit different, as we are restructuring client data model in this way to be more array like and less as key/base other prop like so change to the structure is mandatory for other multiple reasons.

On topic of upgrading json structure:
Usually from what i have seen, when json structure receives new keys, it is not a problem neither for client/server but when certain keys change value types the fun starts.
I have also seen people tend to do a mix between overwrite/merge of the structures like template based system where they use new “template” version of the collection they are working on, and then try to merge keys which can be merged. This i have seen mostly in MongoDB related documents.
This is something also what came to my mind,to have for certain collections multiple records per user for a version. The only thing i did worry about was later on data deletion. Deleting expired records, haven’t seen cron ability in nakama lua runtime, but i belive it could be probably be made with stored proceedure to run on some interval or as part of backup process.

But yeah, would like to know more if you had in your experiance with other teams a need to update a json structure within collection :slight_smile:

would like to know more if you had in your experiance with other teams a need to update a json structure within collection

We’ve not seen any clients who have needed this level of continual data migration for their titles.

@tom
Same Situation here.
When I authenticate a new user to game call this Api in nakama
const afterCreate: nkruntime.AfterHookFunction<
nkruntime.Session,
nkruntime.AuthenticateEmailRequest

= function (
ctx: nkruntime.Context,
logger: nkruntime.Logger,
nk: nkruntime.Nakama,
data: nkruntime.Session,
req: nkruntime.AuthenticateEmailRequest
)
This api i will initialized all the user collections.
example.

(works this only one time - init Player wallet, player artifacts like that…so on…)

After this, I updated my game to add a new object for this wallet and player artifact collections…
Server code updates and Collection change, Fields add … It will effect to Newly authenticated user,

how will I migrate or add this collection and wallets fields and changes to old user also…

@Albertcleetus You can solve this on multiple ways, just well how much work do you want to do :smiley:

We can divide whole process in following steps:

  1. Setting somewhere client/server/structure versions
  2. Checking does client/structures needs to be updated
  3. Converting either schemas or manuall structure changes for each structure
  4. Using either RPC or authenticate after hooks

1.1 For versioning, i found it for me it was best to make a separate file “supported_versions” in which i have:

  • Version compatibility per platform for each client
  • Account needed version
  • Wallet needed version
  • Structures needed versions

2.1) I didn’t use auth* after requests as for my game it is not important but while loading an rpc is invoked which checks client integrity and versioning. The rpc alone will fetch user account data, and from account metadata it will try to read account version and for other structures their version. If the versions do not much, a patch/upgrade module is invoked for each structure and it does updating.

Example of structure updating:

After all patching has been done, i invoke nakama multiupdate and then let player proceed with loading.

3.1) For converting schemas in Lua since i wasn’t able to find good implementation for nakama of avro schems and using that for versioning i did it manually :confused: like in screenshot above.

4.1) For Rpc starting code…

@Eatos Thanks a lot for you looking in to this.
Thanks , I didn’t want to change any flow in data initialization.

I need to implement an structure for updating data if I want,

Also set a version_number for data when writing collections, Check this versions in server from client at login or loading time, and version different call the “structure for updating data”.

I am all using in typescript ,so I need to do this in that. :grinning:

Thank you again @Eatos

I will connect again , if we had any issue with this migration… Now time to implement this.