Storage engine best practices for large collections

Game Design
,
1

Hi,

We are exploring what’s the best way to store data in the database. Our use case is a realtime city builder where players can spawn instances of things on the same space. Each player can add from 50-100 to 1500-2000 instances that need to be recorded in the DB.

Recording happens in the server runtime and involves storing a set of numbers (position, rotation, scale) and a single string (instance name).

We created an RPC that is called after each instance is being spawned and adds the data to the collection using a unique key that is derived from the position coordinates (e.g. x_y_z). That works well but we are worried it may not scale. When a new player will visit the game he will request all these instances using a storageList method.

That method will fetch all instances built by the players and return them but seeing how much metadata each instance holds (permissions, userId, version, key etc) that seems like a lot of data.

We thought another scenario to put everything under a single key and on each write have the server read the current entries on that key (e.g. that would be a big sequential array) and append at the end the new instance data. Though that seems like a lot of work for the server.

Any ideas on what approach works best for us?

2

@Leonidas can you describe in more detail why you need to represent the state of the world in the form of a linked list datastructure? Do you need to know the sequence that these mutations happened to the game world or can you compact them down onto the current representation of the world state?

3

Thanks @novabyte for your reply. Actually we don’t need a linked list or anything, each world change can live on its own in the DB. We are just concerned if what we are doing may flood the DB with a lot of entries.

Here is a sneak peak of what players are allowed to do:

Each item write has a unique key based on its world position (coordX_coordY_coordZ) and the following data:

  "data": [
    0,
    0,
    9.89,
    1.8,
    -164.45,
    0,
    25,
    0,
    0.8
  ]

Here is how we store them right now:

image
image1268×530 23.1 KB

A collection is created and is named based on the area of the world the player currently is (0_1), and the item name (Bush, Palm etc).

Let us know if we can continue using this format. Many thanks.

4

@Leonidas I think this works well enough but I have a few suggestions:

  • I think you can make the stored data for segments of the world map larger. They’re quite fine grained right now whereas they could contain more of the world. Don’t try mirror the level of slicing the world you have with the game client with the size of data objects you’ve got that represent parts of the world in the storage engine.
  • The client must know enough about the slices of the world map that need to be rendered right? This makes me think that you should be able to fetch those portions of the world with a batch get rather than storage listings. This will give you a better performance time on the network requests to obtain that data.

Hope this helps.

1 Like
5

Got it, that makes sense, we will explore that.

Yes indeed, the client knows exactly the slices of the world map it requires. Do you have an example of a batch get like that?

Many thanks!