Seeking Advice (or Service) on Data Synchronization Across Cloned Skyblock Servers

[Kraapy]

Hello everyone,

I'm in the process of setting up multiple cloned Skyblock servers and am exploring the best practices for synchronizing player data across these environments. I am particularly focused on ensuring consistent updates for player inventories, Ender Chest contents, economic details, experience levels, potion effects, and more. While I have looked into plugins like HuskSync and TAB for managing features such as tab completion, tablists, chat, and teleport (tpa) commands, it seems that the most successful multi-server setups don't use these tools. I'm curious about the alternative methods or systems they might be employing.

Additionally, I'm interested in understanding how these setups manage player islands across different servers—specifically, how they determine on which server a player's island should be placed.

Could anyone with experience in these areas share their insights or point me towards resources that could help? I'm keen on understanding both the technical setup and any challenges I might face in implementing such synchronization.

Thanks in advance for your help!

 

[JimSpecter]

Hello everyone,

I'm in the process of setting up multiple cloned Skyblock servers and am exploring the best practices for synchronizing player data across these environments. I am particularly focused on ensuring consistent updates for player inventories, Ender Chest contents, economic details, experience levels, potion effects, and more. While I have looked into plugins like HuskSync and TAB for managing features such as tab completion, tablists, chat, and teleport (tpa) commands, it seems that the most successful multi-server setups don't use these tools. I'm curious about the alternative methods or systems they might be employing.

Additionally, I'm interested in understanding how these setups manage player islands across different servers—specifically, how they determine on which server a player's island should be placed.

Could anyone with experience in these areas share their insights or point me towards resources that could help? I'm keen on understanding both the technical setup and any challenges I might face in implementing such synchronization.

Thanks in advance for your help!

We have experienced people who can help you on that, just join our server and create a support ticket. You can click here to join our discord server.

We look forward to potentially help you out with this.

 

[ImIllusion]

Hey there, I've been in the industry for a while and have both worked at big servers that use this kind of system, as well as written it on my own.

Sharding and synchronization is a complex topic, so I'll discuss multiple approaches.
Assuming you're writing it all from scratch, we need to identify the following problem areas:

• Deciding where a player's island should be loaded
• How data flows when a player transfers between instances
• What happens when a player quits the server.

For the island loading part, given this is a sharded system, there are two main approaches:

• Have a dedicated service app running that decides it all (Separation of concerns is nice)
• Have the service embedded in your plugin, either in the instance of proxy (Requires no additional setup for the end user)

The dedicated service app approach allows you to have a central off-site place to perform your decision-making as long as it has context of every instance that is available and its load (how many players / islands is it managing), which allows you to implement a "strategy" to pick which server to use (the one with the least players / least islands / most uptime), as well as knowing who's on what server (for a /tpa request, for example). When developing one, you will need to consider certain factors, such as hard server crashes, getting existing data in case the service restarts and handling multiple requests at once. In my opinion, this is the better approach.

Having the service embedded in your plugin can lead to weird synchronization issues but it is also a solid approach if you're planning to sell your project to as many people as possible.

Now that island picking is out of the way, let's talk about data flowing between instances:

• Data should be bound to a lifecycle (also referred to as scope), such as the player's own (Data is only loaded while the player is loaded), or time (kept in a cache for 30 minutes)
• When a player requests a server transfer, the player's current data should be saved (both to a low-latency cache and persistent storage), as well as transferred to the target server BEFORE the player is sent (and held in the target server for ~20 seconds in case the player times out). This ensures the player's data is loaded before the player joins.
• When the player leaves the network, the data is kept in cache for a little bit before being saved to the database.

Given we're talking about islands, the island should be loaded for as long as the data is in memory or in cache. Caching data for a limited time (~30 minutes) allows us to not waste performance saving and loading islands or their data if the player is relogging.

If you have any questions feel free to post them below or reach out on discord at illusion_dev

 

[Kraapy]

Hi ImIllusion,

Thank you for the very comprehensive response—it has significantly helped clarify things for me! Since you've worked on similar projects, I have a few follow-up questions:

1. Implementation Time: How long does it typically take to set up?
2. Memory Allocation: Do you usually reduce the memory allocated to each server, perhaps to 6-8GB instead of 12+GB?
3. Performance Impact: Does this whole process is worth enough ?

Thanks again for your guidance!

 

[ImIllusion]

Hi ImIllusion,

Thank you for the very comprehensive response—it has significantly helped clarify things for me! Since you've worked on similar projects, I have a few follow-up questions:

1. Implementation Time: How long does it typically take to set up?
2. Memory Allocation: Do you usually reduce the memory allocated to each server, perhaps to 6-8GB instead of 12+GB?
3. Performance Impact: Does this whole process is worth enough ?

Thanks again for your guidance!

Hey there,

To answer your questions:

• The implementation time varies from project to project. For small-scale servers the usual implementation time is under a week. This varies with complexity
• When specifying memory allocation we need to have a target usage in mind. My usual goal with instances is to handle 125-150 players max, 8gb per instance is fine.
• At a super small scale (where 1 instance would suffice) there is no return for this added complexity. The whole objective of a sharded structure is to increase the maximum capacity of a given system, not to make it more efficient (efficiency influences max capacity too, but at a lesser extent than having more machines)