Help with timings

[Daaku]

Hey everyone,

I recently set up my minecraft server (beta 3 months ago. release 13 days ago).
Problem is that we don't have the most stable server. Our average TPS is about 19.8 but we just can't ever hit 20tps. I'd like to have some help with my server's timings. I would greatly appreciate it and also offer 50$ after it's fixed and can run a nice 20tps. which I really think is possible. The server used to run a solid 20tps 2 weeks ago hence why I really suspect crazy auctions.

Disclaimer:
Yes I'm using Skript, Skript is essential for us and ye we run like 50 different skripts. However most of these are inactive for 99.9% of the time. Only 1 skript is constantly doing stuff which is 1 thing a second per player.
I'm pretty good with skript so most stuff is as lag free and efficient as it can be. Our gamemode is completely custom so we can't do without. (And I dont know java and neither am I willing to hire developers)

Specs:
Before someone tells me to upgrade my server. Here are my server's specs. I think this shouldn't be much of a problem at this point (only running like 110 ppl at peak)
CPU: Intel Xeon E5-1650v2
Ram: 128GB DDR3 ECC (126 allocated to the server rn. Might wanna run multiple smaller ones like a hub later with like 8GB)
Storage: 2x 2TB HDD (In a raid 0 configuration)

We are running paper spigot 1.12.2
Our world where people can build is running at 1 Random Tickspeed, The others are at 0

Our problem:

So we have 2 problems with the server.
Like I said we run a stable 19.8tps (or 19.9 as of writing this with 40 people online) however this often dips for a couple seconds as something lags it and the server tries to recover. There's 1 known problem which Id like help on and 1 I suspect.

Timings:
25-30 players: https://timings.aikar.co/?id=af4db11b71ca4168be080c9289c3e76a
75-85 players: https://timings.aikar.co/?id=1b78f749a4bf4f48afd17ed28f691dde
80-100 players: https://timings.aikar.co/?id=0bcbc58d1b64401fa76ba5b6d1e698fa
1. Average TPS
So our average tps as said there ^ isn't exactly 20. It runs at 11.7ms/tick right now on average (total, all) but in total, lag it shows 86.61ms/tick. It says crazy auctions is 42.44ms/tick on this (0.75x per tick, 56.65 per).
We have about 2-3 pages of auctions at any time and I'm wondering if this plugin is just really laggy and if there is any alternative. If not should I make my own (well... in skript)
If this isn't the main culprit what else could it be?

2. FAWE / WorldEdit
On our server we have a very custom gamemode. Part of this gamemode is that every faction has it's own plot. These plots are well over 2.5 million blocks if we exclude air. FAWE tries to paste this all in at the same time and excludes the air of it. Still this drops our server TPS heavily for a second whenever a plot is generated. (over 350 of them have been made so far in 2 weeks... so ye) Is there any way to make FAWE/WorldEdit do this piece by piece and not try to paste EVERYTHING at once?

If so where can I change this?

Fawe config:

# These first 6 aren't configurable
issues: "https://github.com/boy0001/FastAsyncWorldedit/issues"
wiki: "https://github.com/boy0001/FastAsyncWorldedit/wiki/"
date: "1 Feb 2019 00:00:00 GMT"
build: "https://ci.athion.net/job/FastAsyncWorldEdit/1244"
commit: "https://github.com/boy0001/FastAsyncWorldedit/commit/56a71fa"
platform: "bukkit"
# Options: cn, de, es, fr, it, nl, ru, tr
# Create a PR to contribute a translation: https://github.com/boy0001/FastAsyncWorldedit/new/master/core/src/main/resources
language: ''
# Enable or disable automatic updates
#  - true = update automatically in the background
#  - confirm = prompt an admin to confirm each update
#  - false = do not update the plugin
update: "false"
# Send anonymous usage statistics
metrics: true
# Set true to enable WorldEdit restrictions per region (e.g. PlotSquared or WorldGuard).
# To be allowed to WorldEdit in a region, users need the appropriate
# fawe.<plugin>  permission. See the Permissions page for supported region plugins.
region-restrictions: true
# FAWE will skip chunks when there's not enough memory available
prevent-crashes: false
# FAWE will cancel non admin edits when memory consumption exceeds this %
#  - Bypass with `/wea` or `//fast` or `fawe.bypass`
#  - Disable with 100 or -1.
max-memory-percent: 95

clipboard:
  # Store the clipboard on disk instead of memory
  #  - Will be slightly slower
  #  - Uses 2 bytes per block
  use-disk: true
  # Compress the clipboard to reduce the size:
  #  - TODO: Buffered random access with compression is not implemented on disk yet
  #  - 0 = No compression
  #  - 1 = Fast compression
  #  - 2-17 = Slower compression
  compression-level: 1
  # Number of days to keep history on disk before deleting it
  delete-after-days: 1

lighting:
  # If packet sending should be delayed until relight is finished
  delay-packet-sending: true
  async: true
  # The relighting mode to use:
  #  - 0 = None (Do no relighting)
  #  - 1 = Optimal (Relight changed light sources and changed blocks)
  #  - 2 = All (Slowly relight every blocks)
  mode: 1
  # If existing lighting should be removed before relighting
  remove-first: false

# Generic tick limiter (not necessarily WorldEdit related, but useful to stop abuse)
tick-limiter:
  # Enable the limiter
  enabled: true
  # The interval in ticks
  interval: 20
  # Max falling blocks per interval (per chunk)
  falling: 300
  # Max physics per interval (excluding redstone)
  physics-ms: 10
  # Max item spawns per interval (per chunk)
  items: 256
  # Whether fireworks can load chunks
  #  - Fireworks usually travel vertically so do not load any chunks
  #  - Horizontal fireworks can be hacked in to crash a server
  fireworks-load-chunks: false

web:
  # Should download urls be shortened?
  #  - Links are less secure as they could be brute forced
  shorten-urls: false
  # The web interface for clipboards
  #  - All schematics are anonymous and private
  #  - Downloads can be deleted by the user
  #  - Supports clipboard uploads, downloads and saves
  url: "https://empcraft.com/fawe/"
  # The web interface for assets
  #  - All schematics are organized and public
  #  - Assets can be searched, selected and downloaded
  assets: "https://empcraft.com/assetpack/"

extent:
  # Don't bug console when these plugins slow down WorldEdit operations
  #  - You'll see a message in console if you need to change this option
  allowed-plugins: []
  # Should debug messages be sent when third party extents are used?
  debug: true

# Experimental options, use at your own risk
#  - UNSAFE = Can cause permanent damage to the server
#  - SAFE = Can be buggy but unlikely to cause any damage
experimental:
  # [UNSAFE] Directly modify the region files. (OBSOLETE - USE ANVIL COMMANDS)
  #  - IMPROPER USE CAN CAUSE WORLD CORRUPTION!
  anvil-queue-mode: false
  # [SAFE] Dynamically increase the number of chunks rendered
  #  - Requires Paper: ci.destroystokyo.com/job/PaperSpigot/
  #  - Set your server view distance to 3 (spigot.yml, server.properties)
  #  - Based on tps and player movement
  #  - Please provide feedback
  dynamic-chunk-rendering: -1
  # [SAFE] Allows brushes to be persistent
  persistent-brushes: false
  # [SAFE] Enable CUI without needing the mod installed (Requires ProtocolLib)
  vanilla-cui: false
  # Disable using native libraries
  disable-natives: false
  # [SAFE] Keep entities that are positioned in non-air blocks when editing an area
  # Might cause client-side FPS lagg in some situations
  keep-entities-in-blocks: false
  # [SAFE] Experimental scripting support for Java 9
  #  -
  modern-craftscripts: false
  # [SAFE] Experimental freebuild region restrictions
  #  - PERM: fawe.freebuild
  #  - PERM: fawe.freebuild.<plugin>
  freebuild: false

# This relates to how FAWE places chunks
queue:
  # This should equal the number of processors you have
  #  - Set this to 1 if you need reliable `/timings`
  parallel-threads: 6
  progress:
    # Display constant titles about the progress of a user's edit
    #  - false = disabled
    #  - title = Display progress titles
    #  - chat = Display progress in chat
    display: "false"
    # How often edit progress is displayed
    interval: 1
    # Delay sending progress in milliseconds (so quick edits don't spam)
    delay: 5000
  # When doing edits that effect more than this many chunks:
  #  - FAWE will start placing before all calculations are finished
  #  - A larger value will use slightly less CPU time
  #  - A smaller value will reduce memory usage
  #  - A value too small may break some operations (deform?)
  target-size: 64
  # Force FAWE to start placing chunks regardless of whether an edit is finished processing
  #  - A larger value will use slightly less CPU time
  #  - A smaller value will reduce memory usage
  #  - A value too small may break some operations (deform?)
  max-wait-ms: 1000
  # Increase or decrease queue intensity (ms) [-50,50]:
  #     0 = balance of performance / stability
  #     -10 = Allocate 10ms less for chunk placement
  # Too high will can cause lag spikes (you might be okay with this)
  # Too low will result in slow edits
  extra-time-ms: 0
  # Loading the right amount of chunks beforehand can speed up operations
  #  - Low values may result in FAWE waiting on requests to the main thread
  #  - Higher values use more memory and isn't noticeably faster
  preload-chunks: 32
  # Discard edits which have been idle for a certain amount of time (ms)
  #  - E.g. A plugin creates an EditSession but never does anything with it
  #  - This only applies to plugins improperly using WorldEdit's legacy API
  discard-after-ms: 60000

history:
  # Should history be saved on disk:
  #  - Frees up a lot of memory
  #  - Persists restarts
  #  - Unlimited undo
  #  - Does not affect edit performance if `combine-stages`
  use-disk: true
  # Use a database to store disk storage summaries:
  #  - Enables inspection and rollback
  #  - Does not impact performance
  use-database: true
  # Record history with dispatching:
  #  - Much faster as it avoids duplicate block checks
  #  - Slightly worse compression since dispatch order is different
  combine-stages: true
  # Higher compression reduces the size of history at the expense of CPU
  # 0 = Uncompressed byte array (fastest)
  # 1 = 1 pass fast compressor (default)
  # 2 = 2 x fast
  # 3 = 3 x fast
  # 4 = 1 x medium, 1 x fast
  # 5 = 1 x medium, 2 x fast
  # 6 = 1 x medium, 3 x fast
  # 7 = 1 x high, 1 x medium, 1 x fast
  # 8 = 1 x high, 1 x medium, 2 x fast
  # 9 = 1 x high, 1 x medium, 3 x fast (best compression)
  # NOTE: If using disk, do some compression (3+) as smaller files save faster
  compression-level: 3
  # The buffer size for compression:
  #  - Larger = better ratio but uses more upfront memory
  #  - Must be in the range [64, 33554432]
  buffer-size: 531441
  # The maximum time in milliseconds to wait for a chunk to load for an edit.
  #  (50ms = 1 server tick, 0 = Fastest).
  #  The default value of 100 should be safe for most cases.
  #
  # Actions which require loaded chunks (e.g. copy) which do not load in time
  #  will use the last chunk as filler, which may appear as bands of duplicated blocks.
  # Actions usually wait about 25-50ms for the chunk to load, more if the server is lagging.
  # A value of 100ms does not force it to wait 100ms if the chunk loads in 10ms.
  #
  # This value is a timeout in case a chunk is never going to load (for whatever odd reason).
  # If the action times out, the operation continues by using the previous chunk as filler,
  #  and displaying an error message.  In this case, either copy a smaller section,
  #  or increase chunk-wait-ms.
  # A value of 0 is faster simply because it doesn't bother loading the chunks or waiting.
  chunk-wait-ms: 1000
  # Delete history on disk after a number of days
  delete-after-days: 7
  # Delete history in memory on logout (does not effect disk)
  delete-on-logout: true
  # If history should be enabled by default for plugins using WorldEdit:
  #  - It is faster to have disabled
  #  - Use of the FAWE API will not be effected
  enable-for-console: true
  # Should redo information be stored:
  #  - History is about 20% larger
  #  - Enables use of /redo
  store-redo: true
  # Assumes all edits are smaller than 4096x256x4096:
  #  - Reduces history size by ~10%
  small-edits: false

# Paths for various directories
paths:
  tokens: "tokens"
  # Put any minecraft or mod jars for FAWE to be aware of block textures
  textures: "textures"
  heightmap: "heightmap"
  history: "history"
  # Multiple servers can use the same clipboards
  #  - Use a shared directory or NFS/Samba
  clipboard: "clipboard"
  # Each player has their own sub directory for schematics
  per-player-schematics: true
  commands: "commands"

# Region restriction settings
region-restrictions-options:
  # What type of users are allowed to WorldEdit in a region
  #  - MEMBER = Players added to a region
  #  - OWNER = Players who own the region
  mode: "MEMBER"
# The "default" limit group affects those without a specific limit permission.
# To grant someone different limits, copy the default limits group
# and give it a different name (e.g. newbie). Then give the user the limit
# permission node with that limit name (e.g. fawe.limit.newbie  )
limits:
  default:
    # Max actions that can be run concurrently (i.e. commands)
    max-actions: 1
    # Max number of block changes (e.g. by `//set stone`).
    max-changes: 50000000
    # Max number of blocks checked (e.g. `//count stone` which doesn't change blocks)
    max-checks: 50000000
    # Number of times a change can fail (e.g. if the player can't access that region)
    max-fails: 50000000
    # Allowed brush iterations (e.g. `//brush smooth`)
    max-iterations: 1000
    # Max allowed entities (e.g. cows)
    max-entities: 1337
    # Blockstates include Banner, Beacon, BrewingStand, Chest, CommandBlock,
    # CreatureSpawner, Dispenser, Dropper, EndGateway, Furnace, Hopper, Jukebox,
    # NoteBlock, Sign, Skull, Structure
    max-blockstates: 1337
    # Maximum size of the player's history in Megabytes:
    #  - History on disk or memory will be deleted
    max-history-mb: -1
    # Maximum time in milliseconds //calc can execute
    max-expression-ms: 50
    # Cinematic block placement:
    #  - Adds a delay to block placement (ms/block)
    #  - Having an artificial delay will use more CPU/Memory
    speed-reduction: 0
    # Place chunks instead of individual blocks:
    #  - Disabling this will negatively impact performance
    #  - Only disable this for compatibility or cinematic placement
    fast-placement: true
    # Should WorldEdit use inventory?
    # 0 = No inventory usage (creative)
    # 1 = Inventory for removing and placing (freebuild)
    # 2 = Inventory for placing (survival)
    inventory-mode: 0
    # Should large edits require confirmation (>16384 chunks)
    confirm-large: true
    # List of blocks to strip nbt from
    strip-nbt: []

Worldedit config:

#
# WorldEdit's Configuration File
#
# About editing this file:
# - DO NOT USE TABS. You MUST use spaces or Bukkit will complain and post
#   errors. If you use an editor, like Notepad++ (recommended for Windows
#   users), you must configure it to "replace tabs with spaces."
#   This can be changed in Settings > Preferences > Language Menu.
# - Don't get rid of indentations. They are indented so some entries that are
#   in categories, like "max-blocks-changed", are placed in the "limits"
#   category.
# - If you want to check the format of this file before putting it
#   into WorldEdit, paste it into http://yaml-online-parser.appspot.com/
#   and see if it gives you "ERROR:".
# - Lines starting with # are comments, so they are ignored.
# - If you want to allow blocks, make sure to change "disallowed-blocks" to []
#

limits:
    allow-extra-data-values: false
    max-blocks-changed:
        default: -1
        maximum: -1
    max-polygonal-points:
        default: -1
        maximum: 20
    max-radius: -1
    max-super-pickaxe-size: 5
    max-brush-radius: 5
    butcher-radius:
        default: -1
        maximum: -1
    disallowed-blocks: [6, 7, 14, 15, 16, 26, 27, 28, 29, 39, 31, 32, 33, 34, 36, 37, 38, 39, 40, 46, 50, 51, 56, 59, 69, 73, 74, 75, 76, 77, 81, 83, 137, 210, 211, 255]

use-inventory:
    enable: false
    allow-override: true
    creative-mode-overrides: false

logging:
    log-commands: false
    file: worldedit.log
    # The format of custom log message. This is java general format string (java.util.Formatter). Arguments are:
    #   1$ : date - a Date object representing event time of the log record.
    #   2$ : source - a string representing the caller, if available; otherwise, the logger's name.
    #   3$ : logger - the logger's name.
    #   4$ : level - the log level.
    #   5$ : message - the formatted log message returned from the Formatter.formatMessage(LogRecord) method. It uses java.text formatting and does not use the java.util.Formatter format argument.
    #   6$ : thrown - a string representing the throwable associated with the log record and its backtrace beginning with a newline character, if any; otherwise, an empty string.
    # For details see:
    #   https://docs.oracle.com/javase/8/docs/api/java/util/Formatter.html
    #   https://docs.oracle.com/javase/8/docs/api/java/util/logging/SimpleFormatter.html#format-java.util.logging.LogRecord-
    format: "[%1$tY-%1$tm-%1$td %1$tH:%1$tM:%1$tS %4$s]: %5$s%6$s%n"

super-pickaxe:
    drop-items: true
    many-drop-items: false

snapshots:
    directory:

navigation-wand:
    item: 999
    max-distance: 0

scripting:
    timeout: 3000
    dir: craftscripts

saving:
    dir: schematics

files:
    allow-symbolic-links: false

history:
    size: 15
    expiration: 10

wand-item: 271
shell-save-type:
no-double-slash: false
no-op-permissions: false
debug: false
show-help-on-first-use: true

We are speaking about a 536.77kB schematic being pasted a lot.

 

[Daaku]

FAWE usually doesn't cause lag on decent hardware, you're fine here.

I've used crazyauctions before, it does cause a bit of lag but you should be fine... Have you optimized spigot,bukkit,taco,paper.yml? If not, I'd recommend doing that. https://www.spigotmc.org/threads/guide-server-optimization.283181/

I have nothing else to say, if you can, please do /wg profile -p 1 And send it over

problem is that with fawe im pasting millions of blocks at once

 

[Daaku]

FAWE usually doesn't cause lag on decent hardware, you're fine here.

I've used crazyauctions before, it does cause a bit of lag but you should be fine... Have you optimized spigot,bukkit,taco,paper.yml? If not, I'd recommend doing that. https://www.spigotmc.org/threads/guide-server-optimization.283181/

I have nothing else to say, if you can, please do /wg profile -p 1 And send it over

theres my wg profiling

 

[Aabes]

Update all plugins, stress the server with bots before releasing.
Create a small test server adding each plugin in at a time see what causes any lag.
Check for any memory leaks

 

[Daaku]

Does anyone know any less laggy alternative to Crazy Auctions? Else Ill go write my own I guess. But would be useful if someone already did it.

 

[Managering]

Hi,

You might want to see if Auction house works. I used to love it.

Be aware, theres a chunk duplication bug with paper 12.2.2 right now I believe, they're busy on resolving it.

If you'd like I'm up for hire to assist resolving your issues.

 

[Daaku]

Hi,

You might want to see if Auction house works. I used to love it.

Be aware, theres a chunk duplication bug with paper 12.2.2 right now I believe, they're busy on resolving it.

If you'd like I'm up for hire to assist resolving your issues.

chunk duplication bug is vanilla mc. paper spigot are the only ones I know actually fixing it up with mojang. But the chunk duplication bug is something from vanilla mc world saving. I already have this patched by disabling books all together

 

[Juggled]

Add me on discord @TooToo#6890