Why does nobody implement a gameloop?

.Nostalgia · 10/27/2022, 19:09

While implementing my ECS World and sorting the Systems, I thought I would take a look at how other public sources did it. I was surprised what I found to say the least..

It looks like not a single public source has an actual gameloop/tickrate implementation. Furthermore, I don't see any kind of synchronization anywhere. Am I missing something here? Servers must be plagued by race conditions.

Another question that came up was "How do you monitor performance and make sure the server can process all the information?"

If the client runs at 30fps the server should run at 30 ticks as well to keep everything consistent, no?

Spirited · 10/27/2022, 23:33

Quote:

Originally Posted by .Nostalgia

Furthermore, I don't see any kind of synchronization anywhere. Am I missing something here? Servers must be plagued by race conditions.

Yup. Many of them have issues with race conditions or try to patch around it using locks. It's not a great design. I talked about some of my ideas around multi-threaded game server design here:

. If you have any feedback about it, I'm always interested in talking architecture.

Quote:

Originally Posted by .Nostalgia

Another question that came up was "How do you monitor performance and make sure the server can process all the information?"

I don't think a lot of people think that far ahead until they start running a server and have a big player population. That's a very small amount of servers though. For the most part... Conquer Online is just too small to really be concerned. Doesn't mean it's not a good idea, though.

Quote:

Originally Posted by .Nostalgia

If the client runs at 30fps the server should run at 30 ticks as well to keep everything consistent, no?

Hm... I don't think it needs to run that frequently. Though, depends on what you have on the clock. I personally like multiple clocks - one per map. Sorta aligns with my multi-threaded game server design with channels. You do you, though. I know a lot of servers just have a clock per action type that's global... and that seems to work just fine for the scale of Conquer.

.Nostalgia · 10/28/2022, 07:42

Quote:

Originally Posted by Spirited

Hm... I don't think it needs to run that frequently. Though, depends on what you have on the clock. I personally like multiple clocks - one per map. Sorta aligns with my multi-threaded game server design with channels.

Heh, at least 30 ticks, yes.

Quote:

Originally Posted by Spirited

Yup. Many of them have issues with race conditions or try to patch around it using locks. It's not a great design. I talked about some of my ideas around multi-threaded game server design here: . If you have any feedback about it, I'm always interested in talking architecture.

An interesting read. You raise alot of points that apply to the default OOP approach, however most of them simply do not apply with a data driven approach like ECS. There's Systems that process data based on Components. Systems themselves partition the array of entities across threads while the systems run sequentially. Accessing multiple collections for example does not matter as you index everything with the entityId, so there can't be collisions because all the arrays are pre-allocated. An ECS World contains an array of ECS Systems that contain arrays of ECS Components.

For example the incoming packets are processed before any of the systems. The packet handlers just create/update/modify components of the entity, then the systems run on those. Example:

Incoming Packets -> Input System -> Damage System -> Health System -> Death System -> Drop System -> Respawn System -> Outgoing Packets

There's no locking required, there's no allocations, the GC won't run for the lifetime of the server and most importantly, the code is clean, adding a new feature does not create cascading changes in a hundred other files. You just a new system and filter on a different set of components - maybe a new one.

The only think I'll have to syncrhonize is ECS Worlds as I am planning to have a World per Map so I can shard the server. With conquer there doesn't seem to be much synchronization in that regard. Off of the top of my head, I think it might only be chat messages and database writes. Dockerize that and create a swarm and you can scale to any amount of players without lag spikes.

My motivation behind that is keeping the operational costs low, especially with the energy crisis right now. I'd rather rent a few cheap virtual potatos (or go with container hosting) than a beefy dedicated server/

oh dear, i got carried away with my rant, its time to submit the reply..

Spirited · 10/28/2022, 10:49

Quote:

Originally Posted by .Nostalgia

Heh, at least 30 ticks, yes.

An interesting read. You raise alot of points that apply to the default OOP approach, however most of them simply do not apply with a data driven approach like ECS. There's Systems that process data based on Components. Systems themselves partition the array of entities across threads while the systems run sequentially. Accessing multiple collections for example does not matter as you index everything with the entityId, so there can't be collisions because all the arrays are pre-allocated. An ECS World contains an array of ECS Systems that contain arrays of ECS Components.

For example the incoming packets are processed before any of the systems. The packet handlers just create/update/modify components of the entity, then the systems run on those. Example:

Incoming Packets -> Input System -> Damage System -> Health System -> Death System -> Drop System -> Respawn System -> Outgoing Packets

There's no locking required, there's no allocations, the GC won't run for the lifetime of the server and most importantly, the code is clean, adding a new feature does not create cascading changes in a hundred other files. You just a new system and filter on a different set of components - maybe a new one.

The only think I'll have to syncrhonize is ECS Worlds as I am planning to have a World per Map so I can shard the server. With conquer there doesn't seem to be much synchronization in that regard. Off of the top of my head, I think it might only be chat messages and database writes. Dockerize that and create a swarm and you can scale to any amount of players without lag spikes.

My motivation behind that is keeping the operational costs low, especially with the energy crisis right now. I'd rather rent a few cheap virtual potatos (or go with container hosting) than a beefy dedicated server/

oh dear, i got carried away with my rant, its time to submit the reply..

I'm really not sold on using a game engine pattern on a game server due to unnecessary sharding and complexity. You have multiple race condition issues, especially when multiple players are involved... for example: damaging a player when that player should have zero health, but it hasn't been updated yet by the health system. You also need to have those components all be idempotent because you could trigger multiple death and drop events (all of which have to modify and check those states in parallel). If you can explain how these aren't problems, then I'm willing to listen... but I don't see how this is safer or more cost-effective / performant... especially running at 30 ticks per second.

.Nostalgia · 10/28/2022, 12:02

Quote:

Originally Posted by Spirited

I'm really not sold on using a game engine pattern on a game server due to unnecessary sharding and complexity. You have multiple race condition issues, especially when multiple players are involved... for example: damaging a player when that player should have zero health, but it hasn't been updated yet by the health system.

Each tick is a snapshot of the simulation state. It starts with processing the packets that arrived in the queue. Packet Handlers just create components. They do not do any logic beyond that.

Code:

...
                case MsgActionType.ChangeFacing:
                    {
                        var dir = new DirectionComponent(ntt.Id, msg.Direction);
                        ntt.Add(ref dir);
                        break;
                    }
                case MsgActionType.ChangeAction:
                    {
                        var emo = new EmoteComponent(ntt.Id, (Emote)msg.Param);
                        ntt.Add(ref emo);
                        break;
                    }
                case MsgActionType.Jump:
                    {
                        var jmp = new JumpComponent(ntt.Id, msg.JumpX, msg.JumpY);
                        ntt.Add(ref jmp);
                        break;
                    }
...

after all the packets have been processed, the systems run their Update() for every entity that has the required components. Eg. the Jump System using 4 threads because jumps do not affect eachother

Code:

public sealed class JumpSystem : PixelSystem<PositionComponent, JumpComponent, DirectionComponent>
    {
        public JumpSystem() : base("Jump System", threads: 4) { }
        protected override bool MatchesFilter(in PixelEntity ntt) => ntt.Type != EntityType.Item && ntt.Type != EntityType.Npc && base.MatchesFilter(in ntt);

        public override void Update(in PixelEntity ntt, ref PositionComponent pos, ref JumpComponent jmp, ref DirectionComponent dir)
        {
            if(jmp.ChangedTick == PixelWorld.Tick)
            {
                var dist = (int)Vector2.Distance(pos.Position, jmp.Position);
                var direction = CoMath.GetDirection(new Vector2(pos.Position.X, pos.Position.Y),new Vector2(jmp.Position.X, jmp.Position.Y));
                dir.Direction = direction;

                jmp.Time = CoMath.GetJumpTime(dist);
                // Console.WriteLine($"Jumping to {jmp.Position} | Dist: {dist} | Time: {jmp.Time:0.00}");
            }

            pos.Position = Vector2.Lerp(pos.Position, jmp.Position, jmp.Time);
            pos.ChangedTick = PixelWorld.Tick;
            // Console.WriteLine($"Time: {jmp.Time:0.00}");
            jmp.Time -= deltaTime;

            if(jmp.Time <= 0)
            {
                pos.Position = jmp.Position;
                ntt.Remove<JumpComponent>();
            }
        }
    }

after the jump system processed every jumping entity and moved it, the Viewport System has to update and sync visible entities - using a single thread because the quadtree is shared and not threadsafe.

Code:

 public sealed class ViewportSystem : PixelSystem<PositionComponent, ViewportComponent>
    {
        public ViewportSystem() : base("Viewport System", threads: 1) { }
        protected override bool MatchesFilter(in PixelEntity ntt) => ntt.Type != EntityType.Item && ntt.Type != EntityType.Npc && base.MatchesFilter(in ntt);

        public override void Update(in PixelEntity ntt, ref PositionComponent pos, ref ViewportComponent vwp)
        {
            if (pos.ChangedTick != PixelWorld.Tick)
                return;

            Game.Grids[pos.Map].Move(in ntt, ref pos); // one thread because of shared quadtree

            vwp.Viewport.X = pos.Position.X - vwp.Viewport.Width / 2;
            vwp.Viewport.Y = pos.Position.Y - vwp.Viewport.Height / 2;

            vwp.EntitiesVisibleLast.Clear();

            vwp.EntitiesVisibleLast.AddRange(vwp.EntitiesVisible);
            vwp.EntitiesVisible.Clear();

            Game.Grids[pos.Map].GetVisibleEntities(ref vwp);

            if (ntt.Type != EntityType.Player)
                return;

            for (var i = 0; i < vwp.EntitiesVisible.Count; i++)
            {
                var b = vwp.EntitiesVisible[i];
                // todo if visible last contains b continue

                NetworkHelper.FullSync(in ntt, in b);
            }
        }
    }

then it continues processing things like damage, health, death, drops, respawns and eventually the last step will be to sync the state to the client - on a single thread because its very trivial and fast

Code:

public sealed class NetSyncSystem : PixelSystem<NetSyncComponent>
    {
        public NetSyncSystem() : base("NetSync System", threads: 1) { }
        protected override bool MatchesFilter(in PixelEntity ntt) => ntt.Type == EntityType.Player && base.MatchesFilter(ntt);

        public override void Update(in PixelEntity ntt, ref NetSyncComponent c1)
        {
            SelfUpdate(in ntt);

            if (ntt.Type != EntityType.Player)
                return;

            ref readonly var vwp = ref ntt.Get<ViewportComponent>();

            for (var x = 0; x < vwp.EntitiesVisible.Count; x++)
            {
                var changedEntity = vwp.EntitiesVisible[x];
                Update(in ntt, in changedEntity);
            }
        }

        public static void SelfUpdate(in PixelEntity ntt) => Update(in ntt, in ntt);

        public static void Update(in PixelEntity ntt, in PixelEntity other)
        {
            ref readonly var syn = ref other.Get<NetSyncComponent>();
            ref readonly var net = ref ntt.Get<NetSyncComponent>();
            ref readonly var pos = ref ntt.Get<PositionComponent>();
            ref readonly var dir = ref ntt.Get<DirectionComponent>();
            
            if (syn.Fields.HasFlag(SyncThings.Walk))
            {
                ref readonly var wlk = ref other.Get<WalkComponent>();
                if(wlk.ChangedTick == PixelWorld.Tick)
                {
                    var walkMsg = MsgWalk.Create(other.Id, wlk.Direction, wlk.IsRunning);
                    ntt.NetSync(in walkMsg);
                }
            }
            if(syn.Fields.HasFlag(SyncThings.Jump))
            {
                ref readonly var jmp = ref other.Get<JumpComponent>();
                if(jmp.CreatedTick == PixelWorld.Tick)
                {
                    var jumpMsg = MsgAction.Create(0, ntt.Id, pos.Map, (ushort)pos.Position.X, (ushort)pos.Position.Y, dir.Direction, MsgActionType.Jump);
                    ntt.NetSync(in jumpMsg);
                }
            }
            if (syn.Fields.HasFlag(SyncThings.Health) || syn.Fields.HasFlag(SyncThings.MaxHealth))
            {
                ref readonly var hlt = ref other.Get<HealthComponent>();
                if(hlt.ChangedTick == PixelWorld.Tick)
                {
                    var healthMsg = MsgUserAttrib.Create(ntt.Id, hlt.Health, MsgUserAttribType.Health);
                    var maxHealthMsg = MsgUserAttrib.Create(ntt.Id, hlt.MaxHealth, MsgUserAttribType.MaxHealth);
                    ntt.NetSync(in healthMsg);
                    ntt.NetSync(in maxHealthMsg);
                }
            }
            if (syn.Fields.HasFlag(SyncThings.Level))
            {
                ref readonly var lvl = ref other.Get<LevelComponent>();
                if(lvl.ChangedTick == PixelWorld.Tick)
                {
                    var lvlMsg = MsgUserAttrib.Create(ntt.Id, lvl.Level, MsgUserAttribType.Level);
                    ntt.NetSync(in lvlMsg);
                }
            }
            if (syn.Fields.HasFlag(SyncThings.Experience))
            {
                ref readonly var exp = ref other.Get<ExperienceComponent>();
                if(exp.ChangedTick == PixelWorld.Tick)
                {
                    var expMsg = MsgUserAttrib.Create(ntt.Id, exp.Experience, MsgUserAttribType.Experience);
                    ntt.NetSync(in expMsg);
                }
            }
        }
    }

Quote:

... due to unnecessary sharding and complexity.

It is alot less complex than comet. When I read comet code I immediately thought that its convoluted and complicated. Mind you I am a senior dev with 15 years of experience, i do not doubt you know what you are doing and I'm also sure it works well, but its doing everything in a very smart way instead of a very simple one. It takes alot of effort to read and understand the code. With ECS you have KISS absolutism. Only Systems modify state and each system has a single job. You know where everything happens. Something is not syncing to the client? NetSyncSystem. Items do not drop? Drop System. Drops do not despawn? LifetimeSystem.

You work in a single file most of the time. Components contain no logic, just data and are immutable most of the time, for example

Code:

    [Component]
    public struct DirectionComponent
    {
        public readonly int EntityId;
        public uint ChangedTick;
        public Direction Direction;

        public DirectionComponent(int entityId, Direction direction = Direction.South)
        {
            EntityId = entityId;
            Direction = direction;
            ChangedTick = PixelWorld.Tick;
        }

        public override int GetHashCode() => EntityId;
    }

So the only position where a bug can appear is inside a system.

Sharding comes for free. Worlds are self-contained. Take it or leave it, if you need it its available with near zero effort. Performance profiling comes for free. Systems can be timed, you can know where your bottleneck is without 3rd party tools. There's no allocations during runtime (other than strings) so the GC rarely runs and when it does its very quick. You do not need any timers as you always have the gametime reference and the delta times. You can even throttle systems and skip ticks easily if you need more performance, eg. updating the viewport on every even or odd tick only.

I'll leave you with an interesting talk about ECS for Roguelikes, its about simulating worlds and less obvious things - eg complex behaviors with component reuse and remixing.

Spirited · 10/28/2022, 21:30

I don't see how this is less complicated than a queue of actions per map... and the race condition possibilities with inter-dependencies between systems for multiple players terrifies me... but I don't have the patience to vet this fully with you. I'll let others chime in with their opinions; though, it sounds like your mind was already made up before asking why nobody implements this on the server side.

Envy\Craft · 10/28/2022, 22:07

Quote:

Originally Posted by .Nostalgia

While implementing my ECS World and sorting the Systems, I thought I would take a look at how other public sources did it. I was surprised what I found to say the least..

It looks like not a single public source has an actual gameloop/tickrate implementation. Furthermore, I don't see any kind of synchronization anywhere. Am I missing something here? Servers must be plagued by race conditions.

Another question that came up was "How do you monitor performance and make sure the server can process all the information?"

If the client runs at 30fps the server should run at 30 ticks as well to keep everything consistent, no?

when you say "other public sources". are you talking about all those leaked private servers that pretty much derived from the same poorly leaked source?

there is really only like 3 or so servers that are long running that's not binary, and there is a reason for this.

most of our clients run at 500 fps... just saying :P

iBotx · 10/28/2022, 22:38

I believe that most of the so called "developers" around the conquer community doesnt even know what Synchronization and race conditions means. Thats why most of the sources out there have duping issues and much more amazing issues.

There is much more worse than just neglecting synchronization in the public conquer sources.

.Nostalgia · 10/29/2022, 08:01

Quote:

Originally Posted by Spirited

but I don't have the patience to vet this fully with you. I'll let others chime in with their opinions; though, it sounds like your mind was already made up before asking why nobody implements this on the server side.

Sounds like you're in a bad mood. I asked about gameloops and our conversation switched to why I use ECS. You could have a gameloop without ECS. Minecraft has a gameloop, the open source world of warcraft private server, Battlefield servers, .... The conquer community is the outlier here.

Quote:

when you say "other public sources". are you talking about all those leaked private servers that pretty much derived from the same poorly leaked source?

there is really only like 3 or so servers that are long running that's not binary, and there is a reason for this.

most of our clients run at 500 fps... just saying :P

I have seen quite a few that look very unique. comet, cops, coemu, redux, throne, ...

Quote:

I believe that most of the so called "developers" around the conquer community doesnt even know what Synchronization and race conditions means. Thats why most of the sources out there have duping issues and much more amazing issues.

There is much more worse than just neglecting synchronization in the public conquer sources.

Thats slightly terrifying

Spirited · 10/29/2022, 18:55

Quote:

Originally Posted by .Nostalgia

Sounds like you're in a bad mood. I asked about gameloops and our conversation switched to why I use ECS. You could have a gameloop without ECS. Minecraft has a gameloop, the open source world of warcraft private server, Battlefield servers, .... The conquer community is the outlier here.

I have seen quite a few that look very unique. comet, cops, coemu, redux, throne, ...

Thats slightly terrifying

Not in a bad mood, I simply don't owe you my time.

I'm not going to step through every system with you and tell you every little problem wrong with your specific implementation of that design. Especially when it sounds like you already made up your mind. I have a full-time job, family, and list of projects I work on. So I'm going to let other people participate on a forum that's not just me.

KraHen · 10/31/2022, 10:11

I did in my last project. It works fine, no ECS though, so can't talk about that part. In my case all the entities were responsible for updating themselves, and that scaled across multiple threads wonderfully. Of course it has its own particularities but what doesn't.

It's perfectly viable, and not recommended for people who haven't really worked with game dev itself and concurrency in games/realtime apps IMO, but if that's where your experience is from, you can absolutely leverage it like this.

Regarding sharding - don't, unless it's for educational purposes. Your load will never be high enough in this context to justify the added complexity.

.Nostalgia · 03/25/2023, 21:35

3 days running on a oracle cloud vm (free tier). I really have to get the GC under control now

Can be tested at 140.238.170.160, any 5018 client works. Just login and play, acc get created automatically. There's no persistance. Upon server restart it resets everything.

There are no damage calculations yet, most things do not work.

Commands (if you crash the server with them, it restarts automatically)

PHP Code:


			
                case "cc":
                    ref var pos = ref ntt.Get<PositionComponent>();
                    pos.Position.X = ushort.Parse(args[0]);
                    pos.Position.Y = ushort.Parse(args[1]);
                    pos.Map = ushort.Parse(args[2]);
                    var tpc = new TeleportComponent((ushort)pos.Position.X, (ushort)pos.Position.Y, (ushort)pos.Map);
                    ntt.Set(ref tpc);
                    break;
                case "rev":
                    var rev = new ReviveComponent(1);
                    ntt.Set(ref rev);
                    break;
                case "eff":
                    var eff = MsgName.Create(ntt.Id, args[0], MsgNameType.RoleEffect);
                    ntt.NetSync(ref eff, true);
                    break;

L1nk1n*P4rK · 03/31/2023, 22:51

Quote:

Originally Posted by .Nostalgia

3 days running on a oracle cloud vm (free tier). I really have to get the GC under control now

Can be tested at 140.238.170.160, any 5018 client works. Just login and play, acc get created automatically. There's no persistance. Upon server restart it resets everything.

There are no damage calculations yet, most things do not work.

Commands (if you crash the server with them, it restarts automatically)

PHP Code:

case "cc": ref var pos = ref ntt.Get<PositionComponent>(); pos.Position.X = ushort.Parse(args[0]); pos.Position.Y = ushort.Parse(args[1]); pos.Map = ushort.Parse(args[2]); var tpc = new TeleportComponent((ushort)pos.Position.X, (ushort)pos.Position.Y, (ushort)pos.Map); ntt.Set(ref tpc); break; case "rev": var rev = new ReviveComponent(1); ntt.Set(ref rev); break; case "eff": var eff = MsgName.Create(ntt.Id, args[0], MsgNameType.RoleEffect); ntt.NetSync(ref eff, true); break;

One of the things I always wanted to do and constantly think about is exactly what you did here.

Nice Grafana/DataDog dashboard showing you data just like in your screenshot.

At one point I even played around with an Auditing system (trades, sales, vendor interactions, movement, etc) that then evolved in some more "exotic" things like "Replay System" based on the audit trails.
Even more, at one point I was playing with graphs showing "gold path" from the moment it dropped from a mob, to all the trades (including individual name), to the moment it exited the economy (Something very close to how Google Analytics shows traffic on web pages). This also evolved (again, lol) to more "Economical" dashboard, showing things such as Total Gold, Inflation/Deflation, etc.

But you know what, you made me smile with this screenshot

Maybe you can get some ideas from what I wrote above for some clever systems!

Keep them coming!

PS: I did all of this, not for a server, but for myself to practice a bit some services I don't get to use at work (or rarely use).

.Nostalgia · 04/01/2023, 13:47

Quote:

Originally Posted by L1nk1n*P4rK

One of the things I always wanted to do and constantly think about is exactly what you did here.

Nice Grafana/DataDog dashboard showing you data just like in your screenshot.

At one point I even played around with an Auditing system (trades, sales, vendor interactions, movement, etc) that then evolved in some more "exotic" things like "Replay System" based on the audit trails.
Even more, at one point I was playing with graphs showing "gold path" from the moment it dropped from a mob, to all the trades (including individual name), to the moment it exited the economy (Something very close to how Google Analytics shows traffic on web pages). This also evolved (again, lol) to more "Economical" dashboard, showing things such as Total Gold, Inflation/Deflation, etc.

But you know what, you made me smile with this screenshot Maybe you can get some ideas from what I wrote above for some clever systems!

Keep them coming!

PS: I did all of this, not for a server, but for myself to practice a bit some services I don't get to use at work (or rarely use).

Hah, i didnt screenshot that part. Didn't think anyone would care other than me.