Game Server Architecture

Research & development for the PSForever architecture.
User avatar
Chord
Site Admin
Posts: 162
Joined: Wed Oct 14, 2015 4:31 pm
Contact:

Game Server Architecture

Postby Chord » Tue Oct 20, 2015 1:49 am

Okay, PlanetSide is current divided up in two server classes: Login and Game.
It makes sense that PSForever follows this same model to be compatible. It allows for the isolation of account handling (passwords, tokens) to one server and code base.

Login Server
The login server merely logs a user in, presents a list of servers, and provides a hand off to specific servers. The number of packets that the login server uses are limited and this would be simple to implement. The login server is the only server that handles usernames and passwords (through the skip launcher). If we're to implement a Sony-like launcher, then maybe having a lightweight REST API for logging in would be sufficient instead of using the built-in login form.

Game Server Scalability
Sony's original game server was supposedly able to handle 3,500 players. With today's computers, meeting this goal should be trivial. Scalability is difficult to think about early on in a project, but I believe every project should keep scalability concerns in mind from the very beginning. Similar to security, if it isn't thought up at design time, retrofitting it is a terrible, if not impossible process.

One of the core issues with game servers is that clients need status updates of all of the other clients around them. For example, if there are 100 players standing in a circle, strafing back and forth, then every server update will need to send out 100 x 100 position updates. Essentially state updates for games are an O(n^2) problem, where n is the number of clients. Optimizations can be made to limit the number of status updates, such as only sending updates if someone is visible. This requires more server side checks in order to determine what clients are visible to other clients. PlanetSide already does this with people despawning when they walk out of range. Sometimes in big fights, people will be ping-ponging between appearing and reappearing due to either a clientside limitation or a lack of server updates. When and for whom updates are given is also important from a security standpoint. If the clients have all of the players near by in memory, even if they can't see them, then this allows for trivial map hacking. PlanetSide currently sends player updates in, what I believe, is a sphere surrounding the current player, regardless if they are visible. It also prioritizes updates depending on certain factors, such as distance or if the object is a vehicle.

Game Server Security
The reason that current PlanetSide game servers are so vulnerable is that they don't perform enough serverside checks. They trust the client, which is controlled by attackers to assert that events are true. PSForever needs to check the work of the clients in order to detect deviations from the true game state on the server. Any deviations could be the result of bad clientside prediction, a laggy connections, deliberate client editing (hacking), or even a clientside bug. We can't just immediately ban someone because we detected a single deviation. I believe that using statistics and creating a function or threshold for "badness" would allow for small upsets to pass through, but for the real offenders to get a lot of badness. This is similar to the grief system: you shoot someone by accident, okay no problem, you get a small bit of grief. Continue shooting people though and your grief rises faster until you are weapon locked.

Backend Requirements
  • A database engine for storing world and account state
  • One or more login servers
  • One or more game servers
  • A REST API for other applications to query server state

Language Choices
For data storage, SQL is definitely a top choice. Use of MySQL bindings will be excellent and it is what most private servers today do. As for server language, this is a touchier subject. C++ is the first to come to mind due to its performance. C++11 is an even better candidate as its new features for memory management and anonymous functions greatly improves development. One downside to using something as heavyweight as C++ is that crashes are much more likely and without a good framework, there will be a lot of boiler plate. Boost libraries could be used similar to TrinityCore, which would greatly speed up development and allow for a cross-platform server implementation.
There are also less obvious languages, such as Scala, NodeJS, or Go. Scala runs off of the JVM, but it's amazing Akka Actor framework would allow for simple and safe concurrency and by extension parallelism. C++11 actually has a mature actor library that may be an interesting choice (see CAF) The issue with Scala is that many are not familiar with it and it does run off of the JVM (i.e. garbage collection). NodeJS would be a very strange choice and I can't see myself ever making a game server in JavaScript. Go is an interesting candidate. It has native concurrency support like Scala, but has a more C-like syntax and is touted as a 'systems language'. Choosing the right abstraction is important for this project.
[ PSForever admin and developer ]
PSForever Code | Capturing Packets | Community Discord
User avatar
FateJH
Posts: 95
Joined: Mon Nov 02, 2015 8:37 am

Re: Game Server Architecture

Postby FateJH » Mon Nov 02, 2015 10:11 am

Chord wrote:Language Choices
NodeJS would be a very strange choice and I can't see myself ever making a game server in JavaScript.

As someone who has done some work with a larger scope Javascript application work, you will probably not want to. Unless you have something specific in mind, and using a JS framework to realize that outcome makes more sense than another option, "some other language/framework" for the main game server is what I would recommend.

It really is a far-flung concern, but it's good that we're thinking that far ahead and for the sake of contingencies.
VS: FateJH, BR 21 CR 0 TR: FJH, BR 18 CR 1 NC: FateJHNC, BR 14 CR 0
Unknown004
Posts: 10
Joined: Sun Dec 27, 2015 6:49 am

Re: Game Server Architecture

Postby Unknown004 » Wed Jan 06, 2016 7:11 pm

<-- Pro C# :P
User avatar
Chord
Site Admin
Posts: 162
Joined: Wed Oct 14, 2015 4:31 pm
Contact:

Re: Game Server Architecture

Postby Chord » Sat Jan 09, 2016 1:32 pm

Unknown004 wrote:<-- Pro C# :P

Good :) Our tools will be in C#, but the server code will be in Scala. Scala roughly resembles Java (it can do everything Java can), but it's very different.
[ PSForever admin and developer ]
PSForever Code | Capturing Packets | Community Discord
p0intman
Posts: 124
Joined: Sat Oct 17, 2015 8:02 pm

Re: Game Server Architecture

Postby p0intman » Mon Jan 11, 2016 7:37 am

Chord wrote:
Unknown004 wrote:<-- Pro C# :P

Good :) Our tools will be in C#, but the server code will be in Scala. Scala roughly resembles Java (it can do everything Java can), but it's very different.

I'm interested in why Scala was chosen. I know that SWGEmu runs in c#/c++ which would make sense if ps1 originally did too.
ImageImage
Retired NC CR5, Cerberus Company
User avatar
Chord
Site Admin
Posts: 162
Joined: Wed Oct 14, 2015 4:31 pm
Contact:

Re: Game Server Architecture

Postby Chord » Tue Jan 12, 2016 10:28 pm

p0intman wrote:
Chord wrote:
Unknown004 wrote:<-- Pro C# :P

Good :) Our tools will be in C#, but the server code will be in Scala. Scala roughly resembles Java (it can do everything Java can), but it's very different.

I'm interested in why Scala was chosen. I know that SWGEmu runs in c#/c++ which would make sense if ps1 originally did too.

Most emulators seem to use C++. Because of this, it was my first choice and it happens to be my strongest language. But I realized that as one person (hopefully not forever) it would take me ages to get anywhere using C++'s abstraction (i.e starting with nearly nothing). C++ would require me to write a ton of boilerplate and it's likely that the project would never get finished. Scala has some amazing libraries that enable me to work really efficiently. I don't have to write my own network library, I never have to use threads (due to another library Akka), and the type system allows me to move really fast without breaking things. PlanetSide was written in C++ and the server most certainly was, but this doesn't mean I have to as well. Why did Sony choose C++? In 2003 I would have chosen C++ myself. It's a great and powerful language that gives you all of the power and performance. Scala's also a great and powerful language, but it provides stronger guarantees than C++. I don't have to manage my own memory. I don't have to worry about crashing because of a bad pointer. Now, of course I am sacrificing performance of my game server, but is performance really going to be the limiting factor? With C++ maybe I'd say yes, but Scala gives me tools to find performance in ways that C++ just can't.

Scala is also expressive. A lot can be done with a little. Libraries have a ton of power to make normally tedious and error prone things easy. One library I use is called SCodec and it's what I'm using to describe all of PlanetSide's packet structures. Here's what an example packet description looks like

Code: Select all

case class ServerChallengeXchg(time : Long, challenge : ByteVector, pubKey : ByteVector) extends PlanetSideCryptoPacket

implicit val codec: Codec[ServerChallengeXchg] = (
    ("unknown" | constant(2)) ::
      ("unknown" | constant(1)) ::
      ("server_time" | uint32L) ::
      ("challenge" | bytes(0xC)) ::
      ("end?" | constant(0)) ::
      ("objects" | constant(1)) ::
      ("unknown" | constant(hex"03070000000c00".bits)) ::
      ("pub_key_len" | constant(hex"1000")) ::
      ("pub_key" | bytes(16)) ::
      ("unknown" | constant(0x0e))
    ).as[ServerChallengeXchg]


Looks nearly like English. Notice I didn't have to do any manual byte manipulation anywhere. By defining that Codec, I will be able to encode a ServerChallengeXchg to a BitVector and decode a BitVector to ServerChallengeXchg (it even fills in the fields for me and verifies their types). Like the fact that I can do all of that without writing much is just amazing. The reason this is possible is that SCodec created what's known as a "DSL" or Domain Specific Language within Scala. This DSL is concise and intuitive (once you get the hang of it) and it has nearly the same power of a mini-programming language.

Another crazy library that I'm using is called Akka. Akka provides a way of programming known as the actor model. The actor model has been used with great success in the telecom industry (Sony Erikson with their language Erlang) to create extremely high performance and fault tolerant systems. Akka brings this power to Scala. In a nutshell, you create "actors" which are these self contained objects that can only interact with data by sending and receiving messages (except data that an Actor owns). This seems strange, but what this boils down to is that every Actor can run in parallel from every other actor. Got a 32 core CPU? Spin up 32 actors and give them a lot of CPU work to do. Congratulations, you now have 3200% CPU usage. One issue game servers (all game servers) have is that doing a bunch of things in parallel is usually difficult. Input and output (sending a network packet, reading from disk) is easy to do in parallel, but what about simultaneously handling 1000 players? Not so easy. You need to be very careful about how your game server shares data between threads of execution. With Actors, all of these problems go away. This is because with the actor model, you "Do not communicate by sharing memory; instead, share memory by communicating." This point might be a bit subtle for non-programmers or programmers who don't understand concurrency well, but I assure you, the guarantees it provides allow the programmer (me) to worry about other more important things. I never share memory, therefore I won't need to use locks (mutexes), which are really hard to do right and debug.

Another benefit of the Actor model is that because all communication is done through messages, this means Actors can easily be moved to remote systems, essentially for free (minus latency of course).
Lets say that PSForever gets further along in its development. We now have over 100 players playing at the same time. I notice that the game server's CPU or bandwidth is reaching capacity. What do I do? If I had used C++, how the heck would I ever be able to fix this without a rewrite or spending money on a beefier server? Check out TrinityCore and see how many issues they have running servers with many players. What can they do about it? Not much. They share memory and shared memory is not possible across multiple systems. By writing PSForever with Akka, scaling up to multiple systems or processes won't be a limiting factor just based off of the level of abstraction it provides.

At the moment Scala is also something I'm really passionate about getting better with (as you can probably tell). I'm learning so much as a programmer just by using this language. It's really making me think in ways I've never had to before.

tl;dr: Scala is a modern, high performance, language with tons of ready-to-use libraries so I can focus on actually making the game server instead of everything else.
[ PSForever admin and developer ]
PSForever Code | Capturing Packets | Community Discord
User avatar
Sulferix
Posts: 220
Joined: Sat Nov 28, 2015 8:37 pm

Re: Game Server Architecture

Postby Sulferix » Tue Jan 12, 2016 11:15 pm

Chord wrote:
p0intman wrote:
Chord wrote:Good :) Our tools will be in C#, but the server code will be in Scala. Scala roughly resembles Java (it can do everything Java can), but it's very different.

I'm interested in why Scala was chosen. I know that SWGEmu runs in c#/c++ which would make sense if ps1 originally did too.

Most emulators seem to use C++. Because of this, it was my first choice and it happens to be my strongest language. But I realized that as one person (hopefully not forever) it would take me ages to get anywhere using C++'s abstraction (i.e starting with nearly nothing). C++ would require me to write a ton of boilerplate and it's likely that the project would never get finished. Scala has some amazing libraries that enable me to work really efficiently. I don't have to write my own network library, I never have to use threads (due to another library Akka), and the type system allows me to move really fast without breaking things. PlanetSide was written in C++ and the server most certainly was, but this doesn't mean I have to as well. Why did Sony choose C++? In 2003 I would have chosen C++ myself. It's a great and powerful language that gives you all of the power and performance. Scala's also a great and powerful language, but it provides stronger guarantees than C++. I don't have to manage my own memory. I don't have to worry about crashing because of a bad pointer. Now, of course I am sacrificing performance of my game server, but is performance really going to be the limiting factor? With C++ maybe I'd say yes, but Scala gives me tools to find performance in ways that C++ just can't.

Scala is also expressive. A lot can be done with a little. Libraries have a ton of power to make normally tedious and error prone things easy. One library I use is called SCodec and it's what I'm using to describe all of PlanetSide's packet structures. Here's what an example packet description looks like

Code: Select all

case class ServerChallengeXchg(time : Long, challenge : ByteVector, pubKey : ByteVector) extends PlanetSideCryptoPacket

implicit val codec: Codec[ServerChallengeXchg] = (
    ("unknown" | constant(2)) ::
      ("unknown" | constant(1)) ::
      ("server_time" | uint32L) ::
      ("challenge" | bytes(0xC)) ::
      ("end?" | constant(0)) ::
      ("objects" | constant(1)) ::
      ("unknown" | constant(hex"03070000000c00".bits)) ::
      ("pub_key_len" | constant(hex"1000")) ::
      ("pub_key" | bytes(16)) ::
      ("unknown" | constant(0x0e))
    ).as[ServerChallengeXchg]


Looks nearly like English. Notice I didn't have to do any manual byte manipulation anywhere. By defining that Codec, I will be able to encode a ServerChallengeXchg to a BitVector and decode a BitVector to ServerChallengeXchg (it even fills in the fields for me and verifies their types). Like the fact that I can do all of that without writing much is just amazing. The reason this is possible is that SCodec created what's known as a "DSL" or Domain Specific Language within Scala. This DSL is concise and intuitive (once you get the hang of it) and it has nearly the same power of a mini-programming language.

Another crazy library that I'm using is called Akka. Akka provides a way of programming known as the actor model. The actor model has been used with great success in the telecom industry (Sony Erikson with their language Erlang) to create extremely high performance and fault tolerant systems. Akka brings this power to Scala. In a nutshell, you create "actors" which are these self contained objects that can only interact with data by sending and receiving messages (except data that an Actor owns). This seems strange, but what this boils down to is that every Actor can run in parallel from every other actor. Got a 32 core CPU? Spin up 32 actors and give them a lot of CPU work to do. Congratulations, you now have 3200% CPU usage. One issue game servers (all game servers) have is that doing a bunch of things in parallel is usually difficult. Input and output (sending a network packet, reading from disk) is easy to do in parallel, but what about simultaneously handling 1000 players? Not so easy. You need to be very careful about how your game server shares data between threads of execution. With Actors, all of these problems go away. This is because with the actor model, you "Do not communicate by sharing memory; instead, share memory by communicating." This point might be a bit subtle for non-programmers or programmers who don't understand concurrency well, but I assure you, the guarantees it provides allow the programmer (me) to worry about other more important things. I never share memory, therefore I won't need to use locks (mutexes), which are really hard to do right and debug.

Another benefit of the Actor model is that because all communication is done through messages, this means Actors can easily be moved to remote systems, essentially for free (minus latency of course).
Lets say that PSForever gets further along in its development. We now have over 100 players playing at the same time. I notice that the game server's CPU or bandwidth is reaching capacity. What do I do? If I had used C++, how the heck would I ever be able to fix this without a rewrite or spending money on a beefier server? Check out TrinityCore and see how many issues they have running servers with many players. What can they do about it? Not much. They share memory and shared memory is not possible across multiple systems. By writing PSForever with Akka, scaling up to multiple systems or processes won't be a limiting factor just based off of the level of abstraction it provides.

At the moment Scala is also something I'm really passionate about getting better with (as you can probably tell). I'm learning so much as a programmer just by using this language. It's really making me think in ways I've never had to before.

tl;dr: Scala is a modern, high performance, language with tons of ready-to-use libraries so I can focus on actually making the game server instead of everything else.


Wow. Very informative, educational, and interesting.

I'll have you know that I have done limited coding in Java, and almost complete understand the syntax, which is probably the biggest hurtle in learning a coding language. Great to hear that Scala has more efficient code systems and libraries. If it means saving time AND performance, it's an innovation worth utilizing, and I support 100%. Now apart from learning more about reverse engineering, I think I can start to help a little with direct server code construction. Whenever your ready to do that, give me some of the simpler code to get started, but I do learn quickly. Really happy to hear you are using a familiar language.

Many years ago I was all over making a minecraft server, and I spent like an entire year learning basically how Java syntax worked. I edited and wrote a little code. And with the introduction of command blocks, I HAD to learn the code, which taught me a load about how the syntax works. Now obviously it's different here, but I still know enough that I can pretty easily understand what that code means- that's what's so great about Java like/based code. Like you said earlier, it's readable in English. Which can really help you understand what's going on. So whenever your ready, I would love to start helping out with the server code. The more people who work on this the faster it goes.

I won't be able to do to much until April 26, but I would like to get started. But know that on that day and that day forward, I will have more free time to contribute that anyone else on this website. It is imperative therefore that I learn enough to contribute, since I can carry a lot of the heavy lifting that requires time consuming work.
The Ultimate Annihilation of Hackers Is Inevitable
User avatar
Chord
Site Admin
Posts: 162
Joined: Wed Oct 14, 2015 4:31 pm
Contact:

Re: Game Server Architecture

Postby Chord » Wed Jan 13, 2016 12:02 am

Sulferix wrote:Wow. Very informative, educational, and interesting.

I'll have you know that I have done limited coding in Java, and almost complete understand the syntax, which is probably the biggest hurtle in learning a coding language. Great to hear that Scala has more efficient code systems and libraries. If it means saving time AND performance, it's an innovation worth utilizing, and I support 100%. Now apart from learning more about reverse engineering, I think I can start to help a little with direct server code construction. Whenever your ready to do that, give me some of the simpler code to get started, but I do learn quickly. Really happy to hear you are using a familiar language.

Many years ago I was all over making a minecraft server, and I spent like an entire year learning basically how Java syntax worked. I edited and wrote a little code. And with the introduction of command blocks, I HAD to learn the code, which taught me a load about how the syntax works. Now obviously it's different here, but I still know enough that I can pretty easily understand what that code means- that's what's so great about Java like/based code. Like you said earlier, it's readable in English. Which can really help you understand what's going on. So whenever your ready, I would love to start helping out with the server code. The more people who work on this the faster it goes.

I won't be able to do to much until April 26, but I would like to get started. But know that on that day and that day forward, I will have more free time to contribute that anyone else on this website. It is imperative therefore that I learn enough to contribute, since I can carry a lot of the heavy lifting that requires time consuming work.

If you know Java, you're going to love Scala. Scala runs on the same runtime as Java - the Java Virtual Machine (JVM). This means all of my Scala code is immediately portable to any platform. It also means that I can use any Java library in Scala and vice-versa. Pretty neat, huh?

Scala has a lot more syntax quirks to learn than Java, but many help you write clean code. It also has a really complicated, Haskell-like type system, which is where it gets a lot of power, but a lot of complexity as well. It's all about the tradeoffs.
[ PSForever admin and developer ]
PSForever Code | Capturing Packets | Community Discord
User avatar
Sulferix
Posts: 220
Joined: Sat Nov 28, 2015 8:37 pm

Re: Game Server Architecture

Postby Sulferix » Wed Jan 13, 2016 12:17 am

Chord wrote:If you know Java, you're going to love Scala. Scala runs on the same runtime as Java - the Java Virtual Machine (JVM). This means all of my Scala code is immediately portable to any platform. It also means that I can use any Java library in Scala and vice-versa. Pretty neat, huh?

Scala has a lot more syntax quirks to learn than Java, but many help you write clean code. It also has a really complicated, Haskell-like type system, which is where it gets a lot of power, but a lot of complexity as well. It's all about the tradeoffs.


Very neat indeed. Can't wait too use it. Glad I can help in way I know how without having to learn too much. But ya if we can utilize it's lower while retaining the features and even improving upon the live servers, that would be ideal.

I also have a question. Since PlanetSide is an online only game with almost no pve aspects at all, does that make emulation easier? And if so, what seems like a reasonable amount of time to get this thing to perfect running condition? I feel like I can help immensely down the road, and I envision myself having no life when we near release lol. Just hope it doesn't end up taking ten years, like some emulation projects.
The Ultimate Annihilation of Hackers Is Inevitable
User avatar
XxXDEATHKILLXxX
Global Moderator
Posts: 112
Joined: Thu Oct 22, 2015 9:45 pm

Re: Game Server Architecture

Postby XxXDEATHKILLXxX » Wed Jan 13, 2016 12:47 am

Sulferix wrote:
Very neat indeed. Can't wait too use it. Glad I can help in way I know how without having to learn too much. But ya if we can utilize it's lower while retaining the features and even improving upon the live servers, that would be ideal.

I also have a question. Since PlanetSide is an online only game with almost no pve aspects at all, does that make emulation easier? And if so, what seems like a reasonable amount of time to get this thing to perfect running condition? I feel like I can help immensely down the road, and I envision myself having no life when we near release lol. Just hope it doesn't end up taking ten years, like some emulation projects.


It's too bad I know very little Java so I can't really help out with the programming part of this project. Right now I'm uploading all the pictures for the wiki. Next, I plan to complete p0intman's wiki to-do list. After that I can complete any of your side tasks that you don't have time for or whatever the reason is why you dont feel like doing at the moment. If you guys ever need help with any extra tasks, just send me a PM.
For land. For power. Forever.

TR: XxXDEATHKILLXxX | BR 21 CR 2
NC: DEATHKILLNC | BR 11 CR 0
VS: DEATHKILLVS | BR 9 CR 1

Return to “R & D”

Who is online

Users browsing this forum: No registered users and 1 guest