This document is about: SERVER 4
SWITCH TO
SERVER 4 SERVER 5

LoadBalancing Application

This article explains the server-side implementation of the LoadBalancing application.

Concept

LoadBalancing application extends Hive framework and implements features like Rooms, Events, Properties and so on and adds a layer of scalability, that enables you to run the application on multiple servers.

The basic setup is simple:
There is always 1 Master Server and 1..N Game Servers.

LoadBalancing also adds lobby support and matchmaking capabilities.

Photon Server Concept: LoadBalancing Setup
Photon Server Concept: LoadBalancing Setup

The Master Server handles these tasks:

  • keep track of the games that are currently open on the Game Servers.
  • keep track of the workload of the connected Game Servers and assign peers to the appropriate Game Servers.
  • keep and update a list of available rooms for clients in the "Lobby".
  • find rooms (random or by name) for clients and forward the game server address to them.

The Game Servers have these tasks:

  • host the game rooms.
  • regularly report their current work load and the list of their games to the Master Server.

Differences with Photon Cloud

The LoadBalancing Application provides almost the same logic as the Photon Realtime Cloud service.
Some requirements to run as Cloud service just don't apply to a special server with custom logic, so they are removed.
This greatly simplifies the code, too.

  • No virtual Applications. Only one game logic is running per LB instance. The AppId parameter in operation Authenticate is ignored.
  • No player separation by AppVersion parameter. This is part of the virtual apps also.

You can read more about the differences here.

Basic Workflow

The workflow from a client side perspective is quite simple as well:

Clients connect to the Master Server, where they can join the lobby and retrieve a list of open games.

When they call a CreateGame operation on the Master, the game is not actually created - the Master Server only determines the Game Server with the least workload and returns its IP to the client.

When clients call a JoinGame or JoinRandomGame operation on the Master, the Master looks up the Game Server on which the game is running and returns its IP to the client.

The client disconnects from the Master Server, connects to the Game Server with the IP it just received and calls the CreateGame or JoinGame operation again.

Photon Server: LoadBalancing Sequence Diagram
Photon Server: LoadBalancing Sequence Diagram

Master Server

This section explains the Master Server implementation - see the LoadBalancing.MasterServer namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution.

The MasterApplication decides if incoming connections are originated by game clients (on the "client port") or by game servers (on the "game server port").

Master Server: Handling Client Peers

The MasterClientPeer represents a client connection to the Master Server. T
he following operations are available for a MasterClientPeer:

  • Authenticate
    The Authenticate operation has only a dummy implementation.
    The developer should use it as a starting point to implement his own authentication mechanism:

C#

    // MasterClientPeer.cs: 
    private OperationResponse HandleAuthenticate(OperationRequest operationRequest)
    {
        OperationResponse response;

        var request = new AuthenticateRequest(this.Protocol, operationRequest);
        if (!OperationHelper.ValidateOperation(request, log, out response))
        {
            return response;
        }
        
        this.UserId = request.UserId;

        // publish operation response
        var responseObject = new AuthenticateResponse { QueuePosition = 0 };
        return new OperationResponse(operationRequest.OperationCode, responseObject);
    }

  • JoinLobby
    The JoinLobby operation is used to add the MasterClientPeer to the AppLobby, which contains a GameList - the list of all open games on any Game Server.
    The peer receives an initial GameListEvent, which contains the current list of games in the GameList (filtered by the optional Properties of the JoinLobby operation):
    Afterwards, a GameListUpdateEvent is send to the client at regular intervals, which contains the list of changed games (also filtered by the optional Properties of the JoinLobby operation).
    The client will receive the update events as long as it is connected.

    C#

    // AppLobby.cs: 
protected virtual OperationResponse HandleJoinLobby(MasterClientPeer peer, OperationRequest operationRequest, SendParameters sendParameters)
{
    // validate operation
    var operation = new JoinLobbyRequest(peer.Protocol, operationRequest);
    OperationResponse response;
    if (OperationHelper.ValidateOperation(operation, log, out response) == false)
    {
        return response;
    }

    peer.GameChannelSubscription = null;

    var subscription = this.GameList.AddSubscription(peer, operation.GameProperties, operation.GameListCount);
    peer.GameChannelSubscription = subscription; 
    peer.SendOperationResponse(new OperationResponse(operationRequest.OperationCode), sendParameters);

    // publish game list to peer after the response has been sent
    var gameList = subscription.GetGameList();
    var e = new GameListEvent { Data = gameList };
    var eventData = new EventData((byte)EventCode.GameList, e);
    peer.SendEvent(eventData, new SendParameters());

    return null;
}

  • JoinGame / JoinRandomGame
    The JoinGame operation is called when a client wants to join an existing game that is listed in the AppLobby's GameList, specified by a unique GameId.
    If the game exists and the peer is allowed to join, the Master Server returns the IP of the Game Server, on which the Game is actually running, to the client.

    The Master Server also updates the GameState and adds the peer to its list of "joining peers".
    It will be removed once it has joined the game on the Game Server (or after a certain timeout).
    This way, the Master Server can keep track of peers that are in transition between the Master and the Game Server.

    JoinRandomGame works in a similar way, except that the game is chosen by random by the Master Server and the GameId is returned to the client.

  • CreateGame
    The CreateGame operation is called when a client wants to create a new game.
    The Master Server determines a Game Server on which the new game will be created and returns the Game Server's IP to the client.
    See the "LoadBalancing Algorithm" section for more details.

    In addition, a GameState object is created and added to the GameList and the peer is stored as a "joining peer".
    Note that this GameState is only used to keep track of the games - the game itself only exists on a Game Server.

Handling Game Server Peers

The Master Server always knows which Game Servers are available, how many games they host and how the current workload is.

To achieve this, each Game Server connects to the Master Server on startup.
The MasterApplication maintains a GameServerCollection, in which IncomingGameServerPeers are stored.

The Game Server can only call one operation:

  • RegisterGameServer
    The Game Servers call the RegisterGameServer operation once after they are connected to the Master Server.
    The Game Server is added to the Master's GameServerCollection and to its LoadBalancer (see the "LoadBalancing Algorithm").
    It will be removed from the GameServerCollection on disconnect.

Check the "Game Server" section to see how the Game Server sends further updates about its games and its workload to the Master.

Game Server

This section describes the Game Server implementation.
See the LoadBalancing.GameServer namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution.

Game Server: Handling Client Peers

As soon as a client has received a Game Server address from the Master, the client can call any operation on the Game Server that is available in Hive (while joine to the room).
The only difference is that we have separate operation codes for JoinGame and CreateGame on the Game Server.

Reporting Game States to the Master

The connection to the Master server is represented as an OutgoingMasterServerPeer in the Game Server.
Once the connection is established, the Game Server calls a Register operation on the Master Server.
Afterwards, the Game Server publishes all existing game states to the master server:

C#

// OutgoingMasterServerPeer.cs: 
protected virtual void HandleRegisterGameServerResponse(OperationResponse operationResponse)
{
    // [...]
    
    switch (operationResponse.ReturnCode)
    {
    case (short)ErrorCode.Ok:
        {
            log.InfoFormat("Successfully registered at master server: serverId={0}", GameApplication.ServerId);
            this.IsRegistered = true;
            this.UpdateAllGameStates();
            this.StartUpdateLoop();
            break;
        }
    }
}

This is done by sending a message to each Game that tells it to send it's game state to the Master:

C#

// OutgoingMasterServerPeer.cs:
public virtual void UpdateAllGameStates()
{
    // [...]
    
    foreach (var gameId in GameCache.Instance.GetRoomNames())
    {
        Room room; 
        if (GameCache.Instance.TryGetRoomWithoutReference(gameId, out room))
        {
            room.EnqueueMessage(new RoomMessage((byte)GameMessageCodes.ReinitializeGameStateOnMaster));
        }                
    }
}

The Game handles this in the ProcessMessage method and calls the UpdateGameStateOnMaster method to send an UpdateGameEvent to the Master:

C#

 protected virtual void UpdateGameStateOnMaster(
            byte? newMaxPlayer = null, 
            bool? newIsOpen = null,
            bool? newIsVisble = null,
            object[] lobbyPropertyFilter = null,
            Hashtable gameProperties = null, 
            string newPeerId = null, 
            string removedPeerId = null, 
            bool reinitialize = false)
        {            
            // [...]
            
            var e = this.CreateUpdateGameEvent();
            e.Reinitialize = reinitialize;
            e.MaxPlayers = newMaxPlayer;
            // [ ... more event data is set here ... ]
            
            var eventData = new EventData((byte)ServerEventCode.UpdateGameState, e);
            GameApplication.Instance.MasterPeer.SendEvent(eventData, new SendParameters());
        }
}

The game state is also updated on the master whenever a game is created, joined or left by a client or its properties are changed.

LoadBalancing Implementation

The next section describes how the game servers report their current work load to the Master Server
and how the Master Server determines the Game Server that is best suited to handle new CreateGame requests - the actual LoadBalancing algorithm.

Determine Workload

See the LoadBalancing.LoadShedding namespace in the "\src-server\Loadbalancing\Loadbalancing.sln" solution for implementation details.

The Game Servers regularly report their current work load to the master server.
The work load includes, for example:

  • CPU usage
  • Traffic
  • some Photon-specific values, like ENet + Business Queue Length, the average time the server spends on each request, etc.
  • Latency (when sending requests to itself)

The most important (and easiest to understand) factor is the CPU load, so we will focus on the CPU load in this documentation.

All these factors are summarized in a single value - the "Load Level" of a Game Server, which is reported to the Master.

The lower the load level, the better is the Game Server suited to host new games.

Implementation Details

The Game Server collects "Feedback" about the above-mentioned factors.
There is one FeedbackController object for each factor - it consists of a FeedbackName and a FeedbackLevel:

C#

internal enum FeedbackName
{
    CpuUsage,     
    Bandwidth,
    TimeSpentInServer
}
    
public enum FeedbackLevel
{
    Highest = 4, 
    High = 3, 
    Normal = 2, 
    Low = 1, 
    Lowest = 0
}

The DefaultConfiguration class defines the thresholds for each value.
e.g. a server has the "lowest" FeedbackLevel up to 20% CPU usage, it reaches the "highest" FeedbackLevel at 90% CPU and so on.

C#

// DefaultConfiguration.cs: 

internal class DefaultConfiguration
{
    internal static List<FeedbackController> GetDefaultControllers()
    {
        var cpuController = new FeedbackController(
        FeedbackName.CpuUsage,
        new Dictionary<FeedbackLevel, int>
                {
                    { FeedbackLevel.Lowest, 20 },
                    { FeedbackLevel.Low, 35 },
                    { FeedbackLevel.Normal, 50 },
                    { FeedbackLevel.High, 70 },
                    { FeedbackLevel.Highest, 90 }
                },
        0,
        FeedbackLevel.Lowest);

    // [...]
}

These values can also be configured in a "workload.config" file.
See "workload.config Example".
The LoadBalancing.LoadShedding.Configuration namespaces takes care of reading values from a config file, or applies the DefaultConfiguration if no config exists.

At regular intervals, the Game Server checks some Windows Performance Counters, sets the current values for all its FeedbackControllers and calculates a new "overall feedback".

This is done in the WorkloadController class:

C#

private void Update()
{
    FeedbackLevel oldValue = this.feedbackControlSystem.Output;

    if (this.cpuCounter.InstanceExists)
    {
        var cpuUsage = (int)this.cpuCounter.GetNextAverage();
        Counter.CpuAvg.RawValue = cpuUsage;
        this.feedbackControlSystem.SetCpuUsage(cpuUsage);
    }
    
    // [...]
    
    if (this.timeSpentInServerInCounter.InstanceExists && this.timeSpentInServerOutCounter.InstanceExists)
    {
        var timeSpentInServer = (int)this.timeSpentInServerInCounter.GetNextAverage() + (int)this.timeSpentInServerOutCounter.GetNextAverage();
        Counter.TimeInServerInAndOutAvg.RawValue = timeSpentInServer;
        this.feedbackControlSystem.SetTimeSpentInServer(timeSpentInServer); 
    }
    
    this.FeedbackLevel = this.feedbackControlSystem.Output;
    Counter.LoadLevel.RawValue = (byte)this.FeedbackLevel; 

    if (oldValue != this.FeedbackLevel)
    {
        if (log.IsInfoEnabled)
        {
            log.InfoFormat("FeedbackLevel changed: old={0}, new={1}", oldValue, this.FeedbackLevel);
        }

        this.RaiseFeedbacklevelChanged();
    }
}

If the overall feedback level changes, the OutgoingMasterServerPeer will report the new server state to the Master:

C#

 public void UpdateServerState()
 {
    // [...]
    this.UpdateServerState(
        GameApplication.Instance.WorkloadController.FeedbackLevel,
        GameApplication.Instance.PeerCount,
        GameApplication.Instance.WorkloadController.ServerState);
}

workload.config Example

If you want to know how we configure the workload for traffic then just take 90% of the maximum traffic for "Highest", 70% for "High", 50% for "Normal", 35% for "Low" and 20% for "Lowest".
But you are free to declare lowest/low/normal/high/highest as you want if you don't want to use those percentage values.

If your server's bandwidth is 20 Mbps (20000000 b/s) then you can have the following values:

  • Lowest: 4000000
  • Low: 7000000
  • Normal: 10000000
  • High: 14000000
  • Highest: 18000000

XML

<?xml version="1.0" encoding="utf-8" ?>
<FeedbackControlSystem>
  <FeedbackControllers>
    <add Name="Bandwidth" InitialInput="0" InitialLevel="Lowest">
      <FeedbackLevels>
        <add Level="Lowest" Value="4000000"/>
        <add Level="Low" Value="7000000"/>
        <add Level="Normal" Value="10000000"/>
        <add Level="High" Value="14000000"/>
        <add Level="Highest" Value="18000000"/>
      </FeedbackLevels>
    </add>
  </FeedbackControllers>
</FeedbackControlSystem>

Only "Highest" has consequences, where if its value is exceeded then no more games are created on this Game Server.
If you want to use higher limits you should test if the application can handle the number of messages.

LoadBalancing Algorithm

See the LoadBalancing.LoadBalancer class in the "\src-server\Loadbalancing\Loadbalancing.sln" solution for implementation details.

The Master Server stores the LoadLevel for each Game Server in the LoadBalancer class.
It also holds an additional list of all the servers that currently have the lowest load level.

Whenever a client calls the CreateGame operation, the Master Server fetches the address of a server with the lowest load level from the LoadBalancer and returns it to the client, which then connects to that server.

Configuration and Deployment

For demonstration purposes, the SDK contains a setup of 1 Master and 1 Game Server in its deploy directory:

  • "deploy\LoadBalancing\Master"
  • "deploy\LoadBalancing\GameServer"

This setup is only intended for local development.

Deploying a Game Server

When you deploy your LoadBalancing project to a production server, you should not host 2 Game Server applications on one server.

You need to make sure that the Game Servers can register at the Master Server.

Set the MasterIPAddress in the "Photon.LoadBalancing.dll.config" to the Master's public IP.

You also need to make sure that the game clients can reach the Game Servers.
On each Game Server, you need to set the Public IP address of that Game Server.
If you leave the value empty, the Public IP Address will be auto detected.

XML

<Photon.LoadBalancing.GameServer.GameServerSettings>
      <setting name="MasterIPAddress" serializeAs="String">
        <value>127.0.0.1</value>
<setting>      
    <setting name="PublicIPAddress" serializeAs="String">
        <value>127.0.0.1</value>
        
        <!-- use this to auto-detect the PublicIPAddress: -->
        <!-- <value></value> -->
      </setting>
      
      <!-- [...] -->
</Photon.LoadBalancing.GameServer.GameServerSettings>

You can also use the Photon Control to set a Public IP.

Deploying a Master Server

You need to make sure that you only have 1 Master server: either remove all settings for the "Master" application from the PhotonServer.config on your Game Servers or at least make sure that your game servers and clients all use the same IP to connect to the same, single Master server.

Otherwise, no special configuration is required on the master server.

Take a Game Server out of Rotation

As discussed in the "Loadbalancing Implementation" section, the Master server knows the state of the Game Servers, as noted in the "ServerState" enumeration:

  • "online" (that's the default)
  • "out of rotation" (= open games are still listed in the Lobby and players can join existing games on that server but no new games are created)
  • "offline" (existing games on that server can not be joined and no new games are created on that GS)

The GS send their server state to the Master regularly - see the OutgoingMasterServerPeer class:

C#

public void UpdateServerState()
{
  if (this.Connected == false)
  {
    return;
  }

  this.UpdateServerState(
                this.application.WorkloadController.FeedbackLevel,
                this.application.PeerCount,
                this.application.WorkloadController.ServerState);
}

If you want to set a server state programmatically, what you need to do is:

  • modify the WorkloadController class so that it determines the current server state
  • for example, you can add a "file watcher" and read the server state from a text file (0 / 1 / 2).

You can also build an operation that is called from a client, read it from a database or whatever comes to your mind.

Back to top

Cookies,
anyone?

We use cookies and related technologies to enhance your experience, show you personalised content and analyze performance and traffic on our website. By clicking on the „Accept All“ button you consent to the use of non-functional cookies and the subsequent processing of personal data to optimize our website and services as described in more detail in our Privacy and Cookie Policy.

By clicking on the „Customize or Deny all“ button you can decide otherwise. Clicking on the „Customize“ button will take you to a page where you can configure the usage of non-functional cookies (and related technologies) or deny all of them. You can access these settings at any time and also subsequently deselect cookies at any time in the footer area of our website.

ACCEPT ALL

Cookie Overview

We use the following categories of cookies and related technologies to enhance your experience, show you personalized content, and analyze performance and traffic on our website. We respect your right to privacy and accordingly you can chose to not allow some types of cookies (and related technologies). Click on the different category sliders and change our default settings to manage your cookie settings.

For more information on the specific cookies/related technologies we use and on how we use these, please see section 15 E. of our Privacy and Cookie Policy.

ABSOLUTELY NECESSARY

Authentication cookies we use are required to run our services … Cookies are required: Privacy and Cookie Policy

FUNCTIONAL AND MARKETING COOKIES

These cookies collect anonymous data and allow us to optimize our website and user experience. These cookies are listed here: Privacy and Cookie Policy

ANALYTICS / TAG MANAGER

Help us to understand how visitors interact with our services, enables us to analyze and improve our services (also through third party analytics).