Functional Tests with Gauntlet

Client and Server functional tests with Gauntlet

Fri, 07 Oct 2022

Before I get into it, I have to say thank you for Daedalic Entertainment for creating the Daedalic Test Automation Plugin which we will be using to run Gauntlet

What is Gauntlet

Gauntlet is a portion of the automation tool in the engine that allows you to spin up instances of the engine. This is particularly useful when you want to spin up a server and a client, connect the client to the server and then run your tests to simulate a dedicated server environment.

Daedalic Test Automation Plugin

Download the Daedalic Test Automation Plugin. This plugin will help us orchestrate functional tests using Gauntlet. This plugin will require a few minor alterations to make it spin up a server as well as orchestrate server builds
Before jumping into this tutorial, please read the information in the plugins repo and set up the plugin to get it working with your project

The DaeGauntletTestController.cpp located below is an import file. This file is added to each Gauntlet Server/Client that we spin up. This is what orchestrates the tests. Its worth reading and trying to understand

Plugins/DaedalicTestAutomationPlugin/Source/DaedalicTestAutomationPlugin/Private/DaeGauntletTestController.cpp

Navigate to PROJECT_FOLDER\Build\DaedalicTestAutomationPlugin.Automation, if you don’t have this folder, go back to step one and read the directions. Open the .csproj file with Visual Studio
There are 2 files here. One is DaeGauntletTest which creates a config that is supplied to the Gauntlet tool that tells us how to set up our test client/server and the second is DaeTestConfig which adds extra parameters to the cli command that is passed to the engine. These extra commands allow reports to be generated
First things first, include the UnrealBuildTool and Gauntlet.Automation dlls inside of this project. You do this by right clicking Dependencies and then clicking Add Project Reference
The UnrealBuildTool is located at - ENGINE_LOCATION\Engine\Binaries\DotNET\AutomationTool
Gauntlet.Automation is located at - ENGINE_LOCATION\Engine\Binaries\DotNET\AutomationTool\AutomationScripts\Gauntlet

The reason we do this is so we can reference different options to add to our config

DaeTestConfig

I removed the handling of adding the report switches in this file to the DaeGauntletTest file. The main reason I did this is because the server is the authority and I only want the server to spit out a test report and not the clients. By default, all server/clients in your config will spit out a report and they will overwrite each other

using Gauntlet;
using System.Collections.Generic;

namespace DaedalicTestAutomationPlugin.Automation
{
    public class DaeTestConfig : EpicGame.EpicGameTestConfig
    {
        /// <summary>
		/// Where to write a JUnit XML report to.
		/// </summary>
		[AutoParam]
        public string JUnitReportPath;

        /// <summary>
        /// Where to write test reports to.
        /// </summary>
        [AutoParam]
        public string ReportPath;

        /// <summary>
        /// Which single test to run, instead of all available tests.
        /// </summary>
        [AutoParam]
        public string TestName;

        public override void ApplyToConfig(UnrealAppConfig AppConfig, UnrealSessionRole ConfigRole, IEnumerable<UnrealSessionRole> OtherRoles)
        {
            base.ApplyToConfig(AppConfig, ConfigRole, OtherRoles);
        }
    }
}

DaeGauntletTest

I made quite a few adjustments here. I’ll break each section down

using System.Collections.Generic;
using System.Linq;
using UnrealBuildTool;
using Gauntlet;

namespace DaedalicTestAutomationPlugin.Automation
{
    public class DaeGauntletTest : UnrealTestNode<DaeTestConfig>
    {
        public DaeGauntletTest(UnrealTestContext InContext) : base(InContext)
        {
        }

        public override DaeTestConfig GetConfiguration()
        {
            DaeTestConfig Config = base.GetConfiguration();

            // Parsing Params for reporting
            Config.JUnitReportPath = Context.TestParams.ParseValue("JUnitReportPath", "");
            Config.ReportPath = Context.TestParams.ParseValue("ReportPath", "");
            Config.TestName = Context.TestParams.ParseValue("TestName", "");

            // Start 2 instances of win clients
            List<UnrealTestRole> ClientRoles = Config.RequireRoles(UnrealTargetRole.Client, UnrealTargetPlatform.Win64, 2).ToList();
            // Starting 1 dedicated server
            UnrealTestRole ServerRole = Config.RequireRole(UnrealTargetRole.Server, UnrealTargetPlatform.Win64);

            // Max test duration in seconds
            Config.MaxDuration = 60 * 10; // 10 minutes

            // Adding controllers to each role
            foreach(UnrealTestRole role in ClientRoles)
            {
                role.Controllers.Add("DaeGauntletTestController");
            }
            ServerRole.Controllers.Add("DaeGauntletTestController");

            // Adding reporting paths to server
            ServerRole.CommandLine += string.Format(" -JUnitReportPath=\"{0}\"", Config.JUnitReportPath);
            ServerRole.CommandLine += string.Format(" -ReportPath=\"{0}\"", Config.ReportPath);
            ServerRole.CommandLine += string.Format(" -TestName=\"{0}\"", Config.TestName);

            // Ignore user account management.
            Config.NoMCP = true;

            return Config;
        }
    }
}

Here we are grabbing the switches provided by the cli command and parsing them

            // Parsing Params for reporting
            Config.JUnitReportPath = Context.TestParams.ParseValue("JUnitReportPath", "");
            Config.ReportPath = Context.TestParams.ParseValue("ReportPath", "");
            Config.TestName = Context.TestParams.ParseValue("TestName", "");
            ```

This section tells Gauntlets what to spin up.  The syntax is easy to parse.  If you wanted a Linux server, change UnrealTargetPlatform.Win64 to just UnrealTargetPlatform.Linux.  Be warned, I had some issues with this so for now I just do tests with Win64.  In the future I hope to fix this since my final server will be ran in a linux docker container

```csharp
            // Start 2 instances of win clients
            List<UnrealTestRole> ClientRoles = Config.RequireRoles(UnrealTargetRole.Client, UnrealTargetPlatform.Win64, 2).ToList();
            // Starting 1 dedicated server
            UnrealTestRole ServerRole = Config.RequireRole(UnrealTargetRole.Server, UnrealTargetPlatform.Win64);

Gauntlet needs to add controllers to each instance to orchestrate the tests. These controllers will start each test, change maps and tells gauntlet when the tests are over. Examine that TestControllers code from step 2.

            // Adding controllers to each role
            foreach(UnrealTestRole role in ClientRoles)
            {
                role.Controllers.Add("DaeGauntletTestController");
            }
            ServerRole.Controllers.Add("DaeGauntletTestController");

As I mentioned before, I only want the server to spit out a report. This is how I did it

            // Adding reporting paths to server
            ServerRole.CommandLine += string.Format(" -JUnitReportPath=\"{0}\"", Config.JUnitReportPath);
            ServerRole.CommandLine += string.Format(" -ReportPath=\"{0}\"", Config.ReportPath);
            ServerRole.CommandLine += string.Format(" -TestName=\"{0}\"", Config.TestName);

Finally, any time you change this config you must generate the new dlls and have it copied to the Engine folder. You do this by hitting the build button in Visual Studio. Once you build it and its successful, go to your projects .uproject file and click Generate Visual Studio Project Files. This will copy your dll to your engine folder as well. For this to work, you have to set up the plugin correctly with the documentation in the repo.