The following in-house technical research study by Technical Artist Lukas Schmeck explores the use of machine learning and agents for experience design in Unreal Engine.

When Driving Becomes Training Data

During the Hyperdrive project, there were a few exciting ideas around training cars using machine learning, either to use them as ghost cars or derive a race line and driving hints for the player from them. Unfortunately, these did not fit the scope and timeline, and even though I got a good head start over a rainy Sunday setting up basic agents just using an official guide, there were a couple of important questions that we were not able to answer:

I. How must observations and rewards be structured even to encourage racing-type behaviour?

II. (How) Do these agents compare to human performance?

III. Can we derive usable race lines from the average/best/top 10% of agents?

IV. Can agent actions be used to give users input hints? (Ideal braking point, throttling around apexes, etc.)

A couple of rainy days later, I was able to answer most of the above and set up a training process that yields optimised race lines on the Hyperdrive circuit, while agents race at speeds comparable to human performance in the shipped application (steering analytically rather than Hyperdrive's spline-following logic).

Proximal Policy Optimisation

The Learning Agent Plugin uses (what seems to me to be) a fairly standard machine learning architecture under the hood, namely Proximal Policy Optimisation with an Adam/W optimiser and a range of activation functions (e.g., ELU, GELU, TanH).

I decided to stick mostly with the defaults, increasing the Hidden Layer count and size to improve the agents' performance after seeing promising training results. The policy and critic, as well as observations, actions and rewards can be configured and modified in both C++ and Blueprints.

Question I

How must observations and rewards be structured to even encourage racing-type behaviour?

For reward design and shaping, the original GT Sophy paper from Google came in quite helpful as a reference. This is what takes us from cars that either follow the middle of the road dogmatically or race uncontrollably/unsustainably to a more realistic representation of racing cars and a driver's decision-making.

Below is what I ended up with. I tested training periods (pure RL) between 2 and 12h, focusing on average and best lap times per session/iteration, as well as the total number of completed laps. 32 agents are training simultaneously, with randomised starting positions and rotations per Episode.

Note: For the scope of this prototype, I chose to only focus on single-car hot laps. Rankings or interactions (i.e., collisions) between cars, although the process of getting there is fairly well documented in the paper, are not implemented.

Observations: Egocentric position and direction, additional pos/dir samples of different distances along the track, Velocity
Actions: Steering, Throttle, Brake (Gears are automatic for simplicity)
Rewards: Velocity along the track spline (continuous), Reaching a checkpoint along the track (less sparse, 12 checkpoints), Completing a lap (sparse), Completing a new Personal Best lap (sparse), Going off-track (Termination)

Question II

(How) Do these agents compare to human performance?

Disclaimer: Human performance refers to performance within the Hyperdrive application, which includes a mechanic that ensures cars always steer towards a predefined "follow line". Agents steer analytically/from learned actions. In the shipped application, I would expect human drivers to do much worse (compared to simply following the spline), given their ability to steer. This then quickly turns into a discussion around input devices and camera POV, which is outside the scope of this exploration.

All car physics and track parameters are identical between Hyperdrive cars and agents.

From my observations, it is fairly easy to complete a single-player Hyperdrive lap in around 20s by adhering to common braking points and applying some throttle control. The challenge mostly arises from other players on the track, who can collide, boost, or hit tyre stacks, obstructing a clean view of the track. That said, while I don't have access to a large database of laps/players at the time of writing, I expect the average hot lap time to be somewhat higher, around 21-24s.

Agents, after around 6h of training, complete the lap in 22.5-24.5s, with an average time of 23.8s. Lap times for the first lap during training are just under 40 seconds, and we see significant improvement throughout the training process. I would conclude that this result is quite close to average human performance in Hyperdrive, and would expect it to be among the best if manual steering were enabled for humans.

Question III

Can we derive usable race lines from the average/best/top 10% of agents?

Yes and no. While we do get very realistic race lines for all the mentioned configurations, using them in the existing Hyperdrive setup is a bit trickier.

Our look-ahead steering setup favours a different kind of race line, which hugely prioritises safety (i.e. tolerance for bad inputs while staying on track) over just the 'shortest' way along the track. This becomes especially apparent in corners, where agents brake and throttle with near-perfect accuracy, something humans cannot reproduce given the current mechanics and input device. Going off track is punished with a speed decrease, further amplifying the effect.

My laps, based on the example agent’s fastest-iteration racing line, are definitely comparable to the hand-drawn Hyperdrive racing line.

Question IV

Can agent actions be used to give users input hints?

Absolutely. Building observability and infrastructure around the data gathered from agents has been one of the more fun aspects of this exploration. At the moment, we record all completed laps for analysis in a struct FRecordedLap.

struct FRecordedLap
{
GENERATED_BODY()

UPROPERTY(BlueprintReadOnly) int32 AgentId = -1;
UPROPERTY(BlueprintReadOnly) int32 IterationIndex = -1;
UPROPERTY(BlueprintReadOnly) float LapTime = 0.f;
UPROPERTY(BlueprintReadOnly) bool bValid = false;
UPROPERTY(BlueprintReadOnly) TArray<float> Offsets;
};

We display the all-time (session) best lap, a moving live average, as well as the average, worst, and top 10% laps of the latest iteration, each a different take on the "ideal race line".

From here, recording steering, throttle, braking, and gear changes along the track's length to the structure would be trivial. These could easily extend the UI to include heatmaps, tooltips, or deeper live callouts, as seen in Hyperdrive.

Additional Tool: Track geometry

Besides Learning Agents optimising the race line on the Hyperdrive track, I wanted to see if there was a nicer way to create new tracks for training agents. That is, without jumping into Houdini, drawing splines and then manually exporting and importing JSONs.

I fairly quickly ended up with a vibe-coded web app that takes in control points and treats them as polygon vertices, adjusting the corner radius/bevel based on their size. This yields racetrack-looking shapes quite quickly compared to other spline-drawing methods.

Quick rebuild of the Monza Circuit

The control points adapt quite well to touch screens. Additionally, we include varying track width (which is also honoured in our Agents' off-track negative reward) and auxiliary splines (gravel pits, tyre stacks, ...) to complete the track environment.

With a bit of I/O and procedural generation around it, we can export these tracks, have them create the full track geometry in Unreal automatically, and start training Agents on it in seconds.

Potential continuations

There is quite a lot that can be done to speed up the training process.

Ideally, we don't start from scratch with Reinforcement Learning. Instead, the process should start with Imitation Learning (i.e., cars that achieve basic driving following the middle-of-the-road track spline) and then switch to RL in a second step. This can likely be automated as well, so it doesn't require a manual re-initialisation when switching from IL to RL. I tried IL for a bit, strictly following Epic's guide, with success in creating the basic driver skill mentioned above.
Training can be headless and distributed, which should yield quite a large boost in training speed. I wouldn't expect any of these measures to put us near "real-time" speeds (e.g. draw custom track → get RL-based race line and driving hints → race on custom track in one flow), though this might be subject to balancing training speed vs accuracy, better concurrency and careful procedural post-processing steps.

Car behaviour

Ideally, any agent actions are human-like instead of jittery. While this looks okay in the current visualisation, we might want to set up rewards and limit the action frequency (currently at 60 ticks) to avoid training a robo-driver.
Typical agent actions as driving inputs
As mentioned above, adding a ranking and inter-car collisions are logical next steps, although they accomplish a very different task from a hot-lap AI.

Communication pipeline and user flow

Not a focus of this prototype. Implementing a state machine and a protocol like MQTT to publish tracks to would be required for an experience similar to previous projects, where, e.g. users draw tracks that they can then race on.