What everyone already knew - Exploiting destroys the game for everyone

A very interesting post from Elder Scroll Online Creative Director Rich Lambert appeared on 2020-July-27. I'ts not even the fact that they are running a massive test on Area of Effect abilities in an effort to address performance issues that's interesting.

It's all the details that they revealed, about what's causing all the problems, how they make decisions, and how they address consequences.

And of course the problem isn't limited to PvP -- It exists in PvE as well and the symptoms show in how players look at dungeon groups.

ADDENDUM: 2020-Mar-5 - ZOS confirms that "disabling proc sets has breathed new life into PvP gameplay and has made fights more enjoyable and fun". We feel this is validation that excessive exploitation of game systems makes the game less enjoyable.

But first, some background.

About Exploiting - What is it, really?

Ever since the early days of roleplaying games, where it was run on pen and paper and a game played by friends around a table, players quickly realized that exploiting the system ultimately ruined the for everyone -- yes, even minmaxers, when the game becomes a snorefest because they can bulldoze everything.

In any system that allows a lot of freedom -- and where part of the fun is in allowing that freedom of creativity -- there are bound to be unbalancing things you can do. Legitimate by the rules, but so out of bounds of what is intended or can be supported that it caused problems in gameplay.

One of the ways to counter this was an emphasis on the roleplay aspect of the game. Character concept instead of putting together a character because of the best numbers you could get. And it was fairly easy to spot when someone was playing math instead of trying to play the game.

Now we have translated roleplaying games into computer games. One of the key results is there is no longer gamemaster oversight. It's no longer a game played around the table by friends, who are concerned about the fun of the game being ruined by one or two exploiters. There's no longer a gamemaster who can veto what is possible by the rules, but ultimately not fun for everyone.

Is it Exploiting to Meta / Mini-max?

It is natural for people to look for "best in slot" or "meta" or whatever you want to call it. Basically, the best numbers you can get.

Which is often fine, in systems with well-defined limits, because there's only so much you can do. (e.g., Magic Sword +1 or Magic Sword +2? Of course you take the +2.)

BUT...

Minimaxing / meta is only safe when the range of what is possible falls within intended bounds of gameplay. 

If the potential range is too far outside intended bounds of gameplay and what the game content is designed for, then it just becomes a way to exploit the system.

Further, in very open-ended systems with many possible permutations, it is basically unfeasible to track and try to predict useful combinations, much less balance them all. You end up with extreme outlier builds so far out of intended gameplay that they are problematic. And when players insist on using them, they are basically exploiting the game systems.

Legitimate by the rules? Yes. But should you actually use them? No.

But you know people will, to get an advantage.

And you can't simply start chopping out things or making global adjustments because it would hurt the people who are NOT exploiting the system, who are actually having fun with the system, and operating within the range of what is intended and supported by the game content.

ESO Specific Example 1

Players in open world Player versus Player (PvP) do "spawn camping". They wait outside a door where a transition loading screen is required to go in or come out. This is the "spawn point". During this transition period, while the other player is in a loading screen waiting for the instance to be loaded, they cannot interact or even see what is happening, but other players can attack them. So by the time they have fully loaded, they are already dead.

Technically possible by the game engine, but clearly a cheese / exploit, one which ruins gameplay for everyone. So we see that what is allowed or possible isn't necessarily what should be used.

ESO Specific Example 2

In Update 6.0.5., healing in Player versus Player (PvP) was reduced by an additional 10% -- when it was already reduced by 50% -- to try to address the issue of "unkillable" players, builds where it was mostly impossible to actually kill the player due to whatever build they were using.

So we see that they are trying to address an extreme build with results that are like a cheat. Legitimate by the rules, but so extreme that the overall mechanics clearly can't deal with it properly. This is what all "minimaxing" and "meta builds" are essentially about: Looking for results that are beyond intended gameplay with the intention of exploiting it when the game content can't properly support it.

But in doing so, they reduced healing for everyone:

• Players who weren't trying to use such extreme builds.
• Players who didn't encounter such extreme builds.
• Players who were using such builds but were still unkillable after the fix.

To fix a very small aspect of the game, they applied a change that affected everyone. And which may or may not have worked at all.

ESO Specific Example 3

In his article, Rich Lambert reveals that the freedom in the combat system is meant to be kept in check by resource limitations:

The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

And he points out what everyone already knew for a long time -- That players could in various ways get around this limitation, making it moot:

Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities. ...

... At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

And finally, the most critical aspect:

This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it.

Of course this result is not intended. And it is not entirely wrong to allow it -- because a lot of players are using what's in ESO to great effect, without creating extreme builds aimed at going beyond what the system can handle.

But the worst part is, they didn't have an open, honest, conversation with players about it.

They just quietly allowed it. And with basically no plan, even when things were out of control and players were completely circumventing their intended checks and balances.

And now, instead of addressing that real cause -- exploitation of the game system -- they are trying a fix that will address some of the symptoms.

And if these major changes go through, they will affect everyone, not just the players who are running exploitative builds.

How Not to Fix It

Before we even talk about how to fix exploitative builds in ESO, we should really talk about what not to do. What needs to be protected.

• The innocent need to be protected
• The fun needs to be protected

A change targeted at the symptoms of an exploit should not affect players who are not using the exploit, knowingly or otherwise. A change targeting an exploit should not ruin what is possible when the game systems are not being exploited.
So a radical change to how Area of Effect abilities have been used for years is a very unfair move and one dangerous to the overall game system.

But when does a build cross over from "intended" to "unintended" and so extreme as to be an exploit? This is a subjective question, and therefore a solution must not involve this decision.

Remember that what is fun in ESO is also ultimately what allows exploitative builds to be possible. They really can't be disentangled without removing a lot of elements.

How to Fix It

The best way to address the situation is also what a pen-and-paper game between friends would have done: If everyone agreed to play an "extreme" game, then they could have that. It worked because your game was completely separate from what any other GameMaster's group was doing. It was isolated.

So instead of big changes that are unwarranted for the rest of the game, isolate the exploitative builds and the players who use them so they cannot affect other players.

• Other players aren't having performance issues due to overuse of Area of Effects, so they don't need any fixes.
• Gameplay that is already out of intended bounds also do not need any fixes. It's unintended and therefore needs no support whatsoever. It's a problem created by some players -- one that doesn't have to actually affect other players.

But how can we isolate exploitative builds when it is a subjective decision which are within intended bounds and which are not? We can base our decision on a different metric -- PvP ratings.

First, PvP Leagues need to be created. Players get sorted into different leagues based on their ability. This is something that "real world PvP" has when you look at competitive sports. Athletes are sorted by ability into league. Those who are more casual typically happily stay in minor leagues. Athletes who are good and make it their careers, maybe even hiring specialists and buying the most scientific gear to get every little edge -- they aim to be in international, highly competitive leagues, and when they attain that skill are put in a different league.

The crucial part: These leagues do not play each other.

Typically, players who run extreme builds do better, so they would naturally get sorted into a different league and together with their peers -- this sorting based on their PvP ratings.

Over time, exploitative builds would end up playing against only exploitative builds (unless players start exploiting that rating somehow, and you know they will, but that's a different conversation).

And this would actually reduce strife in the game as a whole because players of like mind (about maximizing their advantage through whatever they could do with the game system) would be in the same league, in effect agreeing to play the same kind of game. At the other end of the scale, new players would end up playing new players as well, not bewildered by overpowered players and "cheaters" in their very first matches.

Think of real world competitive sports, a system that has been in place for decades. It evolved a system of leagues for good reason.

If it is not possible to implement leagues into large persistent instances like Cyrodiil or the Imperial City, then a further step would be to move those scenarios into Battleground maps. For example, a map that is a Cyrodiil keep or farm. With Battlegrounds, the total number of participants can be controlled, so performance issues from abilities can be adjusted by changing the maximum team sizes.

And once the system of leagues is working properly, no fixes for exploitative builds needs to happen at all. Performance issues such as those that Area of Effects are suspected of causing can be isolated and investigated and dealt with in the Leagues in which they appear.

Once isolated, the gameplay that is beyond what was intended or supported can be examined in those leagues and with the data gathered, a better decision can be made about what needs to be done, if anything.

Appendix - ZOS_RichLambert post "Update on Cyrodiil Performance & Upcoming AOE Tests"

As part of our ongoing performance optimizations, we have been spending a lot of time analyzing and addressing problems in Cyrodiil. I'm going to give a summary of what has happened over the years, what we have been doing to alleviate the problems, and our future plans.

When talking about ESO server performance, it is important to understand that all abilities in the game (with some exceptions) have a "soft" limit on the number of times they can be executed. An example of a "hard" limit on an ability is an ability that can be executed one time every two seconds, enforced by the server. ESO doesn't have many of those. Instead, we designed the game to have soft limits, which mean you can execute as many abilities as you want as long as you have the resources - Stamina or Magicka - to cast them. There is a global "cool down" timer on all abilities, which is set to 1000 milliseconds.

The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities.

With that in mind, consider how Area of Effect (AOE) abilities work: when cast, they look at a specific target area - almost always the area directly around the caster - find targets, and perform the ability. Each of those steps requires server calculations.

At launch, Cyrodiil’s processes were able to keep up with the number of AOEs cast, because most players couldn’t cast that many of them: they ran out of Magicka or Stamina, so they had to use AOEs judiciously. Over time, as player knowledge grew and regen builds grew in power, more players could cast more and more AOEs before running out of resources.

During these years, we found and fixed many performance issues in Cyrodiil, and we continue to do so. We've fixed significant issues, uncovered more, and continue to find and fix with every Update.

However, one foundational issue remains. At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it. However, as this behavior grew more prevalent, we reached a point where casting so many continuous AOE abilities in such a small area started to overwhelm the server process for that area, leading to situations where the "lag meter" spikes and the server becomes unresponsive for a period of time.

Our initial response to this problem – starting with Update 22 – was to find (and fix) more than a few problems with AOE ability calculations and make them more performant - and we stayed (mostly) ahead of the problem. But as more players reached maximum Champion Level and more players started utilizing this particular method of "AOE spamming", we have reached the point where we cannot fix this issue around the edges: we need to address the core problem, which we will be starting with Update 27.

To do this, we will need to first do some analysis, and we can only do this on the live servers. As much as we try to avoid running tests on live servers, they are the only place where the combination of player behavior, specific builds, and mass battles happen. So, starting the week of August 24, we will be running a series of tests on the live PC servers – both NA & EU, only in Cyrodiil. Please note that we will not be running these tests on any Console servers. Each test should take about a week, but if needed, we will extend the testing time.

The first round of tests we are planning will focus on Area of Effect (AOE) abilities in Cyrodiil and will make it more difficult for AOE abilities to be the only abilities used, adjusting cooldown, cost and regen values of all AOEs (damage and healing).

Specific details on the tests we will be running in Cyrodiil are as follows:

• Test 1 – Shared global AOE cooldown - 3 second timer. This test adds a global 3 second shared cooldown to any AOE ability. This means that when you cast an AOE, you will not be able to cast another for 3 seconds. For example, as a Templar, if I cast Ritual of Retribution, I wouldn’t be able to cast Puncturing Sweep for 3 seconds.
• Test 2 – Individual AOE cooldown - 3 second timer. This test adds an individual AOE cooldown to each AOE ability. This means that when you cast an AOE, you will not be able to cast that same AOE ability for 3 seconds. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.
• Test 3 – No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.
• Test 4 – Individual AOE cooldown – 3 second timer, global ramping AOE cost. This test adds an individual AOE cooldown to each ability as in test 2, but also combines that with a global ramping cost from test 3.

During the times that any of these tests are active, we will be awarding double Alliance Points for anyone active in Cyrodiil.

After we complete the above tests, we may try other combinations of cooldown, cost, and regen values on AOE abilities. However, we need to run these tests first and then assess the data. We will then let everyone know what we found and how we will move forward. We will be very upfront, but please be aware that if these tests confirm our hypothesis, then chain-casting AOE abilities will no longer form the core of the ESO PvP experience in the way it has for the last few years. We would then go through each class and ensure that there are viable builds for each and make adjustments as necessary.

The code for these changes will be going into today’s PTS patch, and we will be running some basic tests throughout the day on Tuesday in Cyrodiil to ensure we’re able to make the above changes without requiring any maintenance or downtime. Once we launch Update 27 for PC on August 24, we will announce when one of the tests are beginning through an in-game announcement and will have a forum thread detailing which test is currently being performed. Remember, these tests will be limited to Cyrodiil, so AOE abilities in other PVP spaces (Imperial City and Battlegrounds) and will remain unchanged for now. We will evaluate how these tests go, and let everyone know next steps.

Thanks for being patient and understanding with these tests. They are extremely important for the team and will help us work towards improving Cyrodiil performance.

Appendix - "disabling proc sets has breathed new life into PvP gameplay and has made fights more enjoyable and fun"

Thanks to everyone for participating in this latest Cyrodiil test where we disabled all proc sets. Similar to what many of you have noted in your feedback, we found this test did not impact performance in a measurable amount; if anything, there was a very slight degradation of performance at times, likely due to a higher population of players in Cyrodiil.

We’ve heard from a lot of you that despite performance largely being the same, disabling proc sets has breathed new life into PvP gameplay and has made fights more enjoyable and fun. To that end, we’ll be leaving proc sets disabled until Update 31 launches in Q3. At that point, we will have implemented some new code so we can have more flexibility to campaign rulesets as it applies to proc sets. We’ll work on applying all this to consoles as well, and we’ll let you know when we have a date for this starting.

Now that we’ve had time to digest a lot of data and information, we have a better idea of next steps and the work needed to produce noticeable improvements to performance in Cyrodiil. This work is complex and will take a fair amount of time and effort. We are committed to improving the PvP experience, though, and we have already begun scheduling out this work. Please note that none of the scheduled work will implement any of the changes we made on PC during past tests, and at this time we don’t plan to run any additional tests.

When Update 29 launches on Monday for PC, we’ll be turning off double AP but proc sets will remain disabled as mentioned above. We appreciate the time everyone spent in Cyrodiil during these tests and all the feedback that’s been submitted.