Busting SWTOR bugs: “everything else is secondary”

Rich Vogel, SWTOR’s executive producer, has hammered out a developer blog post giving a little more insight into how BioWare are tackling the inevitable bugs that any MMO has. It’s a strategically timed post given the griping over patches and remaining in-game issues.

As you’ll read below, Vogel is pretty emphatic about what the development team’s priority is: fixing bugs, with everything else being “secondary”. There’s a lot of fairly basic information in the post, but there’s also some gems. One that tickled my fancy was this one:

Fixing bugs in a game as massive as The Old Republic is a little trickier! The PC is a very complicated platform to develop for and getting the game to work flawlessly on every PC ever made is unlikely, but we want to fix as many bugs that affect players as we can.

I’m not sure merging the issue of bugs versus PC configurations is anything but a little disingenuous – sure some configs will affect gameplay but they’re not bugs as such unless they occur within the range of the acceptable system specs for the game.

A second issue is: surely the complexity of the system was taken into account with the development process, so I’m not sure that really clarifies a lot either.

Finally, I’m going to make a cheap shot at you BioWare: aside from the obvious expense of developing for the Mac side-by-side, its configurations are markedly simpler so perhaps that might’ve been a route to explore?

Anyway, if you’re not that aware of how bug hunting goes on in an MMO, read on for the detailed overview and we’d love your comments on whether the post from Rich Vogel clarifies things for you?

As one of the original members of the development team and having helped create the studio that brought you Star Wars™: The Old Republic™, I’m privileged to have been on this incredible journey that culminated in launching the game. Now that we are through launch I felt this was a great opportunity to talk about how we balance our efforts when it comes to fixing bugs and developing new content.

The good news is that we have a large team dedicated to the game, which gives us the flexibility to both fix bugs and develop new content. I want to make it very clear here that the live game service comes first though, and fixing bugs is our team’s first priority. Everything else is secondary.

Let me start by explaining how we go about determining when to fix an issue in the live game, whether it is a bug or an exploit. Let’s begin with exploits – those cases when we determine that a bug in the code can give an unfair advantage to a player. (Remember, not all bugs are exploits!)

If an exploit is discovered that threatens players’ experience in the game or the in-game economy, we will usually try to create an Emergency Patch to fix that exploit. Before we declare something an emergency though, we have to validate the issue with our internal Quality Assurance, or QA team. They gather data working with our Customer Service, analytics, and development teams to determine the extent of the exploit and the steps needed to reproduce it. Once we have the steps, we work on a plan to fix the issue. Depending how long the issue will take to fix and how risky it is, we will decide if we need to temporarily disable the cause of the exploit in the game.

Generally speaking, we can fix these types of issues pretty quickly once they are detected in the game. Our turn-around time is really good in this area and we’ve dealt with some recent exploits quite quickly. That’s because we have a dedicated Live Support team that handles these issues, made up of on-call engineers, designers, artists, and producers, who are available around the clock. Our extensive in-game metric tools are also always monitoring the game and generating reports – meaning that if anyone is exploiting, we are going to find out about it. Our main goal is to ensure you have a fun, balanced and fair gameplay experience in The Old Republic. We won’t stop looking for exploits and taking steps to correct them to maintain a fair game.

Fixing bugs in a game as massive as The Old Republic is a little trickier! The PC is a very complicated platform to develop for and getting the game to work flawlessly on every PC ever made is unlikely, but we want to fix as many bugs that affect players as we can.

Not every bug is created equal. Some bugs – even ones that might look simple – take a long time to deal with and some might not be able to be reproduced (as they may be random occurrences under special circumstances). With that in mind, we use a triage process to prioritize our bug lists. This helps us determine which bugs we fix first based on the bug’s severity (which is how many people it affects). We estimate the severity of bugs by looking at volume of tickets sent to our Customer Service team, Forum posts, and using our telemetry data from the game, amongst other things.

The first thing the development team does every day is triage incoming bugs. Team leaders from different groups within BioWare Austin meet to review the incoming bugs, which are pulled from our bug tracking software (managed by the QA team). QA gathers these bugs from several sources (player reports, internal QA, Customer Service tickets, and Community). Before any bug is tracked, we need to verify whether it is a real, reproducible bug by trying to reproduce it ourselves. Sometimes that cannot be done either because we don’t have enough information on the bug (so give us plenty of detail in bug reports, please!) or because there are special circumstances that cause this bug to occur which are not found through normal play patterns. If that is the case we reach out to the community or contact the player to help us reproduce. We have done this many times with great success. Thanks to all of you who have helped us track down these types of bugs.

Let’s get back to our triage process. Every day we review incoming bugs as well as existing bugs to determine the order in which we fix these bugs. The highest priority bugs are what we call ‘blocker bugs’. These are any type of bugs that block a player’s progression through the game, and they’re important enough that they may be fixed via an emergency patch. Whether a bug is fixed in an emergency patch or not depends on the complexity of the bug (which dictates how risky it is to implement) and whether or not there is a workaround available to allow a player to progress. If there is a workaround available, we will usually wait to push that bug into the next publish and ensure players know about the workaround. This allows any players affected to continue to progress, while giving us time to ensure the correct fix won’t unduly affect the rest of the game.

The balancing act of fixing bugs while ensuring we maintain a stable game is always tricky. We have to make sure that when we publish a fix to the live game it doesn’t alter the game code in a way that can make things worse. This is a normal part of complex software development, but it means we have to assess the risk on what we publish to the live game very carefully. The more bugs we fix with each publish, the higher the risk of breaking things, as we’ve changed more game code. That is why we try to limit the number of bugs and types of bugs we address with each patch. If we can fix a lot of low risk bugs we will push more out. Our most important goal is remove as many annoying bugs as fast as we can; currently we aim to push out patches which help fix bugs weekly but that can change depending on the severity of the bugs.

Please remember, if we don’t respond publicly to a bug that does not mean we are not going to fix it. We want to make sure we provide a good playing experience to our players. That is why we have a large team of people working every day to bug fix and remove exploits from the game.

In terms of developing new content I wanted to let everyone know that we are committed to publishing new features and content on a regular basis. It is important to make sure the game is always changing. We have several strike teams working on future game additions. We have a huge list of features and content we want to put into the game and we are constantly adding to it based on feedback we get from the community and from playing the game ourselves. Every single one of us on the development team plays the game, and we see the same issues you do and where we need to expand.

Our most important goal is to make sure you continue to have fun playing our game and we will do everything in our power to make you feel that you are getting your money’s worth every month. Thank you very much for playing!

Over to you: do you think BioWare are wearing down the onslaught of bugs?