I have two different processes to explain here : the major patch process, which brings new content features and fixes in batches, and the minor patch process, used to apply small fixes to the live versions rapidly. At the end, I illustrate the theory with a concrete example : last week's patch.
A] Major patch process
Diagram 1 : Major patch process
As you can see, there are three phases :
- Development
- Test (internal tests and ATS tests)
- Live patch
The first phase, the development one, includes the work of different teams : the coders, the level designers, the 3D artists, etc. They have different working techniques, each of them adapted to the work they perform. For example, the coders are using an adapted subset of the Extreme Programming techniques (see especially the Planning Game, Small Releases, Simple Design and Design Improvement for the current subject). They get periodically a set of tasks corresponding to approximately two weeks of work : the iteration.
At the end of some of the coders' iterations, changes to the code and work coming from all the teams are flagged to create a branch : this is basically an internal version number attributed to current code and content. Its form is X_Y_Z, where X is the major version number (currently 1), Y the chapter number (currently 2), Z the current branch number since last chapter (currently 7). Notes filled by the developers when they modified code are compiled to keep track of changes for the current branch : the branch notes; they are absically a technical version of the release notes.
2) Tests
a) Simple regression test
The first thing the testers team does when they get the new branch version to test is to perform a basic regression test. For two days, testers go through a simple checklist, to be sure that the general systems of the game are working correctly, since software as complex as an MMORPG are prone to the butterfly effect, which says that small variations on a dynamical system can have large repercussions.
When an important issue is raised at this stage, a fix is worked on and the version is tested again before entering the following stage. This can of course create delays, but they are needed to avoid breaking the ATS version too hard.
b) Deeper internal tests & ATS
Then, when everything is fine, a patch is created and applied on the ATS. The in-house testers continue their tests, going through a deeper checklist of both existing and new game mechanisms, based on standard checklists and branch notes.
As ATS and internal tests are being performed, bugs can be detected and other issues can be raised. There are three different cases here for each issue : it can be immediately corrected if the fix is small, quick and safe enough; the related feature can be disabled in the current branch and rescheduled for the next one; or if the dependencies of the required changes are too big, the patch is delayed. The basic timeline is two weeks here, but - again - it can be extended depending on how testing goes.
When everything is fine, we enter the final (and most stressful!) stage : the live patch.
3) Live patch
Here, the first thing is to announce the patch to you (on the boards, and now also in the news). Then the patch and release notes are being prepared, and finally, at the announced date and time, the servers go down : the patch is being applied.
While the servers are patched, the sysadmin team use the downtime to update the Linux servers and perform a pre-patch complete game saves backup.
When the servers are patched and ready, the CS teams give them a final check over, increases their alert level, and... tada! You can finally enjoy the so hard worked good bits prepared for you.
B] Minor patch process
Diagram 2 : Ninja patches process
As every MMORPG player knows, there is no such thing as a perfect MMO game, or even patch. So there is a need for a reactive process to deal with pressing issues on the live servers - you wouldn't want us to use the iteration-release process for that, as you would have to wait a number of weeks for a fix to enter the game !
So we have one, which can be described as the "safest path to the quickest fix". When an urgent fix is required, there is a delicate trade off between speed and safety, as safety requires testing and takes time. So there is no general timeline here, as it really depends on what is being fixed.
Back to the days just after the release, we had a hair-raisingly reactive process (which we called internally the "ninja patch"
). Some of the developpers could get a call in the middle of the night, crawl out of bed, fix a problem and roll out the patch within 10 or 20 minutes! But - as described in News from the devs, the lack of proper time to develop and test them wasn't without consequences for the game in the long term. Even the simplest fix needs testing before it can be let loose on a live server. This is why we stopped developing ninja patches and adopted the minor patch procedure.Step by step, the process for minor patch is the following :
- Step 1 : A decision is taken to fix a problem using the minor patch procedure
- Step 2 : The fix is developed
- Step 3 : The coders and testers put together a short test procedure for verifying the fix and checking for side effects
- Step 4 : The fix is tested until the everyone is satisfied that it works correctly - time is important so the tests are not nearly as thorough as they would be for a major patch
- Step 5 : The patch is rolled out to the live servers and announcements are posted
To illustrate theory, there is nothing better than getting a bit more concrete : here I will review last week's patches (53, 55, 56).
1) Patch 53 - Major patch process
- 18/02 : The coders finished an iteration.
- 21/02 : A branch was created and referred to as the 1_2_7 internally.
- 22/02 to 25/02 : The quick regression test was performed by the in-house testers.
- 25/02 : As the test went fine, green light was given and the 1_2_7 branch was patched to the ATS.
- 03/03 : Issues with in-game texts on new rites were raised, so their introduction was postponed.
- 07/03 : An issue with the /afk command was discovered, so the command was deactivated.
- 09/03 : As no further major issues were discovered, the patch was given a green light to go live. The patch was assigned the number 53, and was immediately announced to you.
- 10/03 : The release note was prepared, and published as usual while the servers were down, to give you something to read while you can't play. Finally, the patch went live.
Two issues were fixed in subsequent patches. There was a known language-dependent problem for the name of mounts in some places in the game. This was a translation problem and had been fixed some time before the patch went live. However the fix had not yet had a green light from testing so we prefered not to delay the main patch. The correction only required a client patch, so it was applied later the same day (March 10th) without interruption of the servers. When new players logged in the game the patch was applied, while the others remained in 53 until they relogged out.
After the patch we discovered another issue concerning some players' crafting tools. Because this issue was not trivial and we wanted to ensure that the correction went through a proper test cycle, the customer support team reacted immediately, with all of the customer support staff volunteering to replace problematic tools for free. The fix was cumulated into a minor patch with a couple of other points and patched up yesterday (patch 56).
--
Xavier.