Announcements

Mass Testing

By CCP Habakuk

What is Mass-testing?

Mass-testing events are organized by the CCP QA department on the test servers Singularity and Duality in order to test parts of EVE, which need a high number of characters participating at the same time. In most cases the main interest is on the performance of the server and the client in extreme situations like massive fleetfights.

Mass-Testing provides the following benefits for CCP:

  • allows to gather performance data for EVE (both client and server)
  • allows to test high-priority changes by a large number of EVE players
  • allows to get critical feedback about new features/changes to EVE, before they are released to Tranquility (or "TQ," EVE's production server).

Why is this not being tested by CCP in-house?

Public mass tests are indeed not the only way to test performance related changes. Especially thin clients are a very important tool, which are slimmed down EVE clients without graphics or user interface. For some tests thin clients are much better, for example due to the possibility to repeat a test with exactly the same environment. BUT: Some problems slip through thin client tests, as automated tests never have the same behavior as real players in a fleet fight. For example: Many players get nervous and might click the jump-button multiple times, when the gate jump is very slow - but not all of them. Thin clients also won't find UI-related problems, for example if the overview is causing client-side performance problems.

The standard way to deploy a server performance improvement is the following: First the change is tested locally by the developers and/or QA to make sure that it does not break anything (major). Then it is tested with thin clients - this is the best opportunity to get clear numbers on how much the server performance is improved. Next step is to run a mass-test on Singularity to make sure that it still works with real players. If everything is fine the improvement is deployed to Tranquility at the next opportunity. In most cases the improvement is first turned off on TQ and then turned on a few days later to be able to monitor it better.

How to join a mass test event?

Prerequisite: You need an active account on the test server (Singularity or Duality) , which should be the case if your account was active on Tranquility for the last few months

  • If your account is not active on the test server, please visit the reactivation thread in the test server forum for more information about how to get it fixed.
  • Install / patch a client to be able to connect to the test server, instructions can be found on the Singularity page or the Duality page.
  • It is a good idea to run LogLite, for the case, that any problems are occuring. 
  • As soon as you are logged in: join the chat channel MassTesting, where you will find more detailed instructions for the test in the MOTD (directly before the test).
  • Follow the instructions, which are given in the MassTesting channel by a CCP dev
  • CCP devs and ISD bug hunters are present about 30 minutes before the mass test and help with forming fleets and moving players to the start system.
  • Do NOT engage any other player in combat during the mass test, except when told to do so. Breaking rules during the mass test makes the mass test less useful for CCP and might result in a ban from test servers.

In most events the following instructions will be valid, but sometimes we have to change them for special tests:

  • Use the channel MassQuestions for questions, talking and immediate feedback. The channel MassTesting is normally moderated during mass test events, to make it easier to follow the instructions.
  • For joining a fleet we normally use the channels FleetW and FleetX (and sometimes the channel FleetY). Please do not use other channels to ask for a fleet invite (except if no fleet-invites are sent at all during an event, then ask in MassQuestions).
  • You can use the command /moveme to be moved to the start system
  • Most tests include a fleet-fight at a gate - a PVP ship for medium range is normally useful.
  • Do NOT use smartbombs, bombs or warp-bubbles, except if we ask for it.

What to do after a mass test event?

  • Write feedback in the feedback thread, which will be posted in the testserver forum. The thread will be linked at the end of the test in the MassTesting channel.
  • If you noticed any bugs during the mass test: please write a bugreport through F12 - "Report Bug"
  • If you gathered logs with LogLite and want to send them to us: Please create a bugreport and attach the compressed logs (.rar, .zip or .7z). If your logs are very large you can either attach several files (which is easier when running logserver in server mode) or you can upload the compressed logs to a 3rd-party file-hoster and give the link in the bugreport.

What settings shall I use?

  • It is useful (but not required) to run the client in windowed mode, to make it easier to change to LogLite.
  • The FPS can be displayed with the shortcut Ctrl+F, we often ask for it through a feedback survey or a forum thread
  • During large fleet-fights it can be useful to disable brackets to improve the client performance, which can be done by clicking the small white triangle in the top left of the overview window and selecting "Hide all brackets" (or use the shortcut ALT+SHIFT+Z, if you have not changed the default shortcut). 
  • It is possible to get details about the client performance (for example memory-usage) by opening the client engine tools with CTRL+ALT+SHIFT+M.

Free skill points

As small incentive and reward for participating in mass test events CCP QA adds two million free skill points on our standard test server Singularity to the characters which were present in the mass test.

Rules and clarifications for free skill points:

  • The character has to participate in the mass test. If anybody joins late or leaves early he might get nothing.
  • Rule-breakers might get removed from the list.
  • 2 million skill-points per mass-test, they are normally applied one or two days after the mass test.
  • When a new mirror is applied the skill-points of the last five mass tests (max. 10M SP) are reapplied (within 2 days after the mirror).
  • Questions, complaints about missing skill-points: Please use the appropriate forum thread about the mass test or the new mirror in the test server forums.

More info

  • You can join the mailing list "Mass Testing Info" on Tranquility for getting reminders about mass tests (normally 1 day before the mass test).