The API Dev Blog Trilogy - Volume Three -- He who controls the data (Tyrannis 1.2) | EVE Online

The API Dev Blog Trilogy - Volume Three -- He who controls the data (Tyrannis 1.2)

2010-10-04 - By CCP Prism X

...controls the universe!

Now that we are all aware of the coming data type changes that I covered in my earlier blog it is time to move on to the new and the improved! Originally we had planned on waiting with these changes until the next expansion but, seeing as we had to put out the type changes, we figured we might use the opportunity and give you some other shinies to play around with.

CCP created the API Test Service to train the code...

Up until now any development against new or updated API features has had to wait until expansion release as the only API available to develop against is the live production API. With the incoming changes listed below, as well as the data type changes mentioned in my earlier Dev Blog, the need for a public staging API to develop against has never been greater. The API Test Service will use the same API keys as TQ and will, in most cases, run against Singularity.

The test server can be accessed through: //apitest.eve-online.com

At the time of publishing the test API should be fully updated in accordance with the contents of my [previous blog](//www.eveonline.com/control/The API dev blog trilogy - volume two - the types they are a 'changing (tyrannis 1.2) "//www.eveonline.com/devblog.asp?a=blog&bid=798") as well as this one. If you encounter any errors on the test API please do report it back to us through the Bug Reporting system. Please file all API Test Service BRs under the Technical category, clearly referencing the API in the title as well as setting the Server/Build to Singularity and its build (if possible).

We do of course reserve the right to close access to the test server at any time due to an Incursion of heavily armed Murphies. It is also possible that we'll close down access to the public but allow the developers to request access to it. But these are all future thoughts. For the moment it is open to everybody with an API key.

Father... father, the new API Cache Solution has awakened!

The first thing anyone notices on their first glance on the API is the state of the caching service it utilizes. 

Firstly the CachedUntil values on many of the calls have not reported the actual time of cache expiry. Rather they report the NOW + CALL_CACHE_TIMER as the CachedUntil value for the call. This results in repeated calls that return the same cached result constantly reporting a new cache timer. That can only result in developers simply not trusting the CachedUntil value. As we'd like people to trust our caching service, all CachedUntil values should now return the time of the actual cache expiry.

Secondly the cache was not shared between our servers. This doesn't bother our users as much as it bothers us. Occasionally some really weird defects would pop up which were the result of mismatches between two different connections getting different cache returns. Convincing yourself that this is the case can be quite difficult a month or so after the incident when you cannot even trust the CachedUntil values reported. More importantly it put some serious developmental constraints on the API as memory is a finite resource and utilized very inefficiently with the same data cached all over the place. In order to give ourselves some room to develop new features for new community tools we have moved to a shared memcached solution.

Thirdly, and most annoyingly for you, a lot of the caching "strategy" involved rate-limiting calls from given userID/apiKey pairs. That is a flat-out refusal of service for the lifetime of the cache. This would result in the inability of using the same key from two different applications, if and only if they connected to the same cache pool, as well as general frustration with not being able to query the data you want if you lose your application side cache. Such behavior is neither useful nor intuitive and with the new cache server there is no need for this. Work has already started on migrating calls that use such a caching approach to a more useful strategy. Hopefully that will all get done for Incursion.

The new calls must flow...

A couple of new calls have made it in with the major architectural changes and the MailMessages call has also been reworked to the point where it deserves a mention as a new feature. These calls are currently available on the test server. Note that cache timers are subject to changes.

Call Name: /char/MailMessages.xml.aspx
Call Description: Returns a list of all mail headers in a descending order. It can page back in history with the use of optional inputs. Returns 50 headers by default but can return anywhere between 1 and 256 headers in one call. Also acts as a control mechanism for mail body access.
Required Input: userID, full apiKey, and characterID.
Optional Input: fromID and rowCount.
Cache Timer: 15 Minutes to first page refresh (Only checked if no fromID is supplied).

Call Name: /char/MailBodies.xml.aspx
Call Description: Returns the bodies of headers that have already been fetched with the MailMessage call. Call will also return a list of missing IDs that could not be accessed. As was mentioned above, bodies cannot be accessed if you have not called for their headers recently.
Required Input: userID, full apiKey, characterID and ids. Ids is a list of the messageIDs you want to fetch similar to the characterName and characterID calls.
Cache Timer: Mail bodies are not expected to ever change and are therefore cached forever.

Call Name: /account/AccountStatus.xml.aspx
Call Description: Returns basic account information including when the subscription lapses, total play time in minutes, total times logged on and date of account creation. In the case of game time code accounts it will also look for available offers of time codes.
Required Input: userID and full apiKey.
Cache Timer: 15 Minutes

Call Name: /eve/CharacterInfo.xml.aspx
Call Description: This one is pretty unusual compared to other calls. It will accept a keyless request as well as requests with both types of keys and return data according to the access level. Without a key it will return the same data as a show info call on the character would do in the client. With a limited key it will add total skill points as well as the current ship you are in and its name. Supplied with a full key your account balance and your last known location (which is cached, mind you) will be added to the return.
Required Input: characterID.
Optional Input: userID and limited OR full apiKey.

If we code without rhythm, our calls will not work.

Besides fixing the cache timers on all the pages we've put an effort into resolving any BRs that have made it to my desk. Here is a quick list of fixes:

  • FacWarSystems

    • Now reports all factional warfare systems.
    • Now reports owningFactionID for all systems.
    • Now reports a 0 in the occupyingFactionID for un-occupied systems.
  • FacWarStats

    • Now returns proper data in accordance with client data.
  • Standings

    • Now only report NPC standings as contact pages have replaced player based standings.
  • CharacterSheet

    • Now reports characters ancestry and DoB (Date of Birth).

I'm completely out of Dune references...

That's it for Tyrannis 1.2 API changes! Most of the time has gone into resolving technical debt to make room for future development. When they are completely over and done with we can hopefully start implementing calls from the many wish lists out there to make development of new and awesome community tools possible.

As always I will continue to look for any excuse to post on our forums and will be grateful for any opportunity you provide for me to do so in the comment section of this Blog.

Best Regards,
CCP Prism X
EVE Online Database Developer and Acting API-Dude