The Census API Primer v4

Introduction

Daybreak Games provides access to game data through a RESTful API called Census. This system is a centralized service that receives character, world, and static information from game servers so that it may be accessed by various applications. These include Daybreak Games websites, Game Launchpads, 3rd Party Fan sites and mobile companion apps. Access to this data is available at census.daybreakgames.com, which is available using either HTTP or HTTPS.

Census allows you to request information directly from Daybreak Games regarding the status of a game or game entity (e.g. character, item, spell, etc.). All requests to the REST interface follow a pattern and there are different patterns for different types of data. The REST interface will return all results that match the attributes provided within the query string. Data is returned is in the form of a list containing one or more items from the collection provided in the URL. If no results are found, this container will be empty.

The URL Structure is as follows. Any part of the URL within brackets ("[]") is optional and may be omitted):

/[s:serviceID]/[format]/{verb}/{game}/{collection}/[{identifier}][?{queryString}]​

Service ID

A Service ID or Identifier is a short word or phrase associated with your website or your name which is used by Daybreak Games to monitor and provide further capabilities to sites who use the API. A Service ID is not currently required, but is encouraged. Your service must be registered with Daybreak Games to use this facility. The first step to getting this done is to request a Service Key.

If you do not have a service ID, just omit the "s:serviceID" part of the URL:

census.daybreakgames.com/json/get/eq2/item/2906185062

If you have a service ID, provide it as below:

census.daybreakgames.com/s:serviceID/json/eq2/item/2906185062

We are encouraged to use the Service ID both in data queries and in image searches. For example this image:

census.daybreakgames.com/s:eq2wire/img/eq2/icons/600/item/​

HTTP (GZIP) Compression

If you are making calls to the REST API from a server process, you should consider enabling HTTP Compression on your request. This is already enabled for most browsers, but you need to set an HTTP header when requesting through code.

Accept Encoding: gzip
This will reduce the transfer size of a character request dramatically.
More information about HTTP Compression may be found here.

PHP 5 should handle this automatically if you are using get_file_contents, etc.

If you are using PHP's cURL to make your data requests, then you must set an empty CURLOPT_ENCODING to automatically send the list of all supported encoding types in the request header (which will include gzip). Otherwise, no Accept-Encoding header gets sent.

curl_setopt($curlResource, CURLOPT_ENCODING, "");​

Data Format

Census supports returning data in eXtensible Markup Language (XML) and JavaScript Object Notation (JSON). The recommended format is JSON as it is a very efficient way to access data for most applications. If the XML format is requested, then the results will be provided in a parent <results> element. If JSON results are requested, they will be in a dictionary( {} ).

Using a plugin such as JSONView, JSON data can be presented in a "beautified" format that is easier for humans to read. This means that you don't have to run the output through a plugin or filter just to make sense of the JSON data. JML is particularly useful when troubleshooting queries as you may hover over any field and the data path will pop up in a tooltip.​

The available data formats are:

• xml
• json

In order to access data across the domain boundary, one may want to use JSON Padding3 (JSONP). To do this, append a callback attribute to the query string of the request that uses the json format.

/json/get/eq2/item/2906185062?callback=foo

The results from this request will be padded within the function name, foo(), so that they may be evaluated by the requesting web site within the correct domain.

Verbs

The verb or "action" informs the REST interface about the type of request that is being made. Supported verbs are...

• get - retrieve the objects that match the request criteria
• count - retrieve the number of objects that match the request criteria
• status - retrieve status information (currently used for server status only)

Game

In our case, we are using eq2 for EverQuest II.

Collection

Supported collections are dynamic and dependent on the game providing data to the service. Collections might include Characters, Items, Spells, Guilds, etc. The list of available collections may be viewed here:

http://census.daybreakgames.com/xml/get/eq2/

Some data may be omitted from a collection due to certain policies set by each game team to protect player privacy and/or business requirements. Examples of these policies might include:

• Characters may be marked as private so that they are omitted from searches and rankings.
• Items may be hidden until they are "discovered" naturally by a player.

Identifier

Each object has a unique identifier, which is used for a direct link to the information about that object. This can be found within the structure by the top-level “id” field. All results will have this identifier, even if all other information is hidden.

There are also identifiers that are used in reference to other collections. For example, consider the following collection information from eq2:

"appearanceslots": [{"item": {"id": 3098863614},"id": 2,"name": "head"},{"item": {"id": 2334025620},"id": 3,"name": "chest"}]
The item.id value is a reference to a particular object within the "item" collection. When presenting this information, one can put a link to another query to the REST interface or to resolve that information within its parent object. For more information on how to do this, see the Query Commands section for c:resolve.

Game Images (Attachments)

Game images are also available through this service. These include images representing characters (paperdolls, headshots) or in-game items or abilities. Images are typically returned as 24-bit PNG images with 8-bit transparency. Requests for game images follow this pattern:

/img/game/collection/identifier[/imageType]

For example, a certain piece of armor may have this icon:

census.daybreakgames.com/img/eq2/icons/3635/item/

Each result within a collection can have a number of "attachments" - usually an image, but this may be any file. For games that support many attachments, there can also be a category and items within each category. The category and the ID for items within each category is game-specific.

When retrieving images from the icons collection, the supported values for imageType are:

• item - a 42x42 transparent PNG item icon
• spell - a 42x42 transparent PNG spell icon

​

NOTE: The use of the icon collection is deprecated. Use icons instead.

When retrieving images from the character collection, the supported values for imageType are:[/COLOR]

• paperdoll - a full picture of the character
• headshot - a picture of the character of from the neck up (suitable for icons)
• housethumb - a picture of a in-game house as published
• petpaperdoll - a full picture of the character's pet

​

You may use the c:attachments directive in your queries and it will include URLs to the various images directly into the information returned for that object. These URLs will use the Game Image pattern and correct imageType.

A field named itemInCategory is returned if there is more than one attachment within a category. At this time, the only example of this is for a player who owns multiple houses. Their character will have multiple housethumb attachments.

 

The Daybreak Games Data Feeds API (Census) runs on a Mongo Database system which is an optimized NoSQL solution. Users wanting to request data from Census do so with RESTful API calls. In other words, GET queries where each parameter either requests certain data or filters the resulting data to refine your results. As with any GET query, the first parameter must be preceded by a question mark (?) and subsequent parameters must each be preceded by an ampersand (&) character.

This query would return all level 80 items of type Weapon with a quality (tier) of MYTHICAL.

census.daybreakgames.com/xml/get/eq2/item/?leveltouse=80&type=Weapon&tier=MYTHICAL


Searching

Field Values​

Each request for data may be filtered by the value of certain "fields". Each field in a query string corresponds to a particular field within a particular object within a collection (characters, spells, items, etc.).

Within Census data, fields have a hierarchy or level structure. When accessing a field that is parented under another field, you may reference it by using a dot (".") notation. For example, considering the following data:

"item": {"id": 4126241497,"typeinfo": {"equip_optional": 1,"name": "expendable","casttime": 1.0}}

In order to find all matches for the name "expendable" within typeinfo, one would use the following query string:

?typeinfo.name=expendable
​

Numeric Fields​

When more than one search condition is provided within the query string, on a number, it acts as a boolean 'AND' operation. For instance, consider the following query string:

?id=]10&id=[50

This request would return data where the value of the id attribute was between 10 and 50 (inclusive of 10 and 50).


Text Fields

When more than one search condition is provided within the query string on a text string, it acts as a boolean 'OR' operation. For instance, consider the following query string:

?given_by=racialtradition&given_by=focusabilities

This request would return data where the value of the given_by attribute is EITHER racialtradition OR focusabilities.

A complete list of search modifiers appears below:​

Modifier Example Description

field=<10 --- Include objects where the value is less than 10.

field=[10 --- Include objects where the value is less than or equal to 10.

field=>10 --- Include objects where the value is greater than 10.

field= ]10 --- Include objects where the value is greater than or equal to 10.

field=/string/ --- Include objects where the value contains a particular "string".

field=i/string/ --- Case insensitive search for a particular "string".

field=^string --- Include objects where the value starts with "string".

field=string$ --- Include objects where the value ends with "string".

field=!string --- Include objects where the value does NOT match "string".

field=*regex --- Use a regular expression to match values within the field.


Regular expressions or "regex" provide a great deal of power for searching for objects. Census supports Perl-compatible regular expressions. The "/" and "^" modifiers above are actually examples of simple regular expressions. It is beyond the scope of this document to fully explain regexes, however there are a number of resources available to learn about them, including here and here.

NOTE: Use of i/string/ for case-insensitive searching of Character, Guild, Item, and Spell names is deprecated in favor of standard searches against their lowercase equivalents. Search performance will be substantially improved by using these:

• character.name.first_lower
• item.displayname_lower
• guild.name_lower
• spell.name_lower

​

Has

The presence or absence of certain fields in data can be just as valuable as the fields themselves. For instance, the following query returns items that have the typeinfo.growthdescription field (eg. green adornments or Spirit Stones):

?c:has=typeinfo.growthdescription

There is no separate "Has Not" or "Doesn't Have" query. Instead, if you need to find entries where certain fields are missing, add a ! in front of certain fields:

?c:has=typeinfo.growthdescription,!growth_table​

Distinct

Sometimes you wish to know all the distinct values for a specific field. For instance, to find all possible qualities (tiers) of items:

census.daybreakgames.com/xml/get/eq2/item/?c:distinct=tier

NOTE: c:distinct may not have good performance on extremely large collections. It should be used to build internal cheat sheets/lists to work against. C:distinct is exceptionally slow when used against the Character collection.​

Resolve

Some collections have related data which may or may not be needed when accessing a collection. For instance, when you retrieve a Character, a stripped down list of Spells and Achievements are included, however the underlying data such as the name, description, and stats for these data points is not. If you wish to add this data to your query, you may use ac:resolve to "join" collections together.​

Example of Resolves for Guilds

Before EQ2 started adopting GUIDs (globally unique IDs) for Characters and Guilds, the game used a combination of DBID and WorldID to specify a unique character or guild. The same DBID could exist on multiple servers, so you had to use both DBID and WorldID. When you perform a standard 'pull' of a Guild, you only get the DBID of each character. For example, a guild named Honor on the Permafrost server:

census.daybreakgames.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost

<member_list><member dbid="208021" name="Idemu"/><member dbid="221484" name="Gige"/><member dbid="263553" name="Klehost"/>.</member_list>


To also get the GUIDs of characters in a guild, you'll need to add a c:resolve:

census.daybreakgames.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members

<member_list><member dbid="208021" id="867583601813" name="Idemu"/><member dbid="221484" id="867583615276" name="Gige"/><member dbid="263553" id="867583657345" name="Klehost"/>.</member_list>

NOTE: Characters that have not logged in since Nov 2011 will not have a GUID.


You can expand your c:resolve to load additional Character data besides the GUID as demonstrated below:


census.daybreakgames.com/xml/get/eq2/guild/?name=^Honor&world=Permafrost&c:resolve=members(displayname,type)
​

The current list of resolves can be found by browsing Collections for <resolve>:

• Character
  • Spells
  • Factions
  • AppearanceSlots
  • EquipmentSlots
  • Achievements
  • Warders
  • Statistics
  • Dungeon_Items
• Guild
  • Members
• World
  • Status

​

NOTE: Resolves such as c:resolve=spells or c:resolve=achievements when performed against one or more Characters can take several seconds to perform and also generate a substantial amount of data. If you are feeling really lucky, you may do a c:resolve=all to return as much data as is possible.

 

Limiting Fields in a Resolve

You can limit the returned data within a c:resolve by indicating certain fields in parentheses:

census.daybreakgames.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(name,level,description,duration)

Just as you can use the syntax above to show fields, you can add a ! in front of the query to hide fields instead:

census.daybreakgames.com/xml/get/eq2/character/?c:show=displayname,spell_list&c:resolve=spells(!icon,!cost,!tier)

Just realize that you cannot limit or try to sort based on a field that is contained within the resolve. The join happens, then operations can begin.​

Controlling Output

Once you have written your query, you may wish to control the output, specifying which fields are returned, and sorting the results so that you can page through them.

Show

Only include the specified field(s) in the returned data.

?c:show=displayname,name,account,guild,world,skills​

Hide

Exclude certain field(s) from the returned data.

?c:hide=achievements,spells,stats

Start

If you are paging through items, spells, characters, etc. you can use c:start to start with the Nth object within the results of a query.

?c:start=50​

Limit

By default, any query will return a single result unless you specify a c:limit.

?c:limit=99

NOTE: The maximum number of results you can request at once is hard limited to 100 if you choose not to register a Service ID with Daybreak Games.​

Count

The following query returns the number of level 95 characters:

census.daybreakgames.com/xml/count/eq2/character/?type.level=95

However you may want to perform a query AND get a count of the number of results in the same request. This query would return the first ten level 95 characters and also the number of level 95 characters found:

?type.level=95&c:limit=10&c:count=1

Returns:

<character_list count="57695" limit="10" min_ts="1358361070.009923" returned="10" seconds="0.596140"><character dbid="1639738" displayname="?????? (Barren Sky)" id="1335736468794" last_update="1358361072.616647" playedtime="3154929" ts="1358361072.616647" version="1" visible="1">..</character_list>​

Sort

You can sort the results of your query with the c:sort modifier. You may sort by more than one field in ascending or descending order. For example, to sort ascending on field 1 and then descending on field 2:

?c:sort=field1:1,field2:-1

For example, to get the 20 characters with the highest number of completed Collections, this query would be used:

census.daybreakgames.com/xml/get/eq2/character/?c:sort=collections.complete:-1&c:limit=20&c:show=displayname,type.level,world

To get a list of level 91 Grandmaster spell choices sorted Alphabetically:

census.daybreakgames.com/xml/get/eq2/spell/?tier_name=Grandmaster&level=91&c:sort=name:1&c:show=name,tier_name,level&c:limit=100​

Reverse

If you need to reverse the order of the returned results in a query, c:reverse allows you to do so. The main use for this is to allow you to sort by newest first, but show the results in chronological order in your browser. That way you can see the 25 newest characters, logs, etc in order from oldest to newest. A simpler way to do this is to add a :-1 to the c:sort field, such as c:sort=name:-1.

?c:reverse=1​

Attachments

Attachments are images or other files which are "attached" to a specific field. Generally this is returned as a url path to download the file.

?c:attachments=all

For EverQuest II, the valid categories for c:attachment are paperdoll, headshot, petpaperdoll. You may also specify all to return all available attachments. The format is generally a fully-qualified URL to the external file(usually an image) that can be used within that object with additional, useful metadata.​

Explain (Performance)

Census consists of a cluster of five MongoDB servers. Searching the database against fields that are indexed typically returns results quickly. For example character names, item names, guild names, etc. are all indexed. However most fields are not indexed and thus queries that search against those fields will be somewhat to considerably slower. You can determine if one of your queries is using an unindexed field with the c:explain parameter.

?c:explain=1

If"cursor" is returned with a value of "BasicCursor", then the field you are querying is not indexed.

If "cursor" is returned with a value of "BtreeCursor..." then you are using indexes.

Every query, indexed or otherwise, returns the number of seconds that were required to run the query.​

Case

• c:case has been removed and is only provided here for historical reasons. All requests are now case-sensitive. Use either the i/string/ format or the lowercase fields for fastest performance when searching.

​