API CHANGES

New c:resolve value: "all"

You may now use c:resolve=all which will resolve all defined resolve values for the given object.

This is best used when you are developing your queries.  When you figure out the data that is needed, it is best you explicitly define the c:resolve (and potentially c:show) values to make your queries as efficient as possible.

 

New fields for search and sort

There are now 4 fields new fields available to improve performance for searching. They can also be an effective for sorting.

character.name.first_loweritem.displayname_lowerguild.name_lowerspell.name_lower

These will contain a lowercase equivalent of the data in the following fields (respectively)...

character.name.firstitem.displaynameguild.namespell.name

These fields can be used so that you don't have to perform a case-insentive function on the query. These fields will also be indexed so that your "begins with" searches should be blindingly fast.

So, instead of...

census.daybreakgames.com/jml/get/eq2/character/?name.first=i/^blah/

...you should use...

census.daybreakgames.com/jml/get/eq2/character/?name.first<span style="color: #ff0000;">_lower=^blah

You can also use this field to sort your results.  Case-sensitive sorts will be A-Z then a-z.  Using this field, all of the A's (and a's) will be next to each other.

census.daybreakgames.com/jml/get/eq2/character/?name.first_lower=^blah&c:limit=10&<span style="color: #ff0000;">c:sort=name.first_lower:1

 

Using service ID within img requests

You can (and should) now include your service ID within requests for images.  For some strange reason, we didn't include them before.

Example: census.daybreakgames.com/s:foo/img/eq2/character/2168961825756/paperdoll

This will allow us to track usage by service and provide better support.

 

New data: account.link_id

The game client now has a checkbox for allowing one to make a character visible as an alt.  When checked, that character will have an extra field, link_id, within the account information.

<character displayname="Ooof (Freeport)" id="2168960699649">    <account age="1634" link_id="71350869"/></character>

If you then perform a search query with that value, you will be able to see all of the characters that are linked together.  This query will look like this...

census.daybreakgames.com/xml/get/eq2/character/?account.link_id=71350869&c:show=displayname,account&c:limit=10

This example uses my account, so you now see how little time I have to actually play the game.

 

Fix for c:explain=1 when displaying in XML

A small fix has been deployed to allow formating to work properly.  The top-level element name of the output is now <explanation>.  There have been no functionality or data changes.

 

<span style="font-family: verdana, arial, helvetica, sans-serif;">New c:resolve<span style="font-family: verdana, arial, helvetica, sans-serif;"> for dungeon_items

c:resolve=dungeon_items

You can also limit what is resolved by...

c:resolve=dungeon_items(displayname)

This resolve is also listed in the character collection information...

census.daybreakgames.com/xml/get/eq2/

...which returns...

Code:

<item name="character">* * 
<resolve_list>* * * * 
<resolve>spells</resolve>* * * * 
<resolve>factions</resolve>* * * * 
<resolve>appearanceslots</resolve>* * * * 
<resolve>equipmentslots</resolve>* * * * 
<resolve>achievements</resolve>* * * * 
<resolve>warders</resolve>* * * * 
<resolve>statistics</resolve>* * * **
[b]<resolve>dungeon_items</resolve>[/b]
</resolve_list>
</item>

 

Resolves can now include fields to hide using !

Previously, you could list what fields were to be included in a c:resolve by putting the field names into a comma delimited list in parentheses...

census.daybreakgames.com/xml/get/eq2/character/863289283327?c:show=spell_list&c:resolve=spells(name,classes)

This is the equivilent to the c:show command on the items that are resolved.  You may now use a ! character to indicate that this is a c:hide...

census.daybreakgames.com/xml/get/eq2/character/863289283327?c:show=spell_list&c:resolve=spells(!icon,cost,duration)

As with normal c:show and c:hide commands, you can use one or the other.  You can't mix them.

 

New eq2 collection: world

This collection provides information about each world (or "server") being used for EQ2.

Example: census.daybreakgames.com/jml/get/eq2/world/?c:limit=20

Here is a sample of the data provided (in jml):

{    "id": 100,    "language": "English",    "last_update": 1331089802,    "name": "Test",    "name_lower": "test",    "soemon": "games/eq2/Test"}

id - this is the value used in various places in the data feeds, including item._extended.discovered

language - the language (in addition to "English") supported on the server

last_update - the timestamp of when this was provided...usually this is when the server was started

name - the name of the world

name_lower - the lowercase form of the name

soemon (soon to be renamed) - this is used for our internal monitoring system

 

New Query Command: c:distinct
You may now get the distinct values of a particular field in the data using this command:

census.daybreakgames.com/jml/get/eq2/item/?c:distinct=type

This is very useful for understanding the scope of the data that is provided through the API. *

<span style="color: #ff0000;">IMPORTANT NOTE: This can be a very expensive command since it must iterate over the entire data set looking for the values. * As such, it is impractical to use this on very large collections, such as character.

If you are going to use this on a larger collection, you probably want to use a field that has an index.

The output of this command will be a list of the values.

In json (and jml), this will be simple array:

[* * "Spell Scroll",* * "Weapon",* * "Item",* * "Armor",* * "Shield",* * "Recipe Book",* * "Drink",* * "Food",* * "Container",* * "Decoration",* * "House Item",* * "Book",* * "Ammo"]

In xml, the results will look like:

Code:

<type_list>
<type>Spell Scroll</type>
<type>Weapon</type>
<type>Item</type>
<type>Armor</type>
<type>Shield</type>
<type>Recipe Book</type>
<type>Drink</type>
<type>Food</type>
<type>Container</type>
<type>Decoration</type>
<type>House Item</type>
<type>Book</type>
<type>Ammo</type>
</type_list>

For completeness, here is what you get with yaml:

[Spell Scroll, Weapon, Item, Armor, Shield, Recipe Book, Drink, Food, Container, Decoration, House Item, Book, Ammo]

 

New c:resolve for eq2/world

When you query on this collection, you can resolve for status

census.daybreakgames.com/jml/get/eq2/world/?c:limit=20&c:resolve=status

Alternatively, you can also use c:resolve=all (which is equivalent).

census.daybreakgames.com/jml/get/eq2/world/?c:limit=20&c:resolve=all

This will resolve into the world the same data that would have been retreived using the status command:

census.daybreakgames.com/jml/status/eq2

...or (for a specific world)...

census.daybreakgames.com/jml/status/eq2/Splitpaw

 

Big Numbers

JavaScript has a known issue with very large numbers.  It can only handle exact whole numbers up to about 9000 million million (that's 9 with 15 zeros). Higher than that and you get garbage as the storage of the number overflows.

You can see more information about this by asking the google, including this link. This is really only a problem with the numbers that are used to identify a globally unique ID (a GUID). If you load a GUID into a variable in JavaScript, you may run into this limitation and cause a lookup to fail. The data API stores all data provided to it in its native value (without this limitation), but this may become problematic when processing that data.

To make life more sane for the consumers of the API, values above (2^53-1) into a string when providing results. Internally, we are working together to see if we can minimize the likelihood that this will occur, but at least we are protected when it does happen.

 

New JML enhancements

We are continuing to improve the JML ("JSON as HTML") renderer. This is intended to be used when browsing the API interactively in order to get a better understanding of the data.

• resolve links are highlighted with a green glow and can be clicked to open the related object
• attachment URLs are also highlighted to view that data (opened in a new window)
• all simple data is displayed first so that they appear at the top of the data
• numeric values are rendered in green while strings are in black
• particular fields (i.e. "id", "name", "displayname") are bolded so that they jump out more
• hovering over a timestamp* will reveal the time in a human readable form in a tooltip
• collections may be collapsed by clicking on name or the opening (or closing) brace 
• collapsed collections are highlighted in pale yellow to indicate hidden data
• proper styling for various browsers, including iOS (although IE has fewer shiny parts)

* the tooltip is not attached to actual timestamp fields, but rather numeric values that are likely to be a timestamp

Data in JML is returned as text/html, but the content is valid JSON. You may copy the text from your browser directly into other code or parsers directly - even if data is collapsed.

You can see these features by browsing a character that has been fully resolved, such as...

census.daybreakgames.com/get/eq2/character/863289283327?c:resolve=all&c:attachments=all

 

jml as default format

Up until now, the pattern for requests followed the pattern:

[s:serviceID]/{format}/{verb}/{game}/{collection}/[{identifier}][?{queryString}]

The format part of the URI is now optional.  If you don't specify one, the jml format will be used.

This makes the new pattern slightly different:

[s:serviceID]/[format]/{verb}/{game}/{collection}/[{identifier}][?{queryString}]

Obviously, this is completely backward compatible.

 

Increased resolution of last_update for paging

The last_update field has changed from an integer (seconds since epoch) to a float in order to increase its resolution.  Each will be (effectively) unique so to allow this to be an effective way to page through data when bulk loading data into your own storage.

When you sort by this value - c:sort=last_update:-1 for descending or c:sort=last_update:1 for ascending - an extra attribute would be added to the response: min_last_update (or max_last_update for ascending).  When you want to get the next "page" of results, you can look for objects that are greater (or less) than this min_last_update.  The size of the page is controlled by the c:limit value.

This is a much better way to page through data than c:start, which has to walk through the data linearly to get to the start point. 

The JML format will insert a link (green glowy thang) on the max_last_update field to pull the next page of the same query.  You can use this to manually walk through the data.

Try it out at... 

    census.daybreakgames.com/get/eq2/character/?c:sort=last_update:-1

All new updates will be getting this value from now forward.  We will be going back through existing characters to add a stuff after the decimal to uniquify them over the next hour or so.  The integer part (number of seconds) will NOT be changed during this process.

 

Previously Census added a last_update to each record as it saved it. The last_update attribute had the current timestamp, specified as number of seconds since 1/1/1970. Logs and other databases use ts instead of last_update and we would like to standardize on ts instead of last_update. We ran a process that went through every record in EQ2 and copying last_update to ts. In the process of copying last_update to ts, We are also making sure the value has fractional seconds so that it is exceedingly likely to be unique across the entire record set. So current sites that are using last_update should switch to ts ASAP.Also, some sites are using the c:start feature to skip over records. This turns out to be very inefficient. If you don’t care about the order of the records, the better way is to use the ts field for paging. If a query sorts by ts and includes ts in the result, we added a min_ts or max_ts to the JSON output. This can be used for paging. For example, to page through items, start with this query:census.daybreakgames.com/json/get/eq2/gu...amp;c:limit=100This will give the first 100 guilds and the JSON will include max_ts that you can use in the next query. For example, if max_ts is 1328921641.516, the next query would be:census.daybreakgames.com/json/get/eq2/gu...=1328921641.516

Then you just keep looping until you get a result where returned<100.Also, we added indexes to guild for world, level, member, dateformed, recruiting.minlevel and one for the combination of worldid and guildid. That last one can be used to search or sort by either worldid or the combination of worldid and guildid, but not to only search by guildid.

Here's how you want to query them:
/s:eq2scene/xml/get/eq2/character/?locationdata.world=Valor&locationdata.world=Splitpw&locationdata.world=Sebilis&locationdata.world=Butcherblock &locationdata.world=agafen&locationdata.world=Guk&locationdata.world=Freeport&locationdata.world=Everfrost&locationdata.world=Unrest&locationdata.world=Oasis &locationdata.world=Antonia%20Bayle&locationdata.world=Permafrost&locationdata.world=Crushbone&locationdata.world=Storms&locationdata.world=Harla%20Dar &locationdata.world=Barren%20Sky&name.first=i/abolooc/&c:limit=30&c:show=name,locationdata,guild,level,type&c:attachments=headshot&c:sort=name.first:1

Takes 30+ seconds, but remove the sort by name and it takes 0.6 seconds

/s:eq2scene/xml/get/eq2/character/?locationdata.world=Valor&locationdata.world=Splitpaw&locationdata.world=Sebilis&locationdata.world=Butcherblock &locationdata.world=Nagafen&locationdata.world=Guk&locationdata.world=Freeport&locationdata.world=Everfrost&locationdata.world=Unrest&locationdata.world=Oasis &locationdata.world=Antonia%20Bayle&locationdata.world=Permafrost&locationdata.world=Crushbone&locationdata.world=Storms&locationdata.world=Harla%20Dar &locationdata.world=Barren%20Sky&name.first=i/abolooc/&c:limit=30&c:show=name,locationdata,guild,level,type&c:attachments=headshot

Additionally we added the c:reverse=1 modifier to Census. You can use this to reverse the sort direction only on the current web request. The main use for this is to allow you to sort by newest first, but show the results in chronological order in your browser. That way you can see the 25 newest characters, logs, etc in order from oldest to newest.

census.daybreakgames.com/get/logs/lp.web...amp;c:reverse=1This works especially well with the auto-paging. (Click on the min_ts.)

Thank you