Recommended method to update players data by increment

Hi,

I'm looking for the recommended method to call the Census API to get data of characters that have been recently updated.

Also when my project starts I'll need to request huge list of data at once: list of all items, achievements, quests,...

I made some search but was not able to find a faq that explain the best/recommended method to do that. Even though it seems a basic functionality for anyone maintaining their own census database.

In the case of characters I could call at regular interval a search to get the list of all characters IDs, based on their "ts" or "last_update" field, with a value superior to a specific timestamp. And extract only the information I need.

I could also handle the limitation in the number of rows returned by sorting the results by the character "id" and doing additional search with the "id" superior at the last value I obtained in the previous search, in a loop to extract the full list if needed.

Is there a better approach than this ? Should I apply the same method for items, quests... lists, for the first time I load the data ?

May be there are some recommendations on the fields to use, or to not use, for better performance ? (id / ts / last_update ...)

What is the recommended minimum time interval to call the Census API (every 1s, 30s, 1min, 5min...) ? In order to not get considered as spammer/doing a ddos attack.

Building the request, and calling the API is not an issue so far, but I want to be sure to use the best method to do that, and respect any guidelines/recommendations that may exists or that you may have build while setting up your own services.

Thank you in advance.

 

poor feldon.

so many great ideas about a solution that actively fights him.

 

Hi,

Thank you for all the information.

I have made the necessary to obtain a ServiceID when I started to play with the Census. I have already tested most of the mentionned queries and coded some generic methods to access them. And I usually use Json when possible as it is lighter, more readable, and faster to parse.

As I said initialy I wanted to be sure that the method I have proposed was fine for this purpose and if a better method was existing.

I've no plan to publish any kind of statistics. Only to have a dedicated database to quickly access some data and generate reports for my guild. For exemple we would like to followup the spells tier of each guild member in order to provide a tool were we could enter a spell name and find who may needs it.

Best regards

 

One thing that I found out, is that I'll run a Get Character Info script that updates my character's basic information on my site and if I've played them all then I get all my characters at once; however, I found out that if I tried to get all 6 at once with the same request, it didn't send back everything like I hit a limit. So, now I do 6 API queries in a row, one for each character and I get everything that way.

My latest endeavor was to get my collections. Now I can list which ones I've completed and which ones have items left. Right now I have Completed 533 collections with 293 incomplete collections. Looks like 349 I've never found.. I should probably make something that lists those.

 

This is my request:
c:show=name,last_update,playedtime,bio,displayname,type,statistics,tradeskills,locationdata,secondarytradeskills,quests,ascension_list,stats,familiar_list,equipped_familiar&c:join=item^on:familiar_list.id^to:id^inject_at:item_detail^show:item'displayname&c:join=item^on:equipped_familiar.id^to:id^inject_at:item_detail^show:item'displayname

 

I'm trying to do the same thing with ITEMS.
I want to get items with TS>specificDate, first asking for the count:
http://census.daybreakgames.com/xml/get/eq2/item?c:count=1&ts=]1522540800&c:show=id
In response I get count="158967" items with TS > April 1st 2018.
"last_update" fields contains the same date as "ts".

Looks like every EQ2 hotfix resets items TSs, even if items don't change.

So, do you know an alternate way to get only changed items?

 

It seems something is wrong with the sort order, this query is not returning items id in a correct order on my side:

http://census.daybreakgames.com/xml...:count=1&itemlevel=110&c:show=id&c:order=id:1

I changed the query result to xml so it is easier to read