1. I have corrected the e-mail settings so that outgoing e-mails from these forums should be sent now. If you tried to Register or Reset your Password, please try again!
    Dismiss Notice

Recommended method to update players data by increment

Discussion in 'Census: EverQuest II' started by Olee, Oct 17, 2017.

  1. Olee

    Olee New Member


    I'm looking for the recommended method to call the Census API to get data of characters that have been recently updated.

    Also when my project starts I'll need to request huge list of data at once: list of all items, achievements, quests,...

    I made some search but was not able to find a faq that explain the best/recommended method to do that. Even though it seems a basic functionality for anyone maintaining their own census database.

    In the case of characters I could call at regular interval a search to get the list of all characters IDs, based on their "ts" or "last_update" field, with a value superior to a specific timestamp. And extract only the information I need.

    I could also handle the limitation in the number of rows returned by sorting the results by the character "id" and doing additional search with the "id" superior at the last value I obtained in the previous search, in a loop to extract the full list if needed.

    Is there a better approach than this ? Should I apply the same method for items, quests... lists, for the first time I load the data ?

    May be there are some recommendations on the fields to use, or to not use, for better performance ? (id / ts / last_update ...)

    What is the recommended minimum time interval to call the Census API (every 1s, 30s, 1min, 5min...) ? In order to not get considered as spammer/doing a ddos attack.

    Building the request, and calling the API is not an issue so far, but I want to be sure to use the best method to do that, and respect any guidelines/recommendations that may exists or that you may have build while setting up your own services.

    Thank you in advance.
    Last edited: Oct 17, 2017
  2. Feldon

    Feldon Administrator Staff Member

    Not saying you were going to do this, but I feel obligated to warn you that it is not permitted to use Census to generate any kind of population tracking. I have every reason to believe that they will shut the service down if that is done. Even though all of Daybreak's other games have no problem with the public seeing their population numbers. Regardless of how well-intentioned my motives, I have gotten burned by that hot stove enough times not to touch it again.

    As for your general question, "ts" is the last update timestamp, so that is what you should use to find characters that have been updated since your last scrape.

    You need to use c:sort to make sure the data comes back in a meaningful way. Otherwise it is semirandom and you can miss records. I scanned through recipes without c:sort three times and I kept finding new recipes. So it's pretty much mandatory. c:sort=ts is acceptable.

    And you will need to do pages, so c:start=0&c:limit=50 then c:start=50&c:limit=50 will get you the first 50 results, the next 50 results, etc. Include &c:count=1 to make sure the number of total matching results is included in the query. You can use this to see how much more paging you need to do.
    Last edited: Oct 20, 2017
    • Informative Informative x 1
  3. Feldon

    Feldon Administrator Staff Member

    Also unless you have a very good reason for doing so, I suggest requesting data as JSON. You can use XML for viewing and debugging, but for pulling data, JSON will spare you a lot of grief.

    I strongly suggest using c:show to greatly limit the results being returned to exactly what you need. Requests without c:show are limited to 100 results per query.

    You must have a ServiceID to make more than 10 requests a minute or more than 100 results in 1 query.
    • Informative Informative x 1
  4. Inire

    Inire Not really an evil duck, just misunderstood.

    poor feldon.

    so many great ideas about a solution that actively fights him.
    • Winner Winner x 1
  5. Olee

    Olee New Member


    Thank you for all the information.

    I have made the necessary to obtain a ServiceID when I started to play with the Census. I have already tested most of the mentionned queries and coded some generic methods to access them. And I usually use Json when possible as it is lighter, more readable, and faster to parse.

    As I said initialy I wanted to be sure that the method I have proposed was fine for this purpose and if a better method was existing.

    I've no plan to publish any kind of statistics. Only to have a dedicated database to quickly access some data and generate reports for my guild. For exemple we would like to followup the spells tier of each guild member in order to provide a tool were we could enter a spell name and find who may needs it.

    Best regards
  6. Mysstie

    Mysstie Member

    One thing that I found out, is that I'll run a Get Character Info script that updates my character's basic information on my site and if I've played them all then I get all my characters at once; however, I found out that if I tried to get all 6 at once with the same request, it didn't send back everything like I hit a limit. So, now I do 6 API queries in a row, one for each character and I get everything that way.

    My latest endeavor was to get my collections. Now I can list which ones I've completed and which ones have items left. Right now I have Completed 533 collections with 293 incomplete collections. Looks like 349 I've never found.. I should probably make something that lists those.
  7. Feldon

    Feldon Administrator Staff Member

    Were you using c:show on your queries? If not, then you were possibly hitting out-of-memory somewhere. Characters are big!
    • Like Like x 1
  8. Mysstie

    Mysstie Member

    This is my request:
  9. Feldon

    Feldon Administrator Staff Member

    I'm astounded that you were not able to request 6 results with that.
    • Like Like x 1
  10. Kulavvy

    Kulavvy New Member

    I'm trying to do the same thing with ITEMS.
    I want to get items with TS>specificDate, first asking for the count:
    In response I get count="158967" items with TS > April 1st 2018.
    "last_update" fields contains the same date as "ts".

    Looks like every EQ2 hotfix resets items TSs, even if items don't change.

    So, do you know an alternate way to get only changed items?
  11. Feldon

    Feldon Administrator Staff Member

    I know of no way to get "changed" items. Nearly all items are re-exported every time Maj'Dul reboots. TS and last_update are identical for items.

    Also don't forget to do c:sort on requests where you are walking many results. Without a sort, you will get records back in an unpredictable order, leading to items being missed. If you limit yourself to specific fields, you should be able to request 2,500 items at a time, requiring about 80 pulls to reload everything.
  12. Olee

    Olee New Member

  13. Feldon

    Feldon Administrator Staff Member

    Never used c:eek:rder. Only c:sort.

Share This Page