Quick Questions about service ID

Hey there

i've been into using api since its initial launch
due to lots of other work i've put down my developement for roughly a year
now, i want to resume doing (a program around eq2 items & characters) what i wanted and came back here

i certainly noticed the service-id issue and signed up for it
i however, do have some concern id like to ask here

the design i have in mind clearly favors having stuff cached in a DB on my end
its not a problem to query the api for all items in eq2 via paging
(store them on the adminpart of my solution, then only query api for updates and distribute db-updates i neccessary)

from what ive read, using the service id does not limit you to queries per minute
since the eq2 db yields over 200k items and (at least last time i did it), you are restricted to 100 items/query, it would therefore require about 2k queries...or, looking at the throttle to 6/min, roughly 5+hrs of time to download them all (a total of around 700mb traffic = 40kb/s)

now, what would happen, when i run the import with the service-id?
the code is written asynch, means there are roughly 10 calls at one time, each taking about 3s to complete, so lets say on average of 30 calls perminute over about an hour (=~200kb/s traffic)

this query wouldnt happen very often, certainly not when im finished developing
but from time to time it may be neccessary to reset the db
and during developement time i may need to do it 1x or 2x a week

my question is, where would you draw a line here?
i could improve or throttle the calls either way, but i surely dont want to have my service-id blacklisted
on the other hand i certainly dont want to sit and wait for 5 hrs

cheers
Daniel

ps: no i dont want to consider making the client the calls on demand, id rather drop the project altogether

 

When you use a ServiceID, you are not limited to 100 results. You could query 1,000 items at a time or more. Then it would only be 172 queries. If you pause 5 seconds each one, that's 15 minutes per week of queries.

In your situation, I would keep a local database copy of all the items, with the most useful fields stored as separate columns, and the rest stored as a jsonblob which you can unpack and display as needed. That's what we do on EQ2U for fast sorting and displaying of certain values while still being able to display the whole item.

In a perfect world, you'd only query items from SOE which have been changed since the last time you pulled them. This is done with the last_update field. However the following query shows that out of ~172,000 items in Census, 141,802 were re-exported last Tuesday:
http://census.daybreakgames.com/xml/count/eq2/item/?ts=]1396828800

If nearly all items are reexported every time there is a game update, then that makes the last_update field rather useless.

On EQ2U, we do not refresh an item until someone views the Item Details page. When someone views an item in an item popup, they're seeing a cached version. Only when they investigate further by clicking on the item does it do a load from SOE. This is the best balance we've found between constantly slamming SOE's servers vs. caching inaccurate data. You might consider doing this, or possibly add a little Refresh button next to your items so that users can click on obviously outdated info and get it updated. What I wouldn't do is scrap and reload the database every week.

But that's just my 2 cents.

 

thanks feldon, for the headsup, greatly appreciated

my service id still seems to be limited, though
excited as i was, i quickly changed the code and fired a run, but it still returns only 100 items/query

http://census.daybreakgames.com/s:<myserviceidhere>/xml/get/eq2/item/?c:limit=1000&c:start=0 <- should still be the correct syntax, shouldnt it?

 

I just ran that query with our own serviceid and was able to bring back 1,000 results. How odd.

 

oh well, you have an established service running, whereas im a complete noname
i guess that includes some premium services hehe

gonna wait if some official shows up here *hope*

 

You can try data-feedback (at) soe.sony.com.

Sometimes someone on Platform, specifically Census, reads and comments on this forum, but I'm not sure as often as they'd like. I guess we count our blessings that we have an API. DCUO's Census is in some kind of half-finished limbo.

 

Hi,
I have looked at the code and have found the solution to this mystery. For eq2 only, census limits queries to 100 results unless you include a c:show parameter in your query, in which case 5000 results are allowed. This query will reutrn 5000 results:
http://census.daybreakgames.com/s:soe/get/eq2/item/?c:limit=5000&c:start=0&c:show=displayname,id
Where this one only returns 100:
http://census.daybreakgames.com/s:soe/get/eq2/item/?c:limit=5000&c:start=0
I guess the purpose of this was to encourage users to use c:show to only fetch the fields they are interested in, not the entire record.

 

thanks alot for the reply
since i dont need every field on the items, ill try narrowing that down
gotta be careful on this though
with the cenario feldon described above im stuck back at the 100 returns/query cause if such a large chunk of items being changed chances are there are completely new fields i might wanna catch
cant do that with "show" since that implies i already know them

thnk anyway for the headsup though

@feldon
so you did include a c:show on my query?
i understood you took my syntax and just inserted your id

EDIT
is there a way to get some sort of overview which fields are actually used?
as of now every item consists of dozens of elements and attributes, but none of them holds all

if there would be something as a serverside xsd this might be a solution, i just download it, compare it to the xsd before the "patch" and can easily spot any changes

 

Yes I did c:show so that's why I was able to get more than 100. So I didn't just take your query and swap the serviceid, I made changes. Sorry

And no I've never seen a list of attributes on items. I just add them as I encounter them and there are probably fields out there that I don't support. Our item_popup.php on EQ2U is 45kb and over 1,000 lines nowadays.

If it's any consolation, items have only gotten new fields twice in the last year and a half, so they don't change much.

 

no problem, man...just making sure!

the last time i did it i wrote a complete mapper (around 9000 lines Oo) to catch eventual changes
its not just having new fields, its also shifting old ones around
but this time i think i make a different approach, a recursive one that builds at runtime
should save me a huge chunk of the original wrapper, at least if i can get that done

 

*edit button missing?*

changed some connectionsettings and cyclealgorythms in the code
whole download took me 30 mins in the latest build, which is VERY satisfying

i do however still propose some sort of schema-file for items, that would be great

for now im a happy guy though

 

9,000 lines for an item parser?
Recursive?

My brain hurts already.

By the way if you want to use our item popups on your site, you just need to include a JS file and a CSS file and any item link to our site will be popified.

 

no it wasnt recursively, the way i thought back then i'd still have to include dozens of ifs and switches
this may be due to the fact, that back then i just started learning c#, and i didnt had any other conceptual vision how to handle that
but i showed up with it at stack overflow and they also didnt had any other idea
problem is, if you dont have anything to map against to, you gotta think of everything possible
its a difference if you wanna catch what MIGHT be there, instead of mapping it to something you KNOW its gonna be there
and since i wanted to make it as robust as possible (aka handling future changes), there was no way to do that recursively

today i think i break that original wrapper into two steps
on small recursive method to capture the scheme and catch any changes
and second, a much smaller wrapper that only maps the fields needed
might save me a bunch of those original 9000 lines

 

The item parser on EQ2U contains ~200 IF statements in the View that do different things depending on item type, what fields are present, etc.. The code makes a lot of assumptions based on me comparing the data vs. how items appear in-game. We also don't separate the parsing logic from the display code because this seemed like it would increase complexity by a factor of 10. We just walk through the data and see what matches and what doesn't. It still surprises me how quickly our code executes considering how many different paths it can go depending on any one item.

Trying to "handle future changes" seems like an impossible goal to me. There's no way to know how new fields added by the team will look in-game, and I am trying to accurately reproduce how things look in-game (or better in the case of Spell Weapons, Spirit stones, Reforging, etc.). The team might add a whole new type of gem or adornment slot. Until you see how it displays in game, how would a "future proof" item parser handle and display it?

Just my 2 cents and not trying to discourage or debate anyone on how they do their website. I just know I hear a lot of projects start up with EQ2 data and then they're never heard from again and it's kinda disappointing. I thought there would be all kinds of sites out there doing all kinds of amazing data analysis and gear recommendations and so forth. Instead there are just a handful of sites. I wonder if folks try to make something completely robust and futureproof and then get buried in the complexity of it.

The underlying architecture of EQ2U may be solid and neatly organized and designed to handle changes, but when you get out to the Views and parsing the data, it's very hardcoded and subject to break if SOE changes things. We focused on what we thought we could control, and then handle the rest with lots and lots of IF statements.

 

i think i suggested too much here

i didnt want to imply that my mapper is supposed to be able to handle any future changes in an automatic way, i agree this cant possibly be done
with robust i just meant catching any changes exactly where they happen, whenever they occur, and give me details as where to change code & db

i could create one simple scheme that includes all possible fields and map every item against it
but this would (imho) create a huge overhead since many items dont have all the fields, each category follows a certain pattern
so my mapper sorted all items, based on itemtype, then created a scheme for each individual type (means, leaves all fields that dont occur within that type, thus creating smaller tables for the DB)

i didnt want to write separate methods for every type though, i just copied neccessary IFs, thats why the code got that large

wasnt a big problem though, it took 1-2 min to parse all the downloaded files

sorry if i create(d) confusion here, english isnt my primary language *cough*

 

It's perfectly ok. What you said above makes more sense as far as having all fields "available" and filling them as they are present.