Items: IngameID vs. API-ID

Hey folks!

first of all, thank you, Dan!

holy crap, that was a fast reply on my mail i just didnt see the login because i have a quite large monitor that dulls colours in the corner, so i didnt see the fairly dark link vs. the dark background lol (or i'm just getting old )

anyway, onto the topic

I have some questions regarding item-IDs

i'm currently writing a tool that does some analyzing based on ingame-generated logs

i noticed that item-links in there also include the ID, which i could use for queries

for example

aITEM 924780662 151774692 0:[Ice Comet IV (Master)]/a

bold ID returns exactly what i want:

census.daybreakgames.com/xml/get/eq2/item/924780662

BUT sometimes the ID in the ig-link is negative

aITEM -1388782234 -1129272499 0:[Polished Battleclock Whirliblade]/a

census.daybreakgames.com/xml/get/eq2/item/-1388782234 <= obviously returns nothing useful

so, here are the specific questions:

1. is there a pattern, i could use to transform those negative IDs into the value the api can work with?

2. what is the second # in the Itemlink? just curious

3. is there a general pattern those item-IDs are generated from? they seem to have 9-10 digits....is there a certain range of IDs like 100000000 up to 20000000000 thats reserved for them (of course not all used)?

understanding this is essential for my application, so any input on this is highly appreciated

thx for listening!

^v^ thy ^v^

 

Add 2^32 (4294967296).

If you paste a negative itemid into EQ2U, we handle it internally.

http://u.eq2wire.com/item/index/-1388782234

http://u.eq2wire.com/item/index/2906185062

Same item.

 

I'm not sure about the relationship between those values - that would best be answered by someone on the game team (like Zoltaroth).  You can look up that item by its displayname and see if anything matches.

For instance:

census.daybreakgames.com/jml/get/eq2/item/?displayname=Polished%20Battleclock%20Whirliblade

For names that have spaces in them, you can replace the space character with %20 or + (plus sign) character.

census.daybreakgames.com/jml/get/eq2/item/?displayname=Polished+Battleclock+Whirliblade

-dan

 

The second number is a CRC of the item name.  This is so that people cannot generate valid links from random numbers and see items that we don't intend to be seen yet

 

If it's a negative number then it's the signed version of the item id.  If you convert it from a signed integer to an unsigned integer, you get the item id.  That's how we handle it on EQ2U.

Dethdlr

P.S.: If you're using php, it's as simple as this:

$id = sprintf("%u",$id);

Dethdlr

 

Alrighty, thx for all the answers, this covers lot <img src="/station/images/smilies/283a16da79f3aa23fe1025c96295f04f.gif" border="0" />

my app isnt webbased, its a windows application

about 4-5 months ago i started learning c# with the goal to write an app that helps me with my gaming, managing allts & stuff (was tired of all the excel sheets ^^)

not going too much into the details, dont wanna kill hte surprise before the core isnt running ^^

parts of it are a database for many many things

when this idea arised there wasnt any possibility to access item-details outside the game, so i planned to involve the community by uploading pics of items, then populate the DB, with help of OCR etc....

now with the release of the API, RIGHT IN TIME, thank god, i'm able to scrap this crapload of work, i can draw all i need directly <img src="/station/images/smilies/283a16da79f3aa23fe1025c96295f04f.gif" border="0" />

first of all, i dunno how many users will ever use the app, once i publish it, thus i cannot control or foresay the # of api-calls,if i would let the client draw the data directly

i decided to centralize that by providing a local db the user installs w his app, app only fetches updates i provide...this way i have much more control on the api-calls...essentially i only have to do them once on my own, then publish occasional updates to the user-end

this would require to get all the itemdata at least once, and this exactly is a problem i'm not sure how to handle it

i've never done api-calls before, i dont know how much queries/sec are considered as tolerable

also it seems i can only call only one item at a time w the query (or i dont understand the query commands enough yet?)

as i already mentioned the items have IDs w 9-10 digits ... i wrote a simple webclient+streamreader script which would work fine, but iterating through every single possible ID would result in roughly 2.000.000.000+ calls...this cant be feasible

especially since i noticed that about 95% of them are unused...example:

687077178-687077192 are empty

687077193 is an actual item

687077194-687077207 again empty

so:

1. anyone has an idea how to filter all the unused ones out?

2. and is it somehow possible to query more than one item w one call?

<a rel="nofollow" href="census.daybreakgames.com/xml/get/eq2/item/687077178-687077207"]census.daybreakgames.com/xml/get/eq2/ite...77178-687077207[/url] for example

3. out of curiosity, i assume making one call for a larger bulk of data is still better than making lotsoflots of single calls for smaller chunks?

thanks for your patience <img src="/station/images/smilies/3b63d1616c5dfcf29f8a7a031aaa7cad.gif" border="0" />

an excited newbie-programmer <img src="/station/images/smilies/69934afc394145350659cd7add244ca9.gif" border="0" />

 

You can query multiple items by separating them with a comma:

Code:

census.daybreakgames.com/xml/get/eq2/item/1222785792,2022917255

will return two item entries in one response.

Wanting to cache ALL items is probably not a good idea.  Entries can become stale.  And so far, the REST API seems to be geared toward being able to handle a fairly good load.

Querying the whole collection should be possible, although not recommended.  You would have to query the collection, specifying only the number of items you want returned, and using paging afterward to query the other pages.  For example, this would return the first 25 items of the collection:

Code:

census.daybreakgames.com/xml/get/eq2/item/?c:limit=25

And the next page would be:

Code:

census.daybreakgames.com/xml/get/eq2/item/?c:limit=5&c:start=26

(not sure how reliable paging is however)

 

You might want to read through this thread from a few weeks ago.  You might also want to get feedback from others who have looked at this.

My position is that the data from the API should remain the source of truth.  If you feel you need to cache the data, you may, but you should try to ensure that you update often.  If a customer sees data that is out of date or inaccurate, they will begin to lose trust in the data (and your application).  

If the primary usage of your app is to be used while online, there may not be a reason to cache the data.  Maintaining your own data structures is not a trivial task.

The API lends itself to many uses - web, desktop and mobile.  You may want to consider a hybrid approach - a native application with a web-based UI.  We did this with the SOE LaunchPad (native shell around a chromium web container).  Your choice will depend on the tools in your personal bag-o-tricks and your target audience.

There is no specific limit on queries per second yet.  We are still early and I'm working with our operations team to bulk up our infrastructure to stay ahead of the demand.  I'm watching the load pretty carefully, but the service has been designed to be used.  I'd rather spend more on our end to be able to respond to more requests than to have to require all of you to become data experts.  I'd much rather you focus your time on ways to use and present the data.

-dan

 

Also...you might get the impression that you only get one result be cause the default result is a single item.  The first Query Command is c:limit (as Lantis mentioned).  You can tell that your result was limited because the result will include a "limit" attribute.  If you add c:count=1, you will also get an attribute with the count of the objects that match your query.  Using c:count will slow down your queries as it will have to do more work.

Also as Lantis suggested, you can page through the results but this isn't very reliable since the data set is constantly in flux.  While you are paging, more characters may be changing out from underneath you. 

Another approach you might want to use is to sort by the last_update attribute.  Items are still touched quite a bit (updated in bulk)...we need to fix that for this to be more reliable.

So, to get all of the characters that have been updated since noon today (showing only the displayname attribute)...

census.daybreakgames.com/jml/get/eq2/character/?last_update=%3E1330171200&c:count=1&c:show=displayname

(I often use this site to do timestamp conversion)

Yes...I need to work on improving the documentation!

-dan

 

ty for all the answers

yes, i was aware of the default limit, but i wasnt sure how to set up a query-range via the ID

however, the method suggested by lantis is the much better approach, as it doesnt return any empty IDs, i therefore save a LOT of queries...to be honest, this was basically a problem of understanding the query-command syntax and its options

so in this respect, yes - i might have come up with that myself, if there would be a better docu <img src="/station/images/smilies/3b63d1616c5dfcf29f8a7a031aaa7cad.gif" border="0" /> i'm sure this will happen in the future, meanwhile i'm happy the way questions are answered and help is offered <img src="/station/images/smilies/283a16da79f3aa23fe1025c96295f04f.gif" border="0" />

back to my program -

i havent made a final decision so far, whether i take the local-db or the live-request

thing is, aside from the basics in c#, the primary target of my learning focus past few weeks has been the DB and its access

as i said, when i came up with the idea, there was no API, i was forced to use local DBs and now ive already set up the complete structure the way i want it, i've already build the entity-framework plus most of the LINQ-statements for the queries...i cant simply say i scrap that and move on (not yet at least) =)

but thats a minor reason - i want parts of my app to be avail while oflline also, API now opens up a lot new ideas for me...there will be enough queries, which, i think, are better handled live (they wouldnt make sense offline anyways) and implementing them should be easy

regarding the DB and your concerns, yes i understand you and i will run some tests until i make my final decision

keeping the Db up to date isnt that hard, i could run serverside script that check for updated items every second, if i want that...although i think 2-3 a week should be sufficient for my needs, plus i could also do manual queries whenever i feel its approbiate, but thats not yet of concern now

btw, thanks for the link on the timestamp converter, this would have been my next question ^^

i have two questions left, however

1. you said the query currently isnt limited

yesterday i quickly wrote a small script to try lantis' suggestion,  i played around a bit with small c:limit=10 + many loops (setting c:start + 10 after every loop, thus basically crawling through the whole data) vs. higher c:limit=100 + fewer loops etc ... and wachted the performance

today i wanted to try even larger bulk w

<a rel="nofollow" href="census.daybreakgames.com/xml/get/eq2/item/?c:limit=200&c:start=0"]census.daybreakgames.com/xml/get/eq2/ite...0&c:start=0[/url] (double the amount)

but always got

item_list limit="100" returned="100" seconds="0.026063"

in my output file

so, is there a cap now?

2. would it be possible to somehow retrieve the raw number of entries a collection contains or, more in general, how many hits a query would generate?

maybe use c:limit for that like:

<a rel="nofollow" href="census.daybreakgames.com/xml/get/eq2/item/?c:limit=total"]census.daybreakgames.com/xml/get/eq2/ite.../?c:limit=total[/url]

returns

Collection "items" has 192238718 entries.

or something more generic like

Your request matches 18273 values.

having an information like this would help me optimizing some parts of my code, and yes my primary target is to get the total number of items the collection holds at the time of the request

but i'm sure this could be useful elsewere, for example making a request, how many items have a new timestamp since XYZ

if matches = 0

{

scrap_daily_update();

}

thx for all the help <img src="/station/images/smilies/283a16da79f3aa23fe1025c96295f04f.gif" border="0" />

^v^ thy ^v^

 

gah, i just saw the c:count command, i somehow missed that Oo

mh, i'm not sure how to use it though

would census.daybreakgames.com/xml/get/eq2/item/?c:count=1 return exactly what i want? looking on the recent thread about count-abuse, i dont  really dare to try that *cough* ;D

 

Are you possibly looking for something like this?

census.daybreakgames.com/xml/count/eq2/item/

 

yep

i admit, i tried my link above...at first glance i thought it wasnt what i wanted, but then noticed the same

item_list count="152595"

but your is even more simple...thank you