[Opinions Requested] Item Discovery Data Format

All,There are certain queries for item discovery data that don't seem to be easily supported by the current structure.  In particular, pulling back the items discovered by a given character across all servers.  I'd like to propose a format change but only after hearing from some of my fellow developers to see if my proposed change works for them as well as it does for me.  Also to make sure my request isn't going to mess up any existing process in such a way that the new format wouldn't support.Item discovery data is currently stored in this format:<discovered>  <from_104 charid="446678606263" timestamp="1298598348"/>  <from_105 charid="438087545568" timestamp="1298874823"/></discovered>This format allows you to pinpoint your queries down to a particular world since the nodes are named with the format from_worldid.  However, because of the way the nodes are named, you can't write a query that pulls back all discoveries for a given character across all servers.The other common format that has been used for data in the REST API would have the data represented something like this:<discovery_list>  <discovery worldid="104" charid="446678606263" timestamp="1298598348"/>  <discovery worldid="105" charid="438087545568" timestamp="1298874823"/></discovery_list>This method doesn't allow you to pinpoint your query or results down to a single worldid but should have the added ability of letting you list all the discoveries for a given character across all servers.In order to pull back all of a character's discoveries right now, across all servers, I would have to do 18 separate queries (if my server count is right) and then somehow combine the results into somthing I could use.  This doesn't seem very feasible.Anybody have any opinions on this?  Is the current format better?  If so let me know what I missed.  Would the proposed new format work better, worse, or about the same for you?

Also, is this even doable from a coding perspective?

Thanks

 

Depends how the backend would handle a query for a specific worldid when going with a list as you propose.

Here, I'm only querying for the last item discovered by someone on a specific server (his current one):

Code:

census.daybreakgames.com/xml/get/eq2/item/?_extended.discovered.from_".$charobj>locationdata['worldid'].".charid=".$charid."&c:limit=1&c:sort=_extended.discovered.from_104.timestamp:-1&c:show=displayname",0);

I expect the last discovered item to be somewhat reliable, but I'm not sure anything older is reliable enough to be of use, due to the numerous server merges.  Also, this query is reasonably fast, taking only a bit over 1 second (and I only send one of them when viewing a specific character, so faster performance isn't needed in my case - I am not trying to collect data for multiple characters.  So, in a scenario like mine, where you only want someone's latest disco and don't care about other servers, the current method of returning that info is probably the most efficient.  But for mass queries?  Not sure.  I suspect going with a list would make any query looking for a specific charid to be much more server-intensive.

 

My understanding, which could be completely faulty, is that with the data format change proposed above, you'd lose the ability to query discoveries from a single character on a single server, but gain the ability to query discoveries from a single character on all servers. The problem with single server is <span style="color: #ff6600;">server transfers. A character may have discovered an item on one server and then transferred to another. Unfortunately to get a list of discoveries for a character on all 18 servers currently requires 18 queries. Also, if by some miracle of the cosmos Zoltaroth can unearth the discovery information from the decommissioned servers, then we would have discoveries from many more servers. A guy can dream.

 

I wonder if this is a case where we should duplicate the data in order to support both queries.

 

Here I am writing up one of my famous wall of text messages and you go and solve the biggest part of the problem with one sentence. <img src="/station/images/smilies/3b63d1616c5dfcf29f8a7a031aaa7cad.gif" border="0" />

Yeah, that would probably do it.  And by using either c:show or c:hide, we can get whichever we want or both.

If that's the case, I'll save my wall of text for another message.  <img src="/station/images/smilies/3b63d1616c5dfcf29f8a7a031aaa7cad.gif" border="0" />

 

Just a thought: how about a discovery_list on the character collection, with a list of item IDs that were discovered by that character?

<discovery_list><span style="white-space: pre;"> <item id='123456' worldid='104'><span style="white-space: pre;"> <item id='454545' worldid='104></discovery_list>

 

<span style="font-size: 11px;">The character record doesnt contain that data.

 

We just met with the game team to work this out...here is our proposal:

Within each item, we would change the discovery data from...

<discovered>    <from_104 charid="446678606263" timestamp="1298598348"/>    <from_105 charid="438087545568" timestamp="1298874823"/></discovered>

...to...

<discovery charid="446678606263" worldid="104" timestamp="1298598348">   <world_list>       <world id="104" charid="446678606263" timestamp="1298598348"/>       <world id="105" charid="438087545568" timestamp="1298874823"/>   </world_list></discovery>

We will leave the discovered element in place for a couple weeks to allow for a transition, but this newer format should provide the information that you guys are looking for.

The information at the top level (item.discovery.charid) is the first character to discover this item, universally.

Within that is a world_list (item.discovery.world_list) which lists the characters that were the first to discover the item on each world.

Does this work?

-dan

 

I think this still has the issue of not being able to apply two criteria to a single node.

For what *I* would like to do, I think it works.  But for things like Lantis was looking to do, I think the query language still has limitations.

For example, in the new structure, how do you get a list of all the discoveries a particular character has gotten on a particular server?  Or the most recent one on a particular server?  Or even order a character's discoveries by their discovery data?

If you do this for example:

?discovery.world_list.id=104&discovery.world_list.charid=438087545568

You would pull back the example item below, even though charid 438087545568<span> did NOT discover the item on worldid 104. The way the query language works today, if it matches any of those criteria, even on different nodes, it's considered a match. 

<span>You also lose the ability to sort the results by a particular character's discovery timestamp.  In the new format, if you order by timestamp, it won't be restricted to the particular character's timestamp, right?

That's why I was hoping we could get something like this added in:

?discovery.world_list(id=104,charid=438087545568<span>)

That would say that the criteria for id and charid have to match within the same node.  But I think even that wouldn't allow you to sort by a particular character's discovery timestamp, right?

Thanks,

Dethdlr

P.S.: If both formats were to stay permanently, it might give us the best of both worlds until something better can be thought up. 

 

I'll wait until Lantis can chime in...it seems like that is a pretty big corner case.  I really want to deprecate the old information.

We don't (currently) have the abilty to define subqueries like you are suggesting.  It keeps coming up, so it may be possible in the future.

-dan

 

Makes sense. 

Right now, I can display the items discovered by a character on a single server and show the most recent discoveries first, like this and this.  When you have 1,460 discoveries, sort order kind of makes a difference. 

With the new format, how will sorting work?  Will I be able to sort by timestamp for a given character across servers?  That's the sort order that makes the most sense to the people I've talked to about it.  Newest first.  I guess if it sorts by timestamp, even if it is the newest or oldest timestamp in the list, we could live with it.  Would just seem a bit odd to see something you discovered a few minutes ago listed a few pages back because it had been discovered on another server several months ago. 

We'll adjust regardless. 

Thanks!

 

By the way, I don't think I mentioned that I LOVE the concept of having the earliest discovery in the discovery node.

Code:

<discovery charid="446678606263" worldid="104" timestamp="1298598348">

I've been hoping for this for quite a while. 

 

In my case, there are two types of information I might want in a typical roster display:

- How many items were discovered by a certain character ID (I'm not doing it at the moment due to how complicated such a query would be, multiplied by the number of characters in a guild roster, but it was requested by a user a few weeks ago)

- What was the last item discovered by someone (which I currently use in the just-released RosterMaster 5.2.5, querying only his current serverID to make things simpler).

The second one would probably require sub-queries like deathdlr suggested, otherwise it would force me to drop the last discovered report on the character view.  I also need to be able to sort them by timestamp.

The addition of the worldwide disco is definitely a very nice idea.

Unfortunately I don't really have any better suggestion at the moment, beside having both format available <img src="/smilies/9d71f0541cff0a302a0309c5079e8dee.gif" border="0" alt="SMILEY" />

The list of usage scenarios I could see people might want to use:

1) Get a list of who discovered a specific item.  This can be done in both formats.2) Get a list of items discovered by a certain charid.  Old format involves sending multiple queries if you care about discos from other servers (server transfered character).  Probably not doable with the new format (unless the query language gets extended?)3) Get only the latest item discovered by a certain charid (or the latest "x" items)

 

The second case is actually easy...

_extended.discovered.world.charid=446678606263

This would give you all of the entries that have this charid.

Scenario 3 is difficult without sub-query capability.  In the meantime, you would have to get results and sort them yourself in code (not pretty).

-dan

 

While losing the ability to easily report the latest item discovered by someone would be disappointing, it would hardly be major.  If obtaining a complete list of discos becomes easier with the new format, I could start displaying that instead (and possibly see how complicated it would be for me to sort out that list on my end instead of relying on a server query sorted by timestamp).

I guess I could pull a list of items discovered by someone, iterate through the list of items, and iterating through the list of world entries within the item to generate a temporary array containing itemid and timestamp.  After that I'd sort by timestamp, and be able to extract that most recent discovery once again.  As long the sample query you suggested still retains the timestamp info per world, it's doable.  It will mostly depend on how fast the initial query will return (my current query was taking between 1 and 2 seconds, which was very acceptable - I was using Ajax to insert it inside the already displayed page).

So, more tricky, but apparently not impossible.

 

Yeah I already got requests for Worldwide Discoveries and I said to them I had no idea if/when they would be offered. Glad I can go back and say "it's on its way".

We're walking through the data now 20-at-a-time. I feel a cache in our future.

Now to just get the disco data off decommissioned servers...

 

Sorry for double-post but didn't want this lost in the shuffle.Discos currently include the Test server.If the "first" or worldwide disco information includes the Test server, than Test will be "first" for almost every discovery, since they get access to this content weeks or even months in advance of it hitting live.EQ2Players of the past did not include Test at all. I think a lot of people are happy that Test is available in the API, but it probably makes sense to exclude Test from getting the worldwide disco on items.

 

Yeah, pulling back potentially thousands of item ids and timestamps and sorting them locally is going to be ugly.  I may just sort by WW first timestamp instead but I'll have to see how ugly that gets.

Including the test server shouldn't be a problem IMO.  First off, they get fewer worldwide firsts than you'd think.  Second, anybody looking at the data is going to have enough sense to realize tha a WW first on test that happened before the item made it to live doesn't really count.  <img src="/station/images/smilies/3b63d1616c5dfcf29f8a7a031aaa7cad.gif" border="0" />

P.S.: If anything IS done to exclude the Test server in some way from discoveries, just make sure that discovering an item on Test still makes the item visible in the feeds.  Otherwise we may end up with characters on Test wearing items that haven't been discovered yet which means they wouldn't be in the feeds.  That would be bad.  <img src="/smilies/9d71f0541cff0a302a0309c5079e8dee.gif" border="0" alt="SMILEY" />

 

So it seems that items which only exist on Test are no longer appearing in the feed. This level 91 character shows basically as naked both on EQ2Players ( http://players.everquest2.com/chara...er/429497116073 ) and EQ2U ( http://eq2.me/Test/Soresja ) because they're wearing mostly items that only exist on Test.Hopefully we can adopt what I have requested above. Allow Test to have Discovery information, but do not include Test in the calculation for Worldwide Discoveries.

 

They are *supposed* to.  Checking...

-dan