Project Feedback

My EQ2 API project is to a point where I'm ready to start getting some feedback.  A little background about the project.  I started it on a mediatemple.net DV 4.0 2GB box.  (mt) techs emailed and called me after only a couple days and asked that I remove the project from the box because it was crushing the box.  I've since moved to AWS.

The web server and maintenance crons run on an AWS EC2 small instance.  The database is on a RDS small instance (pushing 20GB in size right now, time to upgrade).  I have code to automatically fire up EC2 micro boxes when I need more workers/crawlers for gathering all the new character and guild data.  All the images are thumbnailed and pushed to S3.

The site runs off CakePHP 2.1 and Twitter Bootstrap with Memcached for caching.

I have a cron/db system so that an unlimited amount of crawler boxes can start up and none of them will work on anything that the other boxes are working on.

Some of you are probably wondering why I'm not just doing API pulls as needed.  Well, there are a few reasons.  One, I want to be able to do historical data (how fast did xyz go from level 1 to 90, what was xyz wearing when he was level 10, etc.).  Two, I know Dan has worked hard to make sure the API is stable and available, but in the event that it isn't, I would like my site to survive downtime.  Three, top secret reasons I can't divuldge at this point, but expect to see some unique site features not found anywhere else.

I present to you, www.eq2scene.com

 

I'm not sure if it's an issue with the character search, or that you are using an older dataset, but I couldn't find my own character (Lantis, on AB).  It's definitely published: http://www.lostrealm.ca/eq2/rmdev/c...id=446676908718

 

I think it is a very good start, Lokiracer!

My initial feedback is primarily cosmetic instead of data related.  The images that you are using for your home page are being scaled up from 183x104 px to 312x177 px.  This is causing them to look very blurry and it detracts from the rest of the site, which is very clean and smart looking.  You probably ought to create your art at a larger size or design your layout to use smaller images.

A similar problem is happening with the character headshots.  You are presenting them from 96x166 px to 52x52 px.  Zooming down to a smaller size is easier and (usually) doesn't introduce visual anomalies.  However, you should either maintain the aspect ratio of the image.  

Your search box appears to want a character ID.  If you don't put an ID in it, it throws a 500 error.  You should probably handle that a bit cleaner.

census.daybreakgames.com/json/get/eq2/character?name.first_lower=NAME&c:limit=50

Be sure to replace NAME with what comes from the search box, in lower case.  You can convert it using strtolower($value)PHP function.

This will return up to 50 matches of this name to allow a customer to choose from.  If you want to search on partial names, you can use the "begins with" syntax (name.first_lower=^NAME).  You can also use a regular expression to use a "contains", but this may be a little slower.

Of course, more information about this sort of thing can be found in the "Developers" forum, which is accessible by Becoming a Developer.

As I said, this is a good first effort.  Definitely keep going!

-dan

 

I am concerned with your historical data.  If a character disapears because they become hidden from the feeds do you delete all records of that character from your DB?

 

Thanks for the feedback. Lantis, my live pulls aren't working right now. Working on some storage/db issues. Zoltaroth, I figured this would be a sensitive subject. Ultimately, right now, I don't have any deleting going on. Realistically, it shouldn't be an issue since if a character is ever put on the net, it will be available indefinitely via Google cache or the-way-back-machine or other services. The real reason I don't have deletions setup is because it breaks data. Discoveries made by characters that are deleted, stats like fastest character to xyz, etc. all get hosed when a character gets deleted. I'd be willing to compromise and say that I won't display character sheets for "deleted" characters, but deleting them completely? Talk to Google first, they're the real problem with that idea.

 

I can understand people might become upset at seeing their data still exposed after they decided to opt out of sharing it.

Personally, I think that marking them as private on your end and not displaying them would be an acceptable compromise.

You can avoid Google's caching by using a meta-tag BTW.

 

I don't want to display their data.  I just want to not have holes all over the place when people go deleting their toons, canceling their accounts, getting banned, etc.

I personally don't understand the reason for having opt out.  Maybe because you don't want people knowing how long you've been playing?

Sure, you can avoid caching by adding a meta tag, but that would require 100% of all API users to use a meta-tag to stop the caching of player data.  Point being, unless it's a policy of using the API, and 100% enforced, stopping data of characters that are "removed" from continuing to exist is going to be hard.

 

I'm not going to wade into the privacy debate, only the technical aspects of this discussion...

EQ2 characters, guilds, and items are not discoverable by Google on either EQ2Players or EQ2U. Both sites require searching to display this information, thus, there's no way for Google to "spider" or cache the data on these sites.

EQ2Scene and EQ2Mission, in contrast, present lists of characters and items to guests, and thus these will end up in Google's cache.

Being able to browse characters and items is feature I've considered for EQ2U, and this discussion adds a new wrinkle to the consideration of that feature.

 

I'm not trolling.  I just don't read the forums much outside of the developer forums and don't watch chat when in the game.

What are the "privacy" concerns?

 

Basically, when the feeds first went live, so much data was exposed that some people flipped out as though you had just set up a webcam in their shower.  They didn't want people to be able to see any more information than you could get from the in-game examine window.  I guess they didn't want people to know that they were level 90 with 50 AA points, 0 adornments on their items and all Apprentice spells.    So they put in place the in-game opt-out option for those that don't want their information to be exposed.

 

Ah.  Now I follow.  I read "privacy" as webcam in the shower, access to email and bank accounts.

 

There are usually numerous arguments behind those requests for privacy.  Some are valid, others are bogus.  To list a few (with no specific order):

- On PvP servers, to avoid being "spotted" as a potential target- Some people not wanting to be judged based on things visible on their profile (gear, AAs, etc...)- Some endgame raider feel that they are in competition with other guilds, and don't want people from those guilds to be able to "steal" their AA setup, gear setup, etc...- Some people are being stalked by former guildies/girlfriends/boyfriends/etc... and don't want to register on the radar.- Some people have done some dubious things (or, have been acused of), and after ending up switching server and name, fear they might be recognized by people looking at their birthdate, gear, etc...

As I said, some of these are probably valid, others are probably bogus - I am not saying I support or reject these.  As a developer, I would obviously love to have access to EVERYTHING, which would open the door to even richer projects.  But I can understand why some people might get emotional over sharing data.

Just warning you that you might get some pretty emotional angry mail from some people feeling you have invaded their privacy (I know Feldon did <img src="/smilies/9d71f0541cff0a302a0309c5079e8dee.gif" border="0" alt="SMILEY" /> )

 

Fair enough.  Those are probably all valid reasons to not want a public toon/profile.

After hearing these, I'd be game to suggest that SOE update their ToS for the API to require that users of the API stop displaying character information when an account is made private.  I think it's important to note that we should still be able to display first, last, prefix, suffix (basically what I call the character full name) in relationship to historical data.

I plan on adding a "Delete this Character" button on my site.  I just haven't figured out how to validate the user of the request as the owner of the character.

 

There is no way to verify that someone is who they say they are. We had one idea, but the effort involved for both us and users did not justify implementation.

 

Ya, the only thing I came up with was like Google's webmaster verification.

Drop a md5 hash in your EQ2 character bio for five minutes.  Go to my site, tell me to delete your character, I pull your bio, confirm the hash, then delete it.

 

I do think that due diligance would be to set a "lifespan" on the data you are storing. When that life span expires, check the feed and see if that player is still allowing thier feed to be published. If not, then remove the data from your storage. This could be handled by a cron job if you have access to that. Or another way is to do psuedo-cron jobs like Word Press does.

From the perspective of a player, it would upset me to know that there are websites out there that are retaining my information against my will, and that it requires action from me for it to be removed. It is impossible to know of all third party websites that require action to opt out.

In the end, the EQ2 game user interface should be the ultimate source of personal settings, and all products bellow it should respect that.

 

I've seen the posts about privacy of toon information, and I agree that I really wouldn't want all of the toon information *I* might want to see personally available for anyone.  Also, just because I may want my data to be available via EQ2U (for example, because they let me control what others see) it doesn't mean that I would want ANYONE accessing the same data.

I also get that there's no way to determine that a person is who they say, and that legal agreements would be necessary for SOE to trust a web site with secured user data, but there is another way (and if this is a repeat that I simply missed, I apologize):

<span style="text-decoration: underline;"><span style="color: #0000ff;">passkeys

<span style="font-family: arial, helvetica, sans-serif; color: #0000ff; font-size: small;">SOE could permit the user to generate pass keys (and revoke pass keys) that the USER provides to the web site when they register.  With a pass key, SOE and the web site have no legal agreement....all terms are with the user.  If I give the keys to my house to the wrong person, I could get robbed, but the lock maker isn't liable.  Also, if I realize I've given my key to the wrong person, I can "change the lock", making their key useless.

<span style="font-family: arial, helvetica, sans-serif; color: #0000ff; font-size: small;">This would require SOE to modify the 'share' to be a 'share public' or 'share private' setting (instead of just on or off) AND provide an interface to permit users to generate passkeys that they can distribute with the option to cancel.  It doesn't really matter if it's done in-game or via account management.

BTW: I know that it doesn't prevent people caching old data...but hey....it's OLD data.  People that NEVER want their data showing up just turn sharing off....those who want the benefit and are willing to wear the chance that their old data may linger (like an old facebook post you regret) opt in, but private....and others who don't mind/care just go full public share.  Responsibility with the user isn't a horrible thing imho.

IF a player wants their old data deleted....maybe they only give a key to those sites that have a "historic delete" privacy policy...and avoid those who don't?

Okay, I'm sure there are holes and reasons the idea is 'stupid', but I wanted to get it out there...so let the flaming commence. *lol*