1. I have corrected the e-mail settings so that outgoing e-mails from these forums should be sent now. If you tried to Register or Reset your Password, please try again!
    Dismiss Notice

Census being turned off...

Discussion in 'Census: General Discussion' started by RhodrisNZ, Sep 27, 2019.

  1. RhodrisNZ

    RhodrisNZ Geographically Challenged

    • Dislike Dislike x 3
  2. Caam

    Caam Member

    The pessimist in me says census is gone for good. I hope I'm wrong.
    • Agree Agree x 2
  3. Rosyposy

    Rosyposy Active Member

    I hope so, too!
    • Agree Agree x 3
  4. Errrorr

    Errrorr Active Member

    Coming next, turning off combat logging. No more ACT etc.

    Got to keep disabling features before realising your games performance is ****
    • Agree Agree x 4
  5. Meneltel

    Meneltel Active Member

    Just another small step to total shutdown.
    • Agree Agree x 1
  6. Noctew

    Noctew New Member

    This might legit be for debugging/tuning purposes...or it might be a sneaky way to desupport Census without players going on the barricades. I hope it's the former; u.eq2wire is an invaluable resource that depends on it.
    • Agree Agree x 3
  7. Rosyposy

    Rosyposy Active Member

    EQ2U is not the only site that relies on census. Kulavvy and Dragons Armory and other public and private sites also utilize it.
    • Agree Agree x 4
  8. Feldon

    Feldon Administrator Staff Member

    I replied to the thread with this:


    When a character zones or camps from the game, the game server momentarily pauses and exports nearly all the character data to Census in one big blob. Fortunately, recipes, quest journal, and collection progress are exported in separate chunks. Still, it's a big hit on the server every time there's an export.

    When Census started out, it was a fairly limited subset of character data being exported. Then Achievements, Racial Traits, Wardrobe, Quests, Recipes, Collections, AA profiles, and Mercenaries were added. Unfortunately the merc data is not complete enough for us to use, but nonetheless what there is is exported for each character. Plus character items have gotten more complicated with the advent of Reforging, Refining, Experimenting, and Infusing.

    This 1 week shutdown is concerning to me. In Census' first 5 years, I could count the yearly downtime in numbers of hours. Over the last few years we've seen downtimes sometimes weeks long. I think there is some room for further optimization in Census which can be done, but I doubt it is fully to blame for game lag and instability. I am hopeful that the Dev Team can find non-drastic solutions which help with these problems without impacting Census users. I've spoken to far too many players who feel that the existence of EQ2U is a dealbreaker to be able to play EQ2 given its current complexity.
    Last edited: Sep 30, 2019
    • Agree Agree x 10
  9. Mermut

    Mermut Well-Known Member

    I find myself baffled that they've decided census is the cause of lag.. when some zones and fights have zero lag and for others the lag is crippling.. and I'm not comparing contested and instanced.. there are instances where the lag is always worse.
    That's a very strong hint that something about the zone art, scripts, etc is at fault, not something (like census) that is consistent from zone to zone.
    • Agree Agree x 7
    • Informative Informative x 1
  10. Meneltel

    Meneltel Active Member

    Honestly, not sure what is the cause of lag, but I really doubt that its caused by the DBG employees playing the game while working...
    • Funny Funny x 3
    • Winner Winner x 1
  11. Sweatypie

    Sweatypie Active Member

    They could prob save some money just shutting it down honestly.

    The lag is the exact same since they turned it off. So they can use this as an excuse and tell everyone they will keep investigating and do nothing until the game shuts down.

    I dont know how I could even enjoy the game without census though, but im sure the white knights at the forum will tell us how in a week.
    Last edited: Sep 30, 2019
    • Agree Agree x 2
  12. Feldon

    Feldon Administrator Staff Member

    People have been surprisingly supportive of Census for the most part. Here's a couple posts I did. Copying them here for posterity...


    Census servers (there are servers for EQ2, DCUO, and Planetside 2) running does not impact EQ2. EQ2 game servers pausing briefly to publish data to Census seems to be the concern.

    You can verify that EQ2 Census is still there. It's just not getting new data.



    This is a sketch of what I think the problem is. It's not meant to make light of or in any way attack the way that EQ2 Census works. EQ2 Census has been one of the most successful things the company has ever done.

    Folks may not remember, but the EQ2Players website broke -- a lot. This was because EQ2Players directly probed EQ2 data. Even the slightest game changes would break the site, requiring someone to constantly update and maintain the EQ2Players website to compensate for these changes. I believe that for EQ2, having a layer of abstraction between game data and Census was considered prudent, so that Census would be insulated from internal game changes. However this abstraction seems to be having some scaling issues.

    EQ2 Census was built upon the work that had been done for the EQ2 Mobile app. A server was setup with MongoDB (then, one of the most efficient, lightweight databases for huge amounts of data) and hooks were added to the EQ2 game code to publish certain data about each character. Scripts were also setup to export static data like Items, Spells, Recipes, Quests, etc. when certain servers reboot.

    When PlanetSide 2 Census was in development, the devs and playtesters ran into performance problems of their own and so MongoDB was scrapped in favor of PS2 census being a direct replication of PS2 Game Servers. This copy (basically an instant live backup) happens incredibly fast at a very low level that cannot impact game performance. The game doesn't even know it's happening.

    There may be other optimizations that can be made. It may be that everyone gets opted out of Census and people have to opt back in. It may be possible to rewrite EQ2 Census to run on EnterpriseDB. I'd like to be involved in the conversation, but I'm not sure that's going to happen.
    • Like Like x 3
    • Informative Informative x 3
  13. Tkia

    Tkia Active Member

    Good backup plan ;)
  14. Fuli

    Fuli Well-Known Member

    I'm skeptical of the lag reasoning.

    However, the way Feldon describes the problem, it may simply boil down it being increasingly costly on all kinds of resources - maybe that's where the lag bit is coming from. Dunno.

    I doubt this is a move to shut census down. I'm sure Longdale is not pleased people can easily infer population data from it, but, she knows the end of census means the end of eq2 revenue.
    Last edited: Oct 1, 2019
    • Agree Agree x 4
  15. Mermut

    Mermut Well-Known Member

    I'm still baffled why they are so obsessive about hiding eq2 numbers and ONLY eq2 numbers.
    • Agree Agree x 2
    • Appreciation Appreciation x 1
  16. Alarra

    Alarra Active Member

    I have no doubt that it is possible to rewrite it to run on EnterpriseDB (Postgres?), however, would they do it.
    I recall they shifted to EnterpriseDB due to the skyhigh licensing costs of Oracle, that had the thing that motivates them the most.....money.

    I'm sure that the Dragon Armory guy would also like to be involved in that conversation.

    The way I saw census updates was that whenever a character logged out properly the data was updated. And yet before a game shutdown I do not perceive much performance issue in the game. I guess it is looking at all the optimization that can be done.
    Or my understanding of the process is misinformed.
    Maybe when people zone it also tries to send Census data as well, making the load times longer??

    I another possible reason is that they may be looking to change the census interface for new features and do some testing.

    The lag I would say would be nested loops and stuff in the spaghetti code, also known as SpLaghetti Code.
    • Like Like x 1
  17. Errrorr

    Errrorr Active Member

    Census updates character data whenever you zone.

    It's a strange one, you'd expect census data being exported 5 years ago to be far more substantial than now (More population), but on the otherhand I'd guess we now export a lot more data per character...

    Perhaps it's a sign that DBG should stop adding bullshit features onto characters and implementing more and more systems on top of each other.

    I'd be amazed if Census ever gets looked at beyond turn it off/on.
    • Agree Agree x 4
    • Like Like x 1
  18. Cindrax

    Cindrax Active Member

    My prediction: census will be turned on again after a month or two (before testing of course) and the lag will be even worse due to the changes done.

    Edit: Yes, I have zero faith in DBG. The only thing they are good at is milking the few customers they have.

    Edit2: The art team is an exception though, they usually do a fantastic work.
    Last edited: Oct 3, 2019
    • Agree Agree x 1
  19. Alarra

    Alarra Active Member

    The art is outsourced now I believe. They just animate it and place it.

    edit: Some art is outsourced, see Feldon's post below mine.
    Last edited: Oct 3, 2019
    • Informative Informative x 2
    • Shocked Shocked x 1
  20. Feldon

    Feldon Administrator Staff Member

    Enemy/monster models have been outsourced for 3+ years now.
    Environments for EQ and EQ2 are done by a combined team led by Tim Heydelaar.
    Aaron (Gnobrin) still does character armor and weapons.
    • Informative Informative x 1
    • Appreciation Appreciation x 1

Share This Page