Status Update and Community Call-to-Action Regarding Recent Increase in Lag-Related Incidents

The recent increase in gameplay-affecting lag spikes have been truly frustrating for players and the TERA staff at EME alike. We have been working with our technology partners and the game’s developers to identify possible root causes since the issues began in May. Though we’ve not yet identified a root cause, we do have some hypotheses as to what might be causing the recent issues.

To help support our investigation and build a case that we can escalate to the appropriate parties, we continue to require supporting data from players. If you experience a gameplay-disrupting lag spike, we ask that you please submit to us the following information in an email to “diag@enmasse.com”:

Internally, our Operations team is collating the data you provide and corroborating it with their own monitoring tools to get a better sense of what’s happening in the TERA server environment at the time these lag spikes occur. While our monitoring hasn't revealed any corresponding network events surrounding recent reports of lag spikes, identifying any impact on the server environment when lag spikes occur will be essential to finding the cause and ultimately arriving at a solution.

It’s important to note that you will not receive a direct response when submitting the information above. We are not using this data to address player-specific issues on a case-by-case basis. Instead, we are looking at the aggregated data to help identify the larger root cause of the recent issues.

The Product and Community teams at En Masse recognize the severity of these issues and remain diligent in monitoring community reports of them. We have established a regular meeting internally to review community reports and raise awareness around the current volume and severity of community reports involving network stability.

I know this is not the update many of you were hoping for, but I wanted to provide the community some transparency into our current efforts and encourage you to continue providing the above information to “diag@enmasse.com”. It truly is helpful to our ongoing investigation of these issues. We appreciate your continued patience and support.
«13456789

Comments

  • SeikilosSeikilos ✭✭
    edited June 20
    After all this time I have come to believe that the issue is not the servers themselves. Something in the game is causing all this to the players as most sipke/lag in dungeons, especially the velik's dungeons vsnm/hm vhnm/hm, with more emphasis in vshm where it is more painful, hence the more complaints and during events.
    More than this, a stronger pointer to this direction is the fact that players from around the world have the same issues. Different countries, ISPs, IPs, routings, companies like zayo etc. All different with one common element, the game. This might also explain the instability of the servers. Since all servers have issues (some more some less) it must be the thing that is loaded on them that's causing them to crash and not some hardware issue.

    Maybe BHS should start thinking about an optimization and the future of the game in western territories way more seriously. They should look deep in their pockets and make a decision*.


    *no sarcasm intended. they really need to think if the current situation is economically viable with more people leaving every day.
    For those that will mention the porting to console I will remind you that the surge from the release to steam + new class +may(when schools stop) lasted 2 months tops.
  • Haggard86Haggard86 ✭✭✭
    I can't help but to be disappointed, after all this time no cause is discovered.
  • SalsychouSalsychou ✭✭
    edited June 20
    Haggard86 wrote: »
    I can't help but to be disappointed, after all this time no cause is discovered.

    could be an ephemeral trigger or multiple triggers for the lag. the situation from when the lag got really bad - wintera/dys attack, savvis - is different from the stuff we're getting now - new class/afk event, zayo - especially since they've upgraded done some changes to their hardware some time in between. since the lag is persistent but in different ways, sort of, it'll be hard to pinpoint and address the true cause of the lag.

    but i'm right there with you, it's hard to stay optimistic at this point too, especially since summer is here and events will push the servers beyond their tiny limits.

    edit: dead grammar
  • Since the issues began in MAY??? it's been doing this since the server upgrade whenever the servers are busy..
  • I've written this before, but I wonder if the problems stem from some obscure hardware or software bug. A lot of stuff can go wrong with code in different environments, and NA is different from EU which is different from Korea.
  • aeee98aeee98 ✭✭✭✭
    There has been problems with NA traffic regardless of game from what I experience (I play outside of NA, but still this instability is nothing I have seen 2 years ago)

    Biggest problem I have seen is the zayo network which has recently shown me a 12-15% constant packet loss, with some cases of a 100% packet loss. This is not healthy for the infrastructure provider themselves, considering their stock prices actually dropped massively (A drop of USD$4ish) around the same period as when I experience the constant packet loss. (Source: http://www.marketwatch.com/investing/stock/zayo look for 6month/1year readings and you will understand what I mean).

    I am not too sure about actual packet loss on the server itself, but my problem on the recent days has almost always stemmed from the horrible routing network post entrance to US cables. A huge 100ms increase of ping is not healthy for any game. And networking has always been a problem for NA servers, I have almost never had any issues with EU (higher average ping but way more stable) and Asian servers (I live in asia, and while there ARE issues despite the lower ping, there has never been a constant packet loss I see with NA.

    Biggest problems are on the providers themselves rather than the end-user datasystem. Sure TERA servers have been hit before and EME is by no means have no liability for that, but they are nowhere as frequent as the bigger boggling problems that the NA internet traffic system where they have no power to provide specialized network traffic for games (low size high speed networks)

  • YamazukiYamazuki ✭✭✭✭✭
    aeee98 wrote: »
    There has been problems with NA traffic regardless of game from what I experience (I play outside of NA, but still this instability is nothing I have seen 2 years ago)

    Biggest problem I have seen is the zayo network which has recently shown me a 12-15% constant packet loss, with some cases of a 100% packet loss. This is not healthy for the infrastructure provider themselves, considering their stock prices actually dropped massively (A drop of USD$4ish) around the same period as when I experience the constant packet loss. (Source: http://www.marketwatch.com/investing/stock/zayo look for 6month/1year readings and you will understand what I mean).

    I am not too sure about actual packet loss on the server itself, but my problem on the recent days has almost always stemmed from the horrible routing network post entrance to US cables. A huge 100ms increase of ping is not healthy for any game. And networking has always been a problem for NA servers, I have almost never had any issues with EU (higher average ping but way more stable) and Asian servers (I live in asia, and while there ARE issues despite the lower ping, there has never been a constant packet loss I see with NA.

    Biggest problems are on the providers themselves rather than the end-user datasystem. Sure TERA servers have been hit before and EME is by no means have no liability for that, but they are nowhere as frequent as the bigger boggling problems that the NA internet traffic system where they have no power to provide specialized network traffic for games (low size high speed networks)

    Certain ISPs have also been shown to intentionally be creating packet losses and limiting speeds. Earlier this year one was brought forward because they were doing this to LoL and other big companies.
  • HippyKHippyK ✭✭
    Suggestion: Open Wonderholme test server again.

    If lag spikes happen on that then at least you know it is neither the hardware nor the hosting companies at fault but something to do with the server software.

    Lag spikes/disconnects so horrible right now. Unplayable.
  • LesbianViLesbianVi ✭✭✭✭✭
    Our problems escalated with HH, 30 man raid patch, our problem got even bigger with Zayo and VS patch, Our problem is wrecking us after Lakan's Fury patch.
  • CinnaMinCinnaMin ✭✭
    edited June 21
    What I find very odd about the random lag issues is that the test server, Wonderholme, didn't lag for me at all; even putting the game into the highest quality settings had me sitting comfortably at about 30FPS, even in Velika. I pushed 60 in the few areas I actually bothered to go to for leveling. I can only really play on Tempest Reach if all my graphics are at minimal graphics, minus textures. The game even ran smoothly; the smoothest I ever had it run for me in my four-or-so years of playing. I mean, I can idle comfortably at max on TR, but I certainly won't be doing anything else with my current machine! xD
  • clfarron4clfarron4 ✭✭✭
    HippyK wrote: »
    Suggestion: Open Wonderholme test server again.

    If lag spikes happen on that then at least you know it is neither the hardware nor the hosting companies at fault but something to do with the server software.

    Lag spikes/disconnects so horrible right now. Unplayable.
    CinnaMin wrote: »
    What I find very odd about the random lag issues is that the test server, Wonderholme, didn't lag for me at all; even putting the game into the highest quality settings had me sitting comfortably at about 30FPS, even in Velika. I pushed 60 in the few areas I actually bothered to go to for leveling. I can only really play on Tempest Reach if all my graphics are at minimal graphics, minus textures. The game even ran smoothly; the smoothest I ever had it run for me in my four-or-so years of playing. I mean, I can idle comfortably at max on TR, but I certainly won't be doing anything else with my current machine! xD

    I believe that WH was hosted in Seattle where the main servers are hosted in Chicago.

    ====

    This post, much like the request by Geletron in his statement last month, says to me that EME is expecting people to be prepared to do (network) diagnostics when stuff goes wrong. It's really time consuming to file support tickets/write up a lag report and most people who want to play a game wouldn't find writing up error reports fun.

    ====

    Also, the traceroute part of your diagnostics tool is pretty bad as it stops at the first node where it can't get a response and it won't go any further. This makes most of the traceroutes using the diagnostics tool from my location useless because the first node on the Internet provided by my ISP is configured so that it routes traffic to its destination, but not reply to ICMP ping packets (but it will forward them). Will you look into including a tool (such as standard Windows traceroute) that can work around this?

    ====
    the issues began in May.

    Are you serious? The community have been seriously vocal about lag issues since the beginning of the RMHM in October last year and it hasn't stopped. We're onto at least the third official thread on this since then, with previous threads getting to 20 and 38 pages respectively, as well as other smaller threads about it and I don't see it slowing down anytime soon.
  • AngelineAngeline ✭✭✭
    edited June 21
    since the issues began in May.

    Are you joking? You must be...

    Allow me to present the following:

    Exhibit A: In which Spacecats makes reference (in late April), to the lag investigations that have been ongoing "for the last couple weeks."
    e1vVQYH.png

    Conclusion: This lag has been going on for far longer than "in May," and the fact that you referenced May as a time frame for this issue highlights, more than anything, just how out of touch you all really are.
  • counterpointcounterpoint ✭✭✭✭✭
    CinnaMin wrote: »
    What I find very odd about the random lag issues is that the test server, Wonderholme, didn't lag for me at all; even putting the game into the highest quality settings had me sitting comfortably at about 30FPS, even in Velika. I pushed 60 in the few areas I actually bothered to go to for leveling. I can only really play on Tempest Reach if all my graphics are at minimal graphics, minus textures. The game even ran smoothly; the smoothest I ever had it run for me in my four-or-so years of playing. I mean, I can idle comfortably at max on TR, but I certainly won't be doing anything else with my current machine! xD

    Keep in mind this thread is about network lag, not framerate lag, which is a totally different issue. If you had higher framerates on the test server, it's probably because there were less player characters running around, and the game is very poorly optimized when handling a large amount of player characters in a given area. But this isn't related to network performance.
  • DpkFicxllDpkFicxll
    edited June 21
    did u guys check how much the nocteniuns changes affected your network traffic? and if it's not overloading your servers like is doing with the players? because, before the VSHM patch I, personaly, wasn't witnessing that tons of lag spikes
  • Haggard86Haggard86 ✭✭✭
    Massive lag spike and servers crashed just now. GG.
Sign In or Register to comment.