What EME is doing to reduce lag in TERA [Updated 5/18]

1246720

Comments

  • Awesome info! More of this please. Thanks for everything you guys are doing to help fix lag. Very much appreciated.
  • I found the problem...


    Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address

    15 58ms 64/ 100 = 64% 64/ 100 = 64% er1.wdc1.enmasse.com [108.59.1.170]

    16 58ms 0/ 100 = 0% 0/ 100 = 0% diag1.wdc1.enmasse.com [207.244.81.152]


    Only lost packets were at: er1.wdc1.enmasse.com[108.59.1.170]

    Sent email with full diag.txt
  • Jordansb1Jordansb1 ✭✭✭
    Stogoth wrote: »
    I found the problem...


    Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address

    15 58ms 64/ 100 = 64% 64/ 100 = 64% er1.wdc1.enmasse.com [108.59.1.170]

    16 58ms 0/ 100 = 0% 0/ 100 = 0% diag1.wdc1.enmasse.com [207.244.81.152]


    Only lost packets were at: er1.wdc1.enmasse.com[108.59.1.170]

    Sent email with full diag.txt

    I was about 12 seconds from sending something similar. er1.wdc1.enmasse.com[108.59.1.170] had about 70% packet loss for me. Nice find though! :)
  • GlacGlac ✭✭
    Geletron wrote: »
    Q: Why aren’t you doing anything about the lag?
    A: We have been trying several things to reduce lag, and we’re working to do a better job informing players what we’re doing to address the issue. In the past year we have:
    • Updated the TERA server hardware (increased CPU, increased RAM, etc)
    • Experimented with load balancing between two different ISPs from the TERA datacenter
    • Worked with our ISPs and data center representatives to find ways to improve network performance
    • Set up an experimental server to test how TERA performs in a cloud-based environment
    • Set up new monitoring software to notify us when server hardware performance decreases

    While I have very little expectations on a lag resolution (seeing that it's been an issue for such a long time, putting any faith would be absurd at this point), I am glad to see you here talking about it directly and keeping us informed. Please continue to stay active on this topic and informed as your stated EME is working to do better on. This at least helps to know the obvious is not falling on deaf ear as before it seemed like EME just didn't care as long as the cash shop was working.

    While I will encourage those around me to send or continue sending diagnostic reports, it sounds like you have a good handle on the main lag problem as a server or outlet routing issue. Most of these spikes occur for everyone at the same time across the US as we all share the experience in the same dungeon party or while in the open world. From your notes, I know you are already tracking this.

    Good luck and I REALLY hope you guys can figure this out sooner than later, but I'd put my money on eggs hatching dinosaurs instead of chickens as a better bet until proven otherwise :)
  • There doing nothing :proud:
  • Lifegiver00Lifegiver00 ✭✭✭
    edited May 2017
    Geletron wrote: »
    Avi wrote: »
    Hi GM! Another person here who is glad the issue is being investigated. The lag has been consistent for a few patches now. I am curious as to if you guys have investigated how other games are running their servers to avoid this issue? Tera has a rather small player base and I have a feeling a lot of people would be okay with downtime if the servers could be consistent. Well, mainly just instance servers. What has changed since maybe SCHMs initial patch to now? I don't think we had as much lag back then if I can remember correctly and I am doubtful that it is the increase in player base. Lag has sadly been the main reason I haven't been able to enjoy playing the game and really look forward to it getting fixed!

    We do talk to Bluehole about how TERA is hosted in other regions and members of our team have worked on large scale games from Arenanet, Microsoft, Daybreak, and other studios worldwide.

    There are almost weekly changes to TERA servers and services so pinpointing an exact change that could affect lag is challenging. One of the reasons we are asking for more information and diagnostic reports from the community is to help us both identify what areas are most prone to lag and to find better ways of identifying signs of when a server may need to be taken down for emergency maintenance.

    I have a question and I would like a genuine answer. Most of the frustration lies in most people, including myself, not understanding how the game works fully. But the lag and crashes on Tera are unlike any game I've every played. My question is why do other games that have a larger player base and more servers able to have little to no instances of lag and yet this is common in Tera? Is it lack of staff? Incompetence on EME/BHS part? Poor optimization? Not enough servers? What then? I genuinely would like to know.

    It just seems odd to me because on other games the forums never have staff asking for lag reports or having such a tremendous difficulty with their servers. They just seem to know what to do and get it done. Forgive my ignorance, but that's exactly what it is. Ignorance. I want to understand.
  • I've wondered about the root cause of the lag myself. Considering these server lag spikes happen to everyone both in open world and in dungeons at the same time. I usually leave open PingPlotter when I'm playing in order to differentiate between connection issues (jitter/packetloss) and actual server lag. Generally speaking my connection is fairly solid even when these lag spikes occur and my framerates are also solid, so I can do my best to say the issue is not on my end, especially when everyone in discord or group/global chat all says lag at the same time. It makes me feel that it's clearly an issue impacting everyone. It's likely not even a connection issue because when this lag occurs the evasive skills I spam when it happens sometimes keeps my character alive, so the packets are getting through.

    Pondering this really makes me think there is some sort of back end process or cross server operation that gets hung up waiting on something and just causes delays in game, which would be indeed hard to isolate since logging would not report any issues since the operation does eventually go through, however players actually see a 1-4 second delay in game which we generalize to "LAG." Obviously narrowing this down would actually require ongoing resource monitoring to match up with player reports. However since the lag rarely lasts more than a few seconds, without precise time codes it's really hard to match player reports up with actual server processes.

    If it would be helpful I'll try to write down the times of the the worst spikes I experience during an evening of gameplay since the issue seems to exacerbate itself during prime time. I'll try to do that, because I feel this game could be much more successful if we could improve the experience for the average player, especially ahead of launching on consoles.
  • edited May 2017
    I am happy to see this thread as it seems both useful and shows the customer base that you guys are actively working on something. I doubt it will stop the haters and venomous people but it will make the less caustic people happy. In the interest of trying to be useful, I will give my two cents.

    I have not had lag problems with TERA before this last month (I had not played the game in the year but played relatively frequently before that). I did change servers (from Celestial Hills to Mount Tyrannas though I am still actually playing on both). My computer is connected via ethernet cable in Georgia (my ISP is notoriously suck but I frequently play with people from California and Arizona who get lag at the same moments I do so I'm guessing it isn't that). I haven't been on today, yet, but I was definitely there when MT crashed and burned twice yesterday because my friend from Arizona and I were making a day of playing the game (we don't get to play together often).

    Before this and still, I have had problems with TERA flat out crashing my video card and/or computer and while, on my old potato pc, it was pretty understandable (how that thing ever ran TERA is a mystery) on this one, it makes no sense. There is plenty in the way of RAM, a relatively high end gpu + RAM to spare (relatively because it's a year old now and in terms of video cards, newest-latest-greatest has already happened more than once since I got this one - aftermarket GTX 970 in case that matters) I mention this because someone else also mentioned the optimization that was done by BHS not being ah...up to par. I am not quite tech savvy enough to know if that has anything to do with the server lag issues but if it isn't optimized on our end, likely the developer tools and whatnot for NA are not optimized terribly well either.

    The only encounters I've had with TERA customer service have been pleasant and while I mostly just read things on the forums, cheers to you guys for putting up with all the grouchy people. I'd have snapped by now.

    inb4someonecallsmeasheep
    haters gonna hate etc
  • Massive server wide lag on TR, ~ 2mins ago.
  • Sounded like EnMasse just dropped a EMP on their services. :p
  • Fix the lag, honestly, fix the lag
  • jongbaejongbae ✭✭✭
    I do understand you guys have it hard since players come from different sides of the world. I'm glad we finally have this thread tho, I was losing hope but at least I can see you guys are working on it and taking it seriously. Just wanted to throw that there <3 No more "it's your ISP" bs honestly, those kind of comments can [filtered] anyone off.
  • Hi, am I the online who hasn't have access to servors ?
  • sraz wrote: »
    Pondering this really makes me think there is some sort of back end process or cross server operation that gets hung up waiting on something and just causes delays in game, which would be indeed hard to isolate since logging would not report any issues since the operation does eventually go through, however players actually see a 1-4 second delay in game which we generalize to "LAG." Obviously narrowing this down would actually require ongoing resource monitoring to match up with player reports. However since the lag rarely lasts more than a few seconds, without precise time codes it's really hard to match player reports up with actual server processes.

    We have been looking at those possibilities as well. We've adjusted daily back up times and have been monitoring event logs on all the servers to see if there is a process running at particular times that could contribute to lag.

  • "Upgraded Servers"
    LMAO Biggest Joke ever, cant even handle all the people coming back cause of new patch.
    Its not even the routing on people because our party consist of different people from EU US and Asia and we lag/spike at the same time
Sign In or Register to comment.