What EME is doing to reduce lag in TERA [Updated 5/18]

=== Edit 5/18/2017 ===

This week we applied hotfixes during the Tuesday and Thursday maintenance periods that should help with some of the recent server stability issues.

On Tuesday, the hotfix addressed server stability. Thursday's maintenance fixed an issue around server transfers.

We're continuing to monitor and to work with Bluehole to resolve lag-related issues reported by players. These hotfixes are part of that work.

Server transfers will be enabled slowly over the course of Thursday (5/18) and we expect them to be fully enabled by the end of the day, barring any unexpected issues.


=== Original Post ===

Hi everyone,

I wanted to come to the forums and provide some answers to common questions players have about lag and what can be done to reduce it.

Q: Why do you always tell us that the problem is probably caused by my ISP?
A: We consider checking your internet settings as the first step in troubleshooting, kind of like how all IT people ask you if you have tried turning it off and on again. However, although this is the first thing we ask as a troubleshooting step, and a common source of lag in TERA, we do realize not all lag-related problems are caused by ISPs and the routing to our servers.

Some lag can occur when there is disruption or delay in data transfer between your computer and our servers in our Chicago data center. Network problems range from trying to connect over a weak Wi-Fi connection to a router in Chicago is having trouble and causing data loss for players in a specific region. It can also be multiple issues at once, compounding to make for a terrible play experience so we ask that you check your network settings first as a way of eliminating even one of the potential causes of lag.

One of the tools we provide in the launcher is the EME Diagnostic tool. It runs a few utilities to get traceroute and basic hardware information. We ask players that run into lag to please run the diagnostic tool and send the resulting log file to diag@enmasse.com so we can see the results. The diagnostic file is in .txt format so if you’re curious, you can open it in Notepad to see what information is sent to us.

Q: Why aren’t you doing anything about the lag?
A: We have been trying several things to reduce lag, and we’re working to do a better job informing players what we’re doing to address the issue. In the past year we have:
• Updated the TERA server hardware (increased CPU, increased RAM, etc)
• Experimented with load balancing between two different ISPs from the TERA datacenter
• Worked with our ISPs and data center representatives to find ways to improve network performance
• Set up an experimental server to test how TERA performs in a cloud-based environment
• Set up new monitoring software to notify us when server hardware performance decreases

Bluehole is also concerned about lag reports and they are working with us to identify causes of lag and possible solutions. This is a high priority item for both Bluehole and En Masse that we actively look at nearly every day.

Q: Why do you rely on us to report lag to you? Isn’t that your job?
A: We rely on customer reports in addition to our monitoring tools to pinpoint the possible lag causes. Our operations team is notified when we see a server crash, when there is an unexpected drop in the number of concurrent players, when any server resource is over-utilized, such as CPU, RAM, Disk, network, etc., or when any game process uses too many resources or crashes. They also monitor all network connectivity into and out of the TERA data center for utilization, dropped packets, etc.

Our tools, however, do not give us metrics for the lag problems observed in the TERA client, such as rubber banding, or when it appears that a dungeon instance freezes for a brief period. We can see it happen occasionally in-game while playing, but identifying the source is more difficult than just experiencing a lag spike. That is one of the main challenges in monitoring for and troubleshooting the causes of lag.

The most useful information for us when submitting a lag report is:
• The exact time you experience a major lag spike
• What you were doing in game when you experienced a lag spike
• What character class you are playing
• What server you are playing from
• Some information on how you connect to the TERA server (what country or region in North America and wi-fi, Ethernet, and ISP).

Please submit this information along with the EME diagnostic report to the diag@enmasse.com email address. While we may not be able to immediately fix what is causing your lag, this information helps us narrow down the root causes and helps us identify potential problems.

Q: What do you do when you get a lag report?
A: We usually do the following when investigating lag:
• We try to narrow down where and when the lag spikes are occurring. TERA runs on a large network of servers (I can’t remember the exact count, but it is more than 30) so narrowing down which servers are experiencing problems is very important. People who are hanging out in Velika on TR are on a different set of servers than players from TR doing a dungeon run which is why we ask for more information on what a character is doing when the lag spikes occur.
• We look at system performance for the servers where most of the reports are coming from and check for unusual spikes in resource use. What we are looking for at this stage are correlations between server resource utilization alerts that the operations team already receives and player lag reports.
• We look at network traffic into and out of the datacenter. We are checking for unusual spikes or dips in overall network traffic across multiple ISPs, as well as looking for reports of packet loss.
• We check the databases of the affected servers to see if there is any database performance problems or bad data that could affect server performance
• We look at related platform systems that could indirectly affect server performance, such as the online store and account servers.

When a server crashes we send a notification to En Masse and Bluehole employees, we generate a crash dump from the server, and we share all the information we have with the Bluehole technical team.

Q: Why don’t we get an official response from En Masse?
A: This is an area that we can improve on. Right now, we delay talking to our players until we have more information on what caused lag or a server outage. One reason why we wait is because saying “We are investigating,” doesn’t feel like a satisfactory answer and usually only leads to more questions. Not every investigation yields a solution, and we know most players just want the issue fixed. We know that players feel we have been too silent about our efforts so far, so we will try to provide more updates to the community moving forward.

We are aware there are lag reports in the following areas in game:
1. High level dungeons
2. Spikes that affect all open world and instance players form that world
3. Average ping/jitter that are different by server, despite being hosted at the same location

Q: What can I do on my side?
A: There are some common things that can cause network performance problems that you can check:
• Make sure that you are not playing over Wi-Fi if you are having lag problems.
• Make sure that you or other people in your house are not doing bandwidth intensive things such as streaming video, downloading large files, etc.
• Try playing at different times to avoid primetime internet traffic from your ISP (usually 9 PM – 11 PM in your local time zone).

Please remember that network lag is not the same as problems with your framerate. BHS is aware of problems in this area as well, but this is not an issue related to our server infrastructure.
«13456720

Comments

  • seraphinushseraphinush ✭✭✭✭
    edited May 2017
    Btw the servers lagged for about 20 seconds just now.. hope you guys are aware o/

    ps. from AV

    Edit : It's reaching 2 minutes
  • We are aware of the problem. The operations team is either restarting AV now or will be doing so soon.
  • seraphinushseraphinush ✭✭✭✭
    Geletron wrote: »
    We are aware of the problem. The operations team is either restarting AV now or will be doing so soon.
    Thank you and appreciate it o/
  • well, thanks for sharing the info with us :)
  • TeekzTeekz ✭✭✭
    > @LHMG4YDWYJ said:
    > No other game has this issue. Its just you guys. I dont mean to sound rude but its the truth. Ive never had a server lag in BDO,Wow,Sun,Cabal and those games have 5x-100000x more players

    Agreed there are honestly only so many excuses you guys can keep throwing out.

    I know I'm my line of work my boss would not let this unproffesionalism fly.

    Fix your [filtered]
  • AkthanakosAkthanakos ✭✭✭
    edited May 2017
    Everyone on my party was also having lag spikes at the exact same time when we were doing Velik sanctuary nm.
  • Teekz wrote: »
    > @LHMG4YDWYJ said:
    > No other game has this issue. Its just you guys. I dont mean to sound rude but its the truth. Ive never had a server lag in BDO,Wow,Sun,Cabal and those games have 5x-100000x more players

    Agreed there are honestly only so many excuses you guys can keep throwing out.

    I know I'm my line of work my boss would not let this unproffesionalism fly.

    Fix your [filtered]

    Please try to stay constructive, guys. The goal of the thread is to outline the steps we're taking to prevent lag, not make excuses for not fixing it. Lag is a constant challenge for any online game, it takes time and effort to address.
  • Thanks for this thread IMO is very important that a member of the staff provides info concerning the Lag to avoid dozens of threads of angry users or frustrated ones speculating what is going on and what are you doing to fix this, so thanks for the info, it is a nice gesture from the staff that allows people to be more understanding since we get a confirmation firsthand that the problems are being treated, all patches from all games come with issues until the new content settles that is normal so take your time guys and good job.
    What I don't understand is why took you so long to talk to us since keeping the players posted about the problems we all or most of us experience is the key to have a healthy communication staff-players , lately you all are more talkative in general so it 's apreciated and the right thing to do
    and probably you won't be getting hundreds of annoying tickets asking the same thing so well done guys
    I really liked that part when you said increased CPU, increased RAM, Imo that's the best way to improve any server having lag, maybe the increase wasn't enough, I'd also restart servers daily or more often at least for other reasons and remove useless channels but you guys know more about this stuff so im confident you'll figure it out or improve it as much as possible, so again thanks for the info, you got this and have a good day
  • Spacecats those kind of posts from those 2 angry birds above are the typical toxic guys showing up all over the important threads with important info for the players destroying and getting threads closed only, they don't intend to discuss anything they lie and insult and affect negatively the forums , can't you just ban them or prevent them to post .no wonder why the staff members are not eager to share info if they read things like that everytime they post, hardware id ban please
  • Kawaguchi3Kawaguchi3 ✭✭
    edited May 2017
    Lag was an issue even before this patch was released. Doesn't take a genius to figure that releasing new dungeon would bring back more players and cause more lag. Maybe you guys should have considered stabilizing this issue first?

    It's always during hours of 12pm-7pm PST that lag gets so bad, can't even play the game
  • dreamcast87dreamcast87 ✭✭✭
    Isn't the main reason Tera lags because the game itself has never been optimized by BHS? It's not on my PC and not my ISP my ping is pretty low compared so those I asked in game 40s to 50s. I get home from work 12:30pm the pop is low and I still get lag, frame drops, freezing, desync, low fps, etc. It's the only online game I play where it's an absolute gaurantee to perform poorly.
  • KillerPenguinsKillerPenguins ✭✭✭✭
    By any chance is the load balancing you implemented dynamically switching players between carriers? i seem to be getting some pretty bad lag spikes (like someone disconnected my modem for a second) and my ping changes as if i'm being swapped from zayo to savvis between them. This makes for a MISERABLE play experience since it swaps between them during dungeons with about a 2 second disconnect while it does it.
  • By any chance is the load balancing you implemented dynamically switching players between carriers? i seem to be getting some pretty bad lag spikes (like someone disconnected my modem for a second) and my ping changes as if i'm being swapped from zayo to savvis between them. This makes for a MISERABLE play experience since it swaps between them during dungeons with about a 2 second disconnect while it does it.

    It is my understanding that the load balancing is not dynamically switching players between ISPs during a play session but I will need to double check with our operations team to make sure I'm not giving you bad information. Some of the lag from today was due to unusual spikes in CPU usage on different servers at different times. We don't know the cause of the spikes, but we are aware of them and we are trying to determine the cause.
  • KossplossKossploss ✭✭✭
    edited May 2017
    amazing :^) , still lagging , more then ever
  • Jordansb1Jordansb1 ✭✭✭
    Geletron wrote: »
    By any chance is the load balancing you implemented dynamically switching players between carriers? i seem to be getting some pretty bad lag spikes (like someone disconnected my modem for a second) and my ping changes as if i'm being swapped from zayo to savvis between them. This makes for a MISERABLE play experience since it swaps between them during dungeons with about a 2 second disconnect while it does it.

    Some of the lag from today was due to unusual spikes in CPU usage

    Umm... this is TERA... when is CPU usage not the problem? :P
«13456720
Sign In or Register to comment.