=== Edit 5/18/2017 ===
This week we applied hotfixes during the Tuesday and Thursday maintenance periods that should help with some of the recent server stability issues.
On Tuesday, the hotfix addressed server stability. Thursday's maintenance fixed an issue around server transfers.
We're continuing to monitor and to work with Bluehole to resolve lag-related issues reported by players. These hotfixes are part of that work.
Server transfers will be enabled slowly over the course of Thursday (5/18) and we expect them to be fully enabled by the end of the day, barring any unexpected issues.
=== Original Post ===
Hi everyone,
I wanted to come to the forums and provide some answers to common questions players have about lag and what can be done to reduce it.
Q: Why do you always tell us that the problem is probably caused by my ISP?
A: We consider checking your internet settings as the first step in troubleshooting, kind of like how all IT people ask you if you have tried turning it off and on again. However, although this is the first thing we ask as a troubleshooting step, and a common source of lag in TERA, we do realize not all lag-related problems are caused by ISPs and the routing to our servers.
Some lag can occur when there is disruption or delay in data transfer between your computer and our servers in our Chicago data center. Network problems range from trying to connect over a weak Wi-Fi connection to a router in Chicago is having trouble and causing data loss for players in a specific region. It can also be multiple issues at once, compounding to make for a terrible play experience so we ask that you check your network settings first as a way of eliminating even one of the potential causes of lag.
One of the tools we provide in the launcher is the EME Diagnostic tool. It runs a few utilities to get traceroute and basic hardware information. We ask players that run into lag to please run the diagnostic tool and send the resulting log file to diag@enmasse.com so we can see the results. The diagnostic file is in .txt format so if you’re curious, you can open it in Notepad to see what information is sent to us.
Q: Why aren’t you doing anything about the lag?
A: We have been trying several things to reduce lag, and we’re working to do a better job informing players what we’re doing to address the issue. In the past year we have:
• Updated the TERA server hardware (increased CPU, increased RAM, etc)
• Experimented with load balancing between two different ISPs from the TERA datacenter
• Worked with our ISPs and data center representatives to find ways to improve network performance
• Set up an experimental server to test how TERA performs in a cloud-based environment
• Set up new monitoring software to notify us when server hardware performance decreases
Bluehole is also concerned about lag reports and they are working with us to identify causes of lag and possible solutions. This is a high priority item for both Bluehole and En Masse that we actively look at nearly every day.
Q: Why do you rely on us to report lag to you? Isn’t that your job?
A: We rely on customer reports in addition to our monitoring tools to pinpoint the possible lag causes. Our operations team is notified when we see a server crash, when there is an unexpected drop in the number of concurrent players, when any server resource is over-utilized, such as CPU, RAM, Disk, network, etc., or when any game process uses too many resources or crashes. They also monitor all network connectivity into and out of the TERA data center for utilization, dropped packets, etc.
Our tools, however, do not give us metrics for the lag problems observed in the TERA client, such as rubber banding, or when it appears that a dungeon instance freezes for a brief period. We can see it happen occasionally in-game while playing, but identifying the source is more difficult than just experiencing a lag spike. That is one of the main challenges in monitoring for and troubleshooting the causes of lag.
The most useful information for us when submitting a lag report is:
• The exact time you experience a major lag spike
• What you were doing in game when you experienced a lag spike
• What character class you are playing
• What server you are playing from
• Some information on how you connect to the TERA server (what country or region in North America and wi-fi, Ethernet, and ISP).
Please submit this information along with the EME diagnostic report to the diag@enmasse.com email address. While we may not be able to immediately fix what is causing your lag, this information helps us narrow down the root causes and helps us identify potential problems.
Q: What do you do when you get a lag report?
A: We usually do the following when investigating lag:
• We try to narrow down where and when the lag spikes are occurring. TERA runs on a large network of servers (I can’t remember the exact count, but it is more than 30) so narrowing down which servers are experiencing problems is very important. People who are hanging out in Velika on TR are on a different set of servers than players from TR doing a dungeon run which is why we ask for more information on what a character is doing when the lag spikes occur.
• We look at system performance for the servers where most of the reports are coming from and check for unusual spikes in resource use. What we are looking for at this stage are correlations between server resource utilization alerts that the operations team already receives and player lag reports.
• We look at network traffic into and out of the datacenter. We are checking for unusual spikes or dips in overall network traffic across multiple ISPs, as well as looking for reports of packet loss.
• We check the databases of the affected servers to see if there is any database performance problems or bad data that could affect server performance
• We look at related platform systems that could indirectly affect server performance, such as the online store and account servers.
When a server crashes we send a notification to En Masse and Bluehole employees, we generate a crash dump from the server, and we share all the information we have with the Bluehole technical team.
Q: Why don’t we get an official response from En Masse?
A: This is an area that we can improve on. Right now, we delay talking to our players until we have more information on what caused lag or a server outage. One reason why we wait is because saying “We are investigating,” doesn’t feel like a satisfactory answer and usually only leads to more questions. Not every investigation yields a solution, and we know most players just want the issue fixed. We know that players feel we have been too silent about our efforts so far, so we will try to provide more updates to the community moving forward.
We are aware there are lag reports in the following areas in game:
1. High level dungeons
2. Spikes that affect all open world and instance players form that world
3. Average ping/jitter that are different by server, despite being hosted at the same location
Q: What can I do on my side?
A: There are some common things that can cause network performance problems that you can check:
• Make sure that you are not playing over Wi-Fi if you are having lag problems.
• Make sure that you or other people in your house are not doing bandwidth intensive things such as streaming video, downloading large files, etc.
• Try playing at different times to avoid primetime internet traffic from your ISP (usually 9 PM – 11 PM in your local time zone).
Please remember that network lag is not the same as problems with your framerate. BHS is aware of problems in this area as well, but this is not an issue related to our server infrastructure.
26
