View Issue Details

IDProjectCategoryView StatusLast Update
0007892Multi Theft Auto : San AndreasServerpublic2013-12-02 00:28
ReporterccwAssigned Toccw 
PrioritynormalSeverityminorReproducibilityhave not tried
Status resolvedResolutionfixed 
PlatformWindowsOS2008 serverOS Version
Product Version1.3.4 
Target Version1.3.5Fixed in Version1.3.5 
Summary0007892: Windows server high CPU usage
Description

User has reported high CPU usage when over 180 players connected, even though performance browser is showing MTA is 75% idle.

Is there a fix by fiddling with some Windows network setting, or installing a patch?

TagsNo tags attached.

Activities

Sasuke*

2013-11-05 05:12

viewer   ~~0019605

http://i.imgur.com/KvSQZLC.png
Image of the Task Manager showing CPU/Ram usage with 160 Players connected

arranTuna

2013-11-05 11:53

manager   ~~0019606

Nothing I've ever had to do.

Screen shot of server info tab?

Sasuke*

2013-11-05 19:49

viewer   ~~0019607

http://i.imgur.com/2AhCHtV.png
This one ? or this one
http://i.imgur.com/kWDlNLm.png
The CPU i think its pretty good if that CPU cant handle for example 400 players then what kind of monster dedicated I have to get o.O

arranTuna

2013-11-05 20:25

manager   ~~0019608

No "server info" tab of performancebrowser.

Sasuke*

2013-11-05 20:47

viewer   ~~0019609

http://i.imgur.com/fOmjIdk.png
http://i.imgur.com/dG6D10G.png

arranTuna

2013-11-05 21:08

manager   ~~0019610

What CPU usage percent does MTA server.exe actually use?

I can't see any reason for it to use so much.

Sasuke*

2013-11-05 21:49

viewer   ~~0019611

Last edited: 2013-11-05 21:56

View 2 revisions

The CPU Usage with 140 Players right now as we speak
http://imgur.com/oBZYuL6,SMb5lYJ,tWjcTRh#0
MTA Server.exe 12% CPU but actually only using 1 core im guessing if 250 players connect or more that core will get to 100% and "Network Trouble" will start appearing unless MTA Server starts using other cores and load balancing..
http://imgur.com/oBZYuL6,SMb5lYJ,tWjcTRh#1
And here is one of the random network dips probably caused by OVH Network, they say no one is having that problem and they say everything is working as it should be..
http://imgur.com/oBZYuL6,SMb5lYJ,tWjcTRh#2

Should I use like an special module ? or probably is the network settings ?
http://i.imgur.com/UjziRuH.png Here are the programs installed on the dedicated server.
Network settings:
http://i.imgur.com/4PnMP5g.png
I even disabled IPV6, I think we dont need it

Dutchman101

2013-11-05 22:14

updater   ~~0019612

Please generate a dump file of the MTA server process actually running.

To do so, open Task Manager, go to the MTA Server process. Right click and press Create Dump File, save it and upload here.

Make sure the problem is going (cpu usage) while making the dump.

ccw

2013-11-05 22:39

administrator   ~~0019613

As I said to you before Sasuke, the 'Server timing' page shows MTA server as being 75% idle.

The CPU usage is probably a Windows networking issue due to something like this:
http://support.microsoft.com/kb/2685007

arranTuna

2013-11-05 22:43

manager   ~~0019614

If it was OVH dip it would be huge, I've had them. Those aren't OVH dips those are most likely a script freezing the logic thread for a bit causing it to drop and then it slightly spikes afterwards because of the catchup.

Also uploading a dump of your whole MTA server.exe exposing passwords so wouldn't recommend it and last time I asked an MTA dev if a dump of the server exe when it's frozen would help they said it doesn't.

Go to "Server timing" tab and see how much .g_pNetServer->DoPulse (); uses. If core usage is almost 100% then .g_pNetServer->DoPulse (); should be almost 100%.

Dutchman101

2013-11-05 22:53

updater   ~~0019615

@Sasuke, download the Windows server hotfix on the page ccw linked to. I think it's possible that's the cause

Sasuke*

2013-11-06 03:46

viewer   ~~0019616

Have installed the Windows Server Hotfilx that ccw provided and rebooted the server
MTA Server still allocating all the load to that CPU Core
120 Players on the server
http://i.imgur.com/KZLIQX3.png
Here is a Pic of the Server timing
http://i.imgur.com/8OF0pJP.png
The problem is not that the server is using 100% of the core. Yesterday we had 180 Players and everything was working fine but the core usage was almost 100% like 95-98.
I wonder if we get 250+ players the core will clog basically and Lag while the 70% of the Whole processor is free. Why does the MTA Server only use 1 core and not all cores/threads.

Dutchman101

2013-11-06 04:38

updater   ~~0019617

@Sasuke,

Ver: 1.3.4-9.05908.0
AC: [Allowed client files: 1] [Disabled AC: None] [Enabled SD: 12,14,15,16]
Voice: Disabled
Glitches: None
Bullet sync weapons: 22,23,24,25,26,27,28,29,30,31,32,33
Vehicle extrapolation: Disabled
Tick rates: Plr:100 Cam:500 Ped:500 UnocVeh:500 KeyRot:100 KeyJoy:100
Syncer distances: Ped:100 UnoccupiedVehicle:130

Look at these (your server). There is listed: Player sync interval being 100, while your screenshot (http://i.imgur.com/fOmjIdk.png) says it is 200.
Also UnocVeh is set to 500, but actually running 2000 (according to your screenshot of server info)

Meaning, it could be a problem with your sync, driving up the CPU usage, since these values doesn't match, there should be something wrong.
Else it just could be that your linked screenshot isn't the right one, but I find that hard to believe... And believe me, syncing problems can literally eat CPU... I've had them.

Sasuke*

2013-11-06 05:20

viewer   ~~0019618

Last edited: 2013-11-06 05:23

View 2 revisions

<bandwidth_reduction>maximum</bandwidth_reduction>

<!-- The following ???_sync_interval parameters determine the time in milliseconds between certain network packets being sent. More information on how each settings works is available here:
<!-- Player sync interval. Default: 100 -->
<player_sync_interval>100</player_sync_interval>
<!-- Lightweight (player) sync interval. Used when players are far apart. Default: 1500 -->
<lightweight_sync_interval>2000</lightweight_sync_interval>
<!-- Camera sync interval. Default: 500 -->
<camera_sync_interval>500</camera_sync_interval>
<!-- Ped sync interval. Default: 400 -->
<ped_sync_interval>500</ped_sync_interval>
<!-- Unoccupied_vehicle sync interval. Default: 400 -->
<unoccupied_vehicle_sync_interval>500</unoccupied_vehicle_sync_interval>
<!-- Keysync mouse rotation sync interval. For limiting key sync packets due to mouse movement. Default: 100 -->
<keysync_mouse_sync_interval>100</keysync_mouse_sync_interval>
<!-- Keysync analog movement sync interval. For limiting key sync packets due to joystick movement. Default: 100 -->
<keysync_analog_sync_interval>100</keysync_analog_sync_interval>

<!-- This parameter can improve the reliability of shots when using certain weapons. However, it uses more bandwidth.
Values: 0 - disabled , 1 - enabled ; default value: 0. -->
<bullet_sync>1</bullet_sync>

Yes when I installed the patch I found out that <unoccupied_vehicle_sync_interval> was on 2000 for some reason maybe a mistake..I fixed to 500 because of MTA SA Wiki
Less annoying unoccupied vehicles
unoccupied_vehicle_sync_interval 500

Also player_sync_interval I put yesterday 200 but the CPU didn't make that much of a difference.. so I put the default value 100
Soo I fixed it but is really not much of a difference I think.
lightweight_sync_interval I put 2000 but really not much difference on the CPU.. bandwidth it was using like 2% less.

What would you think are Optimal Sync values ?

ccw

2013-11-06 08:26

administrator   ~~0019619

Last edited: 2013-11-06 08:31

View 2 revisions

Why does the MTA Server only use 1 core and not all cores/threads.

It does. There is another problem with your Windows server which is causing the CPU usage.

In task manager, select 'View'->'Show Kernel Times' and show another screenshot.

EDIT:
Also, what is the value of this registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BFE\Parameters\MaxEndpointCountMult

Sasuke*

2013-11-06 09:00

viewer   ~~0019620

Last edited: 2013-11-06 09:20

View 2 revisions

Actually there is no Key named "MaxEndpointCountMult"
http://i.imgur.com/o7JRIoy.png Regedit Screenshot
Pic of Task manager with the Show Kernel Times enabled.
http://i.imgur.com/DwNaqMY.png
Actually there is like 20 users on the server so looks very little but you can see the red graph. I will upload another screenshot tomorrow with more users to see
EDIT:
Here you can see some options when I was browsing on google (www.win2008r2workstation.com) if you want to turn your server into a workstation they suggest enabling the following options:
http://i.imgur.com/OB3uyKr.png
http://i.imgur.com/0nLYnVS.png
What do you think about this?

arranTuna

2013-11-06 11:55

manager   ~~0019622

If you don't have that registry value it means you didn't follow that hot fix that ccw linked you to and you have to restart the server for it to take affect.

Though I doubt it's that because we've had way more players and not had this CPU usage problem before we added that fix. It was only an issue when attackers sent UDP packets from 5000 different spoofed IPs, that's when CPU usage went crazy.

There was a program I used, process explorer, that showed you all the threads inside a process and how much each of them used, might be worth finding that program and seeing what uses what:

http://technet.microsoft.com/en-gb/sysinternals/bb896653.aspx

There should be 2 MTA threads. One will use a lot (probably the logic one) and then another will use a fair bit too (the sync thread) but if you've got 100ms puresync then it might be that the sync thread is using more.

Sasuke*

2013-11-06 17:58

viewer   ~~0019624

MTA Server with 80 Players connected
http://i.imgur.com/PllEnpZ.png
Uses 19 Threads
The thread that uses more CPU has to do with MSVCR90.DLL and some Visual C++

arranTuna

2013-11-06 18:22

manager   ~~0019625

That looks normal, same as mine.

Sasuke*

2013-11-06 18:43

viewer   ~~0019626

Can you show us yours the CPU usage in yours looks like mine ?
High usage on 1 core and the rest average usage on some cores?

arranTuna

2013-11-06 20:15

manager   ~~0019627

Last edited: 2013-11-06 20:19

View 2 revisions

http://cit2.net/arran/cpu/usage.png

I don't know which thread is which but I'm guessing the MTA server.exe is the main (logic) thread. If that's true then the sync thread uses more and the one using less is either the raknet or the database thread. ccw will know or maybe not and performancebrowser could tell us the % of each MTA thread?

Sasuke*

2013-11-06 21:22

viewer   ~~0019628

In your case the core 3 its on full load basically.
Then probably if my server gets more players MTA Server will start using other cores I'm guessing.
Could it be the Database? im using the Default MTA Server Database you probably using other?

arranTuna

2013-11-06 21:50

manager   ~~0019629

That's not core 3, that's thread 3 which is core 2.

MTA can't just start using more cores, as each thread in a process can only use a full CPU core.

If you had some super inefficient queries and or are spamming queries there's no way the database thread would use so much. 'debugdb 2' would confirm that. Expect it to be a lot, by spam I mean like more queries per second than there are players. That would definitely be spam. I think it tells you the time it takes as well, might help.

Sasuke*

2013-11-07 05:07

viewer   ~~0019630

Last edited: 2013-11-07 05:17

View 2 revisions

Well I installed the Hot Fix again now there is the Regedit Key that ccw was talking "MaxEndpointCountMult" with a Value of 10. And still not fixed MTA Server was using 1 thread few hours ago with 150+ players
I really don't know what to do anymore ._.
Maybe the CPU is Horse shit ? Even thought I'm paying 89 Dollars for the dedicated server but it comes to like 100 Dollars adding web host of the forum.

Your CPU is E5 and Its faster I think has greater bandwidth probably Hyper threading.. maybe the CPU that my dedicated maybe doesn't have HyperThreading... ?
Any of you guys tomorow can take at least 20min and connect to my Teamspeak server so I can send the Teamviewer details and you guys can move arround the settings? Hopefully we can see where is the problem ? If you guys want if not no problem I know you guys have things to do and I respect that.

EDIT: Maybe the Windows Firewall? Forcing the system even more ?
Settings: http://i.imgur.com/cUA8ygb.png

Dutchman101

2013-11-07 06:19

updater   ~~0019631

@Sasuke,

Only thing I can think of, it being some kind of script/sql query spamming.. maybe a script/sql query, that has a number of mismatches for every single player. When more and more players get on, these might stack up to thousands per second, spamming the thread... and causing the cpu usage.

From which server revision this whole thing actually started happening?

One thing is for sure: The cause isn't with the MTA code/project and therefore not a bug, but it's a problem within your server environment, like the actual server, windows server, or else, the user-dependent aspects of your whole MTA server setup,
like faulty scripts, that might freeze threads as Arran said, or having bad sql queries as said above.

If you want to fix this there are options;

  • It's likely to be script/sql errors. Run debugdb as said by Arran and try to find out if this is the case.

  • Try to find out what initiated the problem, like changing server revision, moving to a new hosting company, or updating a script?

If you can't find the problem this way, last thing that you could try is to secure all scripts/databases and such, and go to the OVH hosting panel. Now perform a 'Factory reset' of your dedicated server, erasing all, making sure the server's installation is out-of-the-box, fixing all eventual problems associated with it.

Then copy your scripts/databases back, only NOT the server itself (binaries, etc), use a fresh, clean server installation for this and make sure it will be in the same state (now the only thing that should be different is that the server files are newly installed, but the same)

This comes to the point your server setup will be exactly the same as the old one, only thing is you'll have eventual problems with the OVH install of Windows Server sorted out, if they even existed, only now you're sure that isn't a possibility anymore.

arranTuna

2013-11-07 11:36

manager   ~~0019632

Your CPU does have hyper threading, which is why it shows as 8 cores in task manager when it's actually a quad core.

Intel Xeon E3-1245 V2 @ 3.40GHz
1,990 - single threaded points

Intel Xeon E5-1620 @ 3.60GHz
1,918 - single threaded points

http://www.cpubenchmark.net/singleThread.html

lol wut.

Your CPU has higher single threaded performance meaning it's a fraction better at running a high demand MTA server. Really though, I wouldn't worry about CPU usage, not until your server FPS is less than 5.

Sasuke*

2013-11-07 12:58

viewer   ~~0019633

Yeah man, I think you'r right but by FPS You mean Logic FPS the ones inside the parenthesis in the MTA Console ?
Also I just formatted the dedicated.. I cleaned everything. Installed latest updates. Framework 4.0 and 4.5 do you think its necessary to install every single existent C++ Redistributable ? or only the one that installs with MTA Setup think its the 2008.
I see your server has this version of MTA Server: 5899
Is it better to stick to 1 version rather to update every single time a Nightly comes out ?

arranTuna

2013-11-07 14:17

manager   ~~0019634

Yes, the logic thread FPS should drop first. It varies though as stuff like logging in, creating vehicles and reloading a gun use a lot of logic thread CPU but if a server was just 100s of players stood around or had really low sync times like 50ms then sync thread would use more first.

You should update nightly whenever there is anything worth updating for such as a server crash fix.

Sasuke*

2013-11-07 21:57

viewer   ~~0019636

I see Added thread performance stats
Im going to check out that

Sasuke*

2013-11-07 23:10

viewer   ~~0019637

Last edited: 2013-11-08 00:41

View 3 revisions

Check this out guys
http://i.imgur.com/sBWjqD6.png
Raknet thread CPU
What is that
EDIT:
Racknet using 94% (Kernel:12%)
Is that bad?
UPDATE
http://i.imgur.com/aZN6lXG.png
Raknet using 95-100% CPU
What is Raknet and why uses soo much

Woovie

2013-11-08 00:42

manager   ~~0019638

RankNet is the networking engine. That's too be expected I feel.

ccw

2013-11-08 00:48

administrator   ~~0019639

Last edited: 2013-11-08 00:56

View 3 revisions

After investigation, it seems like it's not a problem that the Raknet thread is 100% - It works that way for maximum responsiveness.

We will look into adding an option to increase latency (and therefore reduce CPU usage), at some point in the future.

Sasuke*

2013-11-08 01:10

viewer   ~~0019640

Ohh so its normal ?
If more and more players start joining there will be no problem ?

arranTuna

2013-11-08 17:12

manager   ~~0019642

It's normal.

Sasuke*

2013-11-09 19:48

viewer   ~~0019644

UPDATE:
I removed the script.log logging from the MTAServer.conf also I'm using this version https://code.google.com/p/mtasa-blue/source/detail?r=5916
"Reduced server CPU slightly and changed stats output" Slightly ?
Check this out:
http://i.imgur.com/HgndGDO.png
That its insane 166 players and look at the CPU usage is not even like before O.o

arranTuna

2013-11-13 23:39

manager   ~~0019660

Raknet thread CPU 3.1% (Sys: 3%)

That was 100% before, lol.

Issue History

Date Modified Username Field Change