Community


All times are UTC - 5 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Wed Jul 20, 2016 9:00 am 
Offline

Joined: Tue Jul 27, 2010 10:33 am
Posts: 5
Dear Support Team,

I have a problem with the NeoRouter PRO (newest 2.4.0.4410):

Neorouter server continues to connect/ disconnect

Everything has worked fine for 3 weeks on a new machine but today the issue occurred.

Log is attached.
I would be grateful if you could provide us with the advice as soon as possible.

Thank you and kind regards,

MarcinD


Attachments:
NRServer_20160720_154333.log [142.53 KiB]
Downloaded 905 times
Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Wed Jul 20, 2016 12:55 pm 
Offline

Joined: Sun Nov 16, 2008 6:41 am
Posts: 1878
Hi ,

Thanks for your log file.

Based on the log file, I can see many sign in requests from NR clients were queued up and timed out, hence those NR clients tried to reconnect to the server.

2016-07-20 13:44:50,842|03784|02932|I|CClientMgr::OnAccept ServerAuthBacklog limit exceeded
2016-07-20 13:44:50,951|03784|03624|I|CHeartBeat::TimerCallback: network disconnected
2016-07-20 13:44:50,951|03784|03624|I|CHeartBeat::TimerCallback: close socket. (0x0)
2016-07-20 13:44:50,983|03784|02932|I|CClientMgr::OnAccept(256)
2016-07-20 13:44:50,983|03784|02932|I|CClientMgr::OnAccept ServerAuthBacklog limit exceeded
2016-07-20 13:44:51,341|03784|02932|I|CClientMgr::OnAccept(256)
2016-07-20 13:44:51,341|03784|02932|I|CClientMgr::OnAccept ServerAuthBacklog limit exceeded

It's more like your server gets overloaded. It means the server cannot handle so many concurrent request and it's too slow to process all requests in the timeout cycle. It may be caused by the busy CPU or the network throughput exceeded the limit. You may also need to take a look at the performance tuning guide http://www.neorouter.com/documents to optimize your OS and NR server. If need, upgrade your server or the network throughput limit.

Thanks,
KevinZ - NeoRouter team


Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Wed Jul 20, 2016 3:10 pm 
Offline

Joined: Tue Jul 27, 2010 10:33 am
Posts: 5
Dear KevinZ,

Thank you very much for you quick reply!

You mentioned that our server gets overloaded but it looks like: CPU usage 5% max, RAM usage 30% max therefore I don't think it's the hardware problem (please correct me if I'm wrong).
In our office there is a 25/25 Mbit fiber internet connection.

Can you be so kind and suggest some specific NR server parameters from the performance tuning guide? I'm sure your experience is wider than mine.

I've tried to change the ServerAuthBacklog (more than 10, and less than 10) but it didn't solve the problem.

Any help would be greatly appreciated!

Once again thank you and kind regards,

MarcinD


Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Wed Jul 20, 2016 6:46 pm 
Offline

Joined: Sun Nov 16, 2008 6:41 am
Posts: 1878
Hi marcin,

The performance of NR server depends on the number of CPU, CPU speed and network throughput. It doesn't need too much memory and disk space. So you may want to figure out what's the bottleneck, then improve it by optimizing the OS and NR server.

If I understand correctly, your current issue is NR clients keep reconnecting to the NR server.

Based on the information you provided, I cannot provide the accurate parameters, as your information are not clear. But I can provide you with my thoughts and you can test and tune it by yourself.

CPU usage: %5 - good, it also means you didn't use the full power of your CPU.

number of CPU: ? you didn't provide, but it's important

Memory: NR server doesn't really care about it, when the thread-pool size is set, the memory usage will be stable. The modem machine always provides enough memory for NR server.

Network throughput: you said 25/25 Mbit for your company. but we don't know how much for the NR server, as other users in your company may use the network as well.

number of NR clients: ? you didn't provide, it will be useful to estimate the network throughput

what kind of app used over the NR network: remote access or file transfer or video streaming or gaming? you didn't provide, but useful when estimate the network usage.

Anyway, based on the log file, I didn't see errors (may be you filtered them out). The only issue is some sign in requests got delayed. It's controlled by the parameter "ServerAuthBacklog". Since your CPU is not busy, try to increase the value of ServerAuthBacklog to make NR server consume more requests in the time. Once all NR clients sign in, your issue gets resolved. You can set it to 256, then collect the log file see what error do you have, then try 128, 64, 32 and so on, finally you will find a proper value for your server.

The speed of processing requests also related to the MaxInstanceThreads, that should be “2 * number of CPU cores”. So if your machine has 4 core, you can put 8, NR server leverage thread-pool to process requests, it means the NR server can process 8 requests in the same time, while the waiting list is ServerAuthBacklog. The authentication timeout is 17s. So if the CPU is powerful enough to process the queues in 17s, you will be good and won't see the "auth limit exceeded" error in the log.

I also found the following message in the log file:

1901 2016-07-20 13:45:04,835|03784|04788|I|CHeartBeat::TimerCallback: network disconnected
1902 2016-07-20 13:45:04,835|03784|04788|I|CHeartBeat::TimerCallback: close socket. (0x0)
1903 2016-07-20 13:45:04,898|03784|06396|I|CHeartBeat::TimerCallback: network disconnected
1904 2016-07-20 13:45:04,898|03784|06396|I|CHeartBeat::TimerCallback: close socket. (0x0)
1905 2016-07-20 13:45:04,945|03784|06396|I|CHeartBeat::TimerCallback: network disconnected
1906 2016-07-20 13:45:04,945|03784|06396|I|CHeartBeat::TimerCallback: close socket. (0x0)
1907 2016-07-20 13:45:05,023|03784|06396|I|CHeartBeat::TimerCallback: network disconnected
1908 2016-07-20 13:45:05,023|03784|04788|I|CHeartBeat::TimerCallback: network disconnected
1909 2016-07-20 13:45:05,023|03784|06396|I|CHeartBeat::TimerCallback: close socket. (0x0)
1910 2016-07-20 13:45:05,023|03784|04788|I|CHeartBeat::TimerCallback: close socket. (0x0)

Usually it indicates the network is overloaded at the moment, since the CPU is not busy. NR connection has a heart-beat mechanism and uses the heart-beat message to detect if peer is lost. If it cannot receive the message for 1 min, it thinks the peer lost network and closes the socket automatically. If your network is busy or broken for more than 1 min, some the existing connections would close. The peers will reconnect. In this case, you may want to investigate a little bit and figure out why this happens. If it exceeded the network throughput limit, then need to increase it; if it's blocked by firewall or something else, need to resolve it.

Let us know if you have any question or concern.

Thanks,
KevinZ - NeoRouter team


Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Thu Jul 21, 2016 3:23 am 
Offline

Joined: Tue Jul 27, 2010 10:33 am
Posts: 5
Dear KevinZ,

Thank you for your detailed explanations. I will try to answer your questions.

Quote:
If I understand correctly, your current issue is NR clients keep reconnecting to the NR server.

YES

Quote:
number of CPU: ? you didn't provide, but it's important

1 x CPU (4 cores)

Quote:
what kind of app used over the NR network: remote access or file transfer or video streaming or gaming?

Remote access mostly, some clients file transfer. No streaming, no gaming.

Quote:
Network throughput: you said 25/25 Mbit for your company. but we don't know how much for the NR server, as other users in your company may use the network as well.

Total network throughput: 50/50 Mbit; 25/25 Mbit for the office; rest is for "other things". Right now "other things" is only NRServer :)

Quote:
number of NR clients: ? you didn't provide, it will be useful to estimate the network throughput

70 active clients at the maximum

Quote:
Anyway, based on the log file, I didn't see errors (may be you filtered them out).

In the Log.ini file I set a parameter like:
Code:
Mask=268435455
I didn't filter errors out. If I set
Code:
Mask=67108863
then the log file is empty.

Right now I've changed the Feature.ini file to the below settings:
Code:
ServerAuthBacklog=256
MaxInstanceThreads=8
ServerAuthTimeout=30000


Unfortunately it didn't solve the problem. The logs attached.

We had the same issue 3 weeks ago and then we have installed the NRserver from the scratch (the same machine, the same ISP etc.). We were forced to add all the client settings (users and passwords) from the beginning. Mostly all the clients connected automatically without a problem. It has worked fine for 3 weeks till yesterday.

Any additional help would be greatly appreciated.

Kind regards,

MarcinD


Attachments:
NRServer_20160721_092723.log [77.21 KiB]
Downloaded 911 times
Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Thu Jul 21, 2016 2:05 pm 
Offline

Joined: Sun Nov 16, 2008 6:41 am
Posts: 1878
Hi marcin,

Can you change the Mask value to 536870911 , so we can get detailed information. The verbose level value is 4294967295 .

So far you log file looks good and I cannot see the error code. The Log file can be big if you enable the verbose level. Please zip it and send it to us support (at) neorouter (dot) com.

Thanks,
KevinZ - NeoRouter team


Top
 Profile  
 
 Post subject: Re: NR PRO 2.4.0.4410 cycling reconnect
PostPosted: Mon Jul 25, 2016 7:57 am 
Offline

Joined: Tue Jul 27, 2010 10:33 am
Posts: 5
Dear KevinZ,

Thank you for your reply.

On Thursday it started to work therefore I didn't want to change the log level and restart the service (after the restart the issue would reappear for sure).

Unfortunately today (Monday) the issue reappeared once again therefore I was able to change the log level.

I have sent the zipped log file to the provided email.

I'm looking forward to hearing from you.

Kind regards,

MarcinD


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 12 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron