[reSIProcate-users] Tracking resources to prevent memory exhaustion

[Deeptesh Patel <Deeptesh.Patel@xxxxxxxxxx>]

Hi,

 

I am trying to find out the best way to track memory usage or resource usage when the process runs out of memory. I am currently running a 32 bit application on at 64 bit kernel (RHEL 6.0). Under heavy bursty traffic, my application runs out of memory and coredumps. My application supports two types of traffic – one that uses SIP MESSAGEs to exchange data (after the dialog has been established) and the other that does not use any SIP MESSAGEs at all. It seems that the number of SIP MESSAGEs exchanged is contributing to a surge in server transaction count because with the other traffic where no SIP MESSAGES are used, the memory usage does not jump drastically. I am currently logging the following resip statistics every 20 seconds and the following is an extract of the stats just before the app coredumped (memory exhaustion):

 

[19:59:08.909] info:RESIP STATS: TU summary: 2 TRANSPORT 0 TRANSACTION 1 CLIENTTX 10880 SERVERTX 3869 TIMERS 53254

[19:59:28.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 0 CLIENTTX 10953 SERVERTX 3636 TIMERS 57140

[19:59:48.909] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 0 CLIENTTX 7036 SERVERTX 2389 TIMERS 46355

[20:00:08.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 1 CLIENTTX 6067 SERVERTX 3770 TIMERS 38257

[20:00:28.908] info:RESIP STATS: TU summary: 2 TRANSPORT 0 TRANSACTION 0 CLIENTTX 9966 SERVERTX 4560 TIMERS 59469

[20:00:48.908] info:RESIP STATS: TU summary: 152 TRANSPORT 2 TRANSACTION 1 CLIENTTX 11828 SERVERTX 4550 TIMERS 67448

[20:01:08.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 2 CLIENTTX 10088 SERVERTX 3342 TIMERS 62854

[20:01:28.908] info:RESIP STATS: TU summary: 4 TRANSPORT 0 TRANSACTION 1 CLIENTTX 6073 SERVERTX 2829 TIMERS 42442

[20:01:48.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 1 CLIENTTX 5661 SERVERTX 3470 TIMERS 37247

[20:02:08.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 0 CLIENTTX 5051 SERVERTX 2481 TIMERS 35385

[20:02:28.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 1 CLIENTTX 4881 SERVERTX 2428 TIMERS 34926

[20:02:48.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 2 CLIENTTX 4781 SERVERTX 2491 TIMERS 33820

[20:03:08.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 0 CLIENTTX 4319 SERVERTX 2318 TIMERS 32808

[20:03:28.908] info:RESIP STATS: TU summary: 0 TRANSPORT 0 TRANSACTION 1 CLIENTTX 4441 SERVERTX 2495 TIMERS 34361

 

Because of this mixed traffic, it is difficult to come up with an optimal max call capacity. Hence, I would like to find out if anyone have any suggestions on what I can track to start rejecting calls (I’d rather reject calls than crash and lose all calls) – the  “TU summary: 152” (tuFifoSize) and “TIMERS 67448” (activeTimers) appear to be pretty high. Are there any other counters that I can track/profile? I am currently using version 1.7.

 

In the meantime, I am going to try to use Google malloc (Tcmalloc) and try to do a memory introspection at regular intervals.

 

Thanks

-Deeptesh Patel