[reSIProcate] IntrruptableThread dead-lock

[Kennard White]

Hi,

There is an interesting dead-lock case with InterruptableStackThread under
linux. The interrupter can be invoked from within the SipStack thread itself
(one particular case is DnsResult -- see stack trace below). Under heavy
load the pipe can "fill-up" and the write blocks, causing dead-lock (e.g.,
the same thread that drains the pipe is blocked writing to the pipe).

For now, I've made the writes to the pipe non-blocking -- looks like equiv
change has already been made on the Windows side. That seems to solve the
problem.

Is there a better (or complementary) way to address this? In particular,
does the asyncHandler need to be invoked within DnsStub? When is DnsStub
invoked outside the SipStack thread?

One caveat: this occurs with epoll, and it is possible that it is some side
effect of how I changed the call structure. But I more likely I think it is
just the increase amount of activity allowed by using epoll enables the pipe
to fill up and expose this bug.

Thanks,
Kennard

(gdb) where
#0  0x00007ffff798204b in write () from /lib/libpthread.so.0
#1  0x00000000004838a6 in resip::SelectInterruptor::interrupt (this=<value
optimized out>) at SelectInterruptor.cxx:91
#2  resip::SelectInterruptor::handleProcessNotification (this=<value
optimized out>) at SelectInterruptor.cxx:51
#3  0x00000000004efb82 in resip::DnsResult::lookupInternal
(this=0x7fffe8837740, uri=<value optimized out>) at DnsResult.cxx:345
#4  0x00000000004f0042 in resip::DnsResult::lookup (this=0x7fffe8837740,
uri=..., enumSuffixes=...) at DnsResult.cxx:224
#5  0x00000000004b96d9 in resip::TransportSelector::dnsResolve
(this=0x7ffff7fc0d30, result=0x7fffe8837740, msg=0x7fffec7f9ce0)
    at TransportSelector.cxx:375
#6  0x00000000004af5e5 in resip::TransactionState::sendToWire
(this=0x7fffe879d220, msg=0x7fffec7f9ce0, resend=false)
    at TransactionState.cxx:2073
#7  0x00000000004b2818 in resip::TransactionState::processClientNonInvite
(this=0x7fffe879d220, msg=<value optimized out>)
    at TransactionState.cxx:710
#8  0x00000000004b3e9b in resip::TransactionState::processSipMessageAsNew
(sip=0x7fffec7f9ce0, controller=..., tid=...)
    at TransactionState.cxx:298
#9  0x00000000004b49c6 in resip::TransactionState::process (controller=...)
at TransactionState.cxx:579
#10 0x00000000004a4140 in resip::TransactionController::processEverything
(this=0x7ffff7fc0c48, fdset=<value optimized out>)
    at TransactionController.cxx:107
#11 0x0000000000497cfd in resip::SipStack::process (this=0x7ffff7f9d010,
fdset=...) at SipStack.cxx:589
#12 0x000000000049c194 in resip::InterruptableStackThread::thread
(this=0x842de0) at InterruptableStackThread.cxx:41
#13 0x000000000053249f in threadIfThreadWrapper (threadParm=0x6) at
ThreadIf.cxx:51
#14 0x00007ffff797aa04 in start_thread () from /lib/libpthread.so.0
#15 0x00007ffff6b26d4d in clone () from /lib/libc.so.6
#16 0x0000000000000000 in ?? ()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.resiprocate.org/pipermail/resiprocate-devel/attachments/20101120/e95620ad/attachment.htm>