[reSIProcate] Random.cxx and MultiCore systems

[Aron Rosenberg]

So this bug report concerns a very strange issue that we noticed on our
brandnew Dual Quad Core machine (8 cpu's) involving duplicate Call-Id's,
Transaction-ID's and Tag's being generated for independent INVITE's.
This behavior would then result in assert failures all over the stack.

 

We have a single instance of DUM/Resiprocate running on its own thread.
Our application generates 4 independent INVITE requests at the same
exact time which results in sequential calls eventually being made to
Random.cxx and then glibc's random() function. Of the four calls we get
the following random values returned

 

Call 1: aaaaaaaaaaa 

Call 2: bbbbbbbbbb

Call 3: aaaaaaaaaaa   (same exact sequence of random values as the first
call)

Call 4: bbbbbbbbbb  (same exact sequence of random values as the second
call)

 

Sometime later, various assert failures would occur due to duplicate TID
values and all sorts of other issues.

 

If pause or sleep the thread for 1 MS then the the problem disappears.
So what the heck is going on....

 

We think that DUM thread is being migrated across CPU's between the
different invocations of glibc's random() function and the "seed" value
is stale in a one of the CPU caches.

 

So how do we fix this - When we dug into the resiprocate Random.cxx code
we noticed that although we had linked against OpenSSL, the OpenSSL
random functions were not being used at all. They would be used to
initialize the seed but not used to actually generate the random values.

 

If we used the crypto versions of the functions the repeatedness issue
went away completely.

 

Here is a small patch which will use the crypto version if USE_OPENSSL
is defined

 

--- rutil/Random.cxx.orig              2008-03-14 23:21:29.000000000
-0700

+++ rutil/Random.cxx    2008-03-15 00:26:59.000000000 -0700

@@ -149,8 +149,9 @@

 Random::getRandom()

 {

    initialize();

-

-#ifdef WIN32

+#if USE_OPENSSL

+   return getCryptoRandom();

+#elif WIN32

    assert( RAND_MAX == 0x7fff );

    int r1 = rand();

    int r2 = rand();

 

 

-Aron

 

Aron Rosenberg

SightSpeed

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.resiprocate.org/pipermail/resiprocate-devel/attachments/20080319/d9467d21/attachment.htm>