[reSIProcate] Random.cxx and MultiCore systems

["Aron Rosenberg" <arosenberg@xxxxxxxxxxxxxx>]

So this bug report concerns a very strange issue that we noticed on our brandnew Dual Quad Core machine (8 cpu’s) involving duplicate Call-Id’s, Transaction-ID’s and Tag’s being generated for independent INVITE’s. This behavior would then result in assert failures all over the stack.

 

We have a single instance of DUM/Resiprocate running on its own thread. Our application generates 4 independent INVITE requests at the same exact time which results in sequential calls eventually being made to Random.cxx and then glibc’s random() function. Of the four calls we get the following random values returned

 

Call 1: aaaaaaaaaaa

Call 2: bbbbbbbbbb

Call 3: aaaaaaaaaaa   (same exact sequence of random values as the first call)

Call 4: bbbbbbbbbb  (same exact sequence of random values as the second call)

 

Sometime later, various assert failures would occur due to duplicate TID values and all sorts of other issues.

 

If pause or sleep the thread for 1 MS then the the problem disappears. So what the heck is going on….

 

We think that DUM thread is being migrated across CPU’s between the different invocations of glibc’s random() function and the “seed” value is stale in a one of the CPU caches.

 

So how do we fix this – When we dug into the resiprocate Random.cxx code we noticed that although we had linked against OpenSSL, the OpenSSL random functions were not being used at all. They would be used to initialize the seed but not used to actually generate the random values.

 

If we used the crypto versions of the functions the repeatedness issue went away completely.

 

Here is a small patch which will use the crypto version if USE_OPENSSL is defined

 

--- rutil/Random.cxx.orig              2008-03-14 23:21:29.000000000 -0700

+++ rutil/Random.cxx    2008-03-15 00:26:59.000000000 -0700

@@ -149,8 +149,9 @@

 Random::getRandom()

 {

    initialize();

-

-#ifdef WIN32

+#if USE_OPENSSL

+   return getCryptoRandom();

+#elif WIN32

    assert( RAND_MAX == 0x7fff );

    int r1 = rand();

    int r2 = rand();

 

 

-Aron

 

Aron Rosenberg

SightSpeed