[reSIProcate] Merged work from branches/b-repro-enh-20120205 to main
Scott Godin
sgodin at sipspectrum.com
Wed May 2 14:40:19 CDT 2012
High Level Summary or new Repro Features (Version changed to 0.9)
- New HTML settings page where you can view all current settings
- New Static / Permanent Registration Support - including HTML interface
for adding
- New Geo Proximity Target sorter processor
- New Request Filter processor and HTML configuration front end
- New Message Silo processor for storing IM's sent to offline users
- New reprocmd restart functionality for applying new settings without
dumping in memory registration table
see below for more details....
-up repro version number 0.9
-added new Settings page to repro web interface to show current command
line / files settings in use
- will also display some some low level stack info and congestion stats
(if enabled)
-added new config setting StatisticsLogInterval to specify how often
statistics are dumped to
the log files
-cleaned up the WebAdmin constructor to take a Proxy object
-added new KeyValueStore class for highly efficient generic storage
- keys are simple integer indexes into a vector and must be allocated
before they are used
- currently supports storage and retrieval of the following types:
Data, bool, short, unsigned short, int, unsigned int, and UInt64
-Converted ConfigParse storage from std::multimap to HashMultiMap
-removed print in TcpConnection that could end up printing garbage at the
end of
messages that are not null terminated
-added KeyValueStore to three strategic locations in repro, allowing custom
Processors
(Monkeys, Lemurs and Baboons) to store state scoped as follows:
-Global Proxy Scope - Proxy::getKeyValueStore
-Request Scope - RequestContext::getKeyValueStore
-Target Scope - Target::getKeyValueStore
Before this storage can be used you must statically allocate a storage key.
See mFromTrustedNodeKey use in the IsTrustedNode class for an example.
-major changes to how repro is started up, to allow easier additions of
custom startup logic, such as adding custom Processors (Monkeys, Lemurs
and
Baboons) to the default Processor chains
- almost all logic that was in repro.cxx has been moved out to a new class
(ReproRunner), and split into smaller virtual methods that can be
overridden
- see comments at the top of repro.cxx for an example of how to add custom
processors
-added option to enable some basic P-Asserted-Identity header handling
- After auth is successful
- if P-Perferred-Identity header is present remove it
- if no P-Asserted-Identity header is present, then add one
- Removal of P-Asserted-Identity if Privacy header is set to "id"
Note: Since we have no better mechanism to determine if destination is
trusted or not we assume that all destinations outside our domain
are not-trusted and will remove the P-Asserted-Identity header
-modified how ;lr param on routes are processed - moved logic from
StaticRoute
to ResponseContext - this slightly changes the behaviour
- previous implementation used to stop the processing chain as soon
as the first matching route with a ;lr is found, new implementation
does not stop Target processing if a ;lr is present
- Multiple targets can now have a ;lr on them and be processed
properly. The ;lr parameter is now examined in
ResponseContext::beginClientTransaction, and if present the RequestUri
is not altered and a Route header is added instead
-fixed a duplicate removeContact call in OutboundTargetHandler (a merge
conflict
victim)
-avoid unnecessary iteration through target list in StaticRoute when try to
determine
if auth is required
-added two new constructors to QValueTarget to make is easier to form
targets from
a NameAddr or Uri only
-removed some unused code in ResponseContext: addOutboundBatch and
mOutboundMap
-renamed Target::targetPtrCompare to Target::priorityMetricCompare to be
more descriptive
-implement proper q-value processing of contacts in a redirect response
-fix for a long standing issue in repro that started in rev6794, where
repro can be over protective and issue 403 responses for legitimate
mid-dialog requests. The issue occurs when a repro domain user forms
a dialog with a user in another external domain. Any mid-dialog
requests coming from the external domain would get 403'd. This
was due to the logic in the AmIResponsible monkey, and the fact that
such requests have the repro endpoints contact address in the
RequestUri (typically the endpoints IP address), so not belonging
to repro's domain, and the From user is not being from repro's domain.
The original code to only do these checks for out of dialog requests
was reintroduced with the following comments:
// only perform relay check for out-of-dialog requests
// !bwc! Um, then all anyone has to do to get us to be their relay
// is throw in a spurious to-tag...
// This smells funny. I am commenting it out.
// .slg. Putting code back in and clarifying the funny smell.....
// We only want to do this check for out of dialog requests,
since
// mid-dialog requests could be 403'd otherwise. Consider
// an INVITE request from a repro domain user to a user in
// another domain. The resulting ACK/200, BYE or any other
// mid-dialog request coming from the remote domain, will
contain
// the repro users contact address in the RequestUri and a
// foreign domain in the from header. We want to ensure
these
// requests are not 403'd. Byron's comment about an
endpoint getting
// us to relay by placing a spurious to tag in the request
still
// stands. Perhaps we ought to be checking the To header
domain in
// this case - however that is also weak, since the To
header is not
// used in routing and easily be set to a URI in our domain
to trick
// repro into forwarding. Note: From header domain
checking is
// stronger than To header domain checking, since if the
domain is
// ours, then it must pass Digest Authentication (at least
for non
// ACK and BYE requests).
-add ability for a repro admin to add manual / permanent registrations -
such
manually added registrations are persisted to the database, and loaded at
startup
- Manual registrations can be added on the Registration Web Page
-add display of registered contact's QValue on registrations web page
-added ability to specify a Path on new static registration feature
-cleanup some old hacks now that we have the ability to manually add
registrations
- ParallelForkStaticRoutes no longer combines StaticRoutes Targets and
LocationServer
Targets
- Static Routes are no longer added as QValueTargets, they are now added
as simple Targets.
So they are no longer susceptible to the various QValue Settings - ie.
QValueMsBeforeCancel.
- Added new ContinueProcessingAfterRoutesFound setting:
By default (false) we will stop looking for more Targets if we have
found
matching routes. Setting this value to true will allow the
LocationServer Monkey
to run after StaticRoutes have been found. In this case the matching
StaticRoutes become fallback targets, processed only after all location
server
Targets fail.
-adding GeoIP library - this commit is the unedited version 1.4.8 available
from http://www.maxmind.com/download/geoip/api/c/
-added new Helper method: Tuple getClientPublicAddress(const SipMessage&
request)
- look at Via headers, and finds the first public IP address closest to
the sending
client.
-added loopback address checking to Tuple::isPrivateAddress
-modified ServerRegistration:
- Ensure that ContactInstanceRecord::mReceivedFrom is always populated -
not just
in outbound use cases - added a new flag to indicate when flow routing
is required
- Added a new mPublicAddress flag member to ContactInstanceRecord - this
will assist
with an upcoming feature to do geo proximity routing
-added new ContactInstanceRecord::mPublicAddress to regsync process
-removed a chatty log line in DialogSetId
-used Symbols::COMMA in a few spots it wasn't being used in
-removed unimplemented targetCount method on ResponseContext
-renamed Helper::isSenderBehindNAT to isClientBehindNAT for consistency
-ensure we only skip the Monkey chain in StaticRoute if we actual found
routes
-added missing license block to ForkControlMessage
-adding new Baboon: GeoProximityTargetSorter
If enabled, then this baboon can post-process the target list.
This includes targets from the StaticRoute monkey and/or targets
from the LocationServer monkey. Requests that meet the filter
criteria will have their Target list, flatened (serialized) and
ordered based on the proximity of the target to the client sending
the request. Proximity is determined by looking for a
x-repro-geolocation="<latitude>,<longitude>" parameter on the Contact
header of a received request, or the Contact headers of Registration
requests. If this parameter is not found, then this processor will
attempt to determine the public IP address closest to the client or
target and use the MaxMind Geo IP library to lookup the geo location.
-requires floating point support for geo proximity calculations
-resip TCP transports can crash repro on uncaught exception - if garbage is
received
on the socket, and there is no Content-Length header, then
SipMessage::Exception can
throw, and it was not caught with the existing ParseException catch
handler. Changed
to catch BaseException instead.
-removed unsafe logging statement in DumTimeout - if DUM and stack are in
different threads, then
crash could occur
-enable Inserter functionality for collections that store pointers
to items. Use InsertP instead of Inserter for such collections.
For example:
std::vector<std::string*> v;
... populate vector ...
std::cout << InserterP(v) << std::endl;
-statictize strings in Inserter/InserterP
-add InserterP test cases to rutil/test/testInserter.cxx
-cleaned up Processor and ProcessorChain classes
- added mName property in anticipation of a future capability to define
processor
chains in the configuration file
- simplified operator<< for processors, to use Name
-modified some Inserter uses to use new InserterP function
-allow custom repro implementations to add themselves to the
RegistrarHandler so that
registration messages can be processed and reacted to
-added a singleResultQuery API to MySqlDb
-series of interfaces to make is easier to implement new Asynchronous
Monkey's / Processors that utilize a common thread pool
-removed getTransactionId from ForkControlMessage - method exists on base
class, so it's not needed
-removed getTransactionId and tid() from UserInfoMessage - getTransactionId
method exists on base class, so they are not needed
-made Worker and abstract base class
-modified ConfigParser to throw a resip style exception when failing to
open configuration file
-minor cleanups to ConfigParser configuration file parsing routine
-added new RequestFilter monkey
- allows user to configure conditions under which an inbound request
should be rejected or not
- allows two regular expression conditions that can be applied to any SIP
message header:
this includes the request-line, standard SIP headers and custom SIP
headers. If a header
that can appear multiple time is specified, then each instance of the
header is checked.
- When conditions are met, allows the action carried out to be defined:
- Accept - accepts this request and stops further processing in Request
Filter monkey
- Reject - rejects this request with the provided SIP status code and
reason text
- SQL query - only available when MySQL support is compiled in - runs an
arbitrary stored
procedure or query, using replacement strings from the 2 condition
regular expressions
- query must return an empty string or "0" to instruct repro to Accept
the request, or a
string containing "<SIP Reject Status Code>[, <SIP Reject Reason>]"
to Reject the request
- using the repro configuration file the SQL Query can be configured
to operate on a
completely different my SQL instance/server than the repro
configuration
- Filters are defined in the HTTP web interface via new Add Filter, Edit
Filter and Show
Filters web pages. There is an ability to test the condition regular
expressions from
the web page as well.
- other Monkey settings are configured in the repro configuration file
or via command line:
DisableRequestFilterProcessor, RequestFilterDefaultNoMatchBehavior,
RequestFilterDefaultDBErrorBehavior, RequestFilterMySQLServer (and
other mySQL related settings)
- can be used to implement a User Blocking functionality - ie. calls and
instant messages
from user X to user Y should always be blocked, because user X is in
user Y's block list
- introduced new FilterStore configuration database table to store the
Filters configured
on the web pages
-added AsyncProcessorDispatcher / thread pool to repro that be shared by
all AsyncProcessors
- currently only used by new RequestFilter monkey
-cleaned up some implementation in AbstractDb to remove some code
duplication
-catch config file parse, and missing file errors (exceptions) at repro
startup
-RouteStore
- stop webpage from being able to add two routes with the same Key
- optimized data fetch for displaying routes on web page
-Repro Web Interface
- added new Add Filter, Edit Filter and Show Filters pages
- made use of HTML tables consistent across all pages
- made table backgrounds white to improved appearance
- added title to right hand side pane
- cleaned up formatting on many pages
- ordered methods in source to match ordering on web page
- added warning to Domains page, that repro must be restarted
-added version information to startup logs and to display on web interface
-added bottom footer on web interface with link to www.resiprocate.org
-added new RequestFilter files to all Visual Studio projects and linux
Makefile
-upped repro version to 0.9
-fix a bug in repro web interface, where fragmented HTTP messages were not
being handled correctly
-fixed possible assert if a transport error is seen after trying to send an
ACK message
-updated to BerkeleyDb 4.8
-added optional MessageSilo support to repro
- stores IM's (ie. SIP MESSAGE requests) for offline users
- replays messages to users when they register (ie. come back online)
- records are persisted to a database table, so they survive shutdowns
- configurable filters exist for DestUri, MimeType, MessageBody size
-moved CommandServer from repro to reprolib
-added restart command to reprocmd - allows restarting repro to apply
configuration changes while leaving
the in memory registration database in tact
-propagate db insert/update failures to callers - web interface now shows
errors if record fails to update in db
-remove unused AbstractDb API's: writeRoute and writeFilter
-added ability for tables to have non-unique keys (ie. duplicate records)
-modified WorkerThread to support work that does not require a response to
be queued back to the stack
-added database transaction support to BerekelyDb and MySQL implementations
-reorganized repro runner startup ordering so that DUM TU is started before
Proxy and added new
virtual createDatastore method - in order to facility new Silo Monkey
startup that requires access to
the Registrar object
-make RequestFilter monkey options - use config flag
-allow a DateCategory to be created from time_t
-Added MySQL C Connector to contrib for Windows builds
-build fixes for linux, add --with-geoip option to autotools configure
-added --with-popt to configure so that test projects can be build easily
with popt support
-added autodetection of epoll support on unix and automatic addition of
HAVE_EPOLL define
-added ability for repro to report a 404 error when attempting to reach a
user that does not exist
- previously repro would always send a 480 response when attempting to
reach an AOR that wasn't
registered
-removed unused SipStack parameter to DigestAuthenticator monkey
-allow reprocmd to run defaulting to 127.0.0.1:5081 for server connection
-added secondary database support to BerkeleyDb - allows tables with a
secondary index
- my sql implementation of AVPs with two indexes was added as well
-message silo records can now be deleted by referencing a unique key (ie.
timestamp+tid)
- transaction support is no longer required
-optimized data copies when reading records from BerkeleyDb
-increase MySQL AVP table size for value field from 1024 to 4096
-implemented logic to periodically cleanup Silo records that have expired
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.resiprocate.org/pipermail/resiprocate-devel/attachments/20120502/ad128546/attachment.htm>
More information about the resiprocate-devel
mailing list