Ruby RPC hurdles
September 11th, 2011
At the early stages of Arachni‘s development I had decided to use XMLRPC over HTTP for all distributed communications because it seemed like the easiest and most interoperable choice.
I wanted to allow everyone to take advantage of what Arachni had to offer and since all languages have an XMLRPC over HTTP library it seemed like the safest bet.
Thing is though… Arachni was started as a personal Ruby exercise which means that, at the time, I was not really familiar with Ruby’s quirks and shortcomings; at the time everything worked well enough so I hadn’t given it a second thought.
Moving forward, Arachni got bigger and complex and the distributed features became more resource intensive, not in a unmanageable way but more than my early day uninformed decisions were able to handle — I speak of course of WEBrick.
WEBrick is the standard Ruby webserver and is also used by the standard XMLRPC implementation but what the docs don’t mention is that you should avoid it like the plague.
I guess it’s alright for simple client-server stuff but if you’re aiming at something more complex with reasonable workloads it then becomes a huge resource hog — not to mention that it’ll block like crazy and may even downright stop responding.
This became apparent during Arachni’s Grid development even after my optimisation efforts.
I trimmed everything, counted data transfers byte-by-byte, tried to avoid blocking calls and used Threads when I couldn’t but still no dice.
The whole process worked better indeed but the resource consumption was crazy, the distributed components (WEBrick and the HTTP/XML parsers) required more resources than the scan itself.
Obviously that was unacceptable.
Which brings me to my point: If you’re thinking of doing the same, don’t.
Don’t use Ruby’s standard lib for anything RPC, look somewhere else.
As for me, I’m thinking of just using secure sockets, use YAML to serialise data in transit and only use data-types supported by all YAML implementations.
Simple implementation, serialise a hash like:
1 2 3 4 | { 'call' => 'modules.load' 'params' => [ 'xss' ] } |
Send it to the server, get back the serialised result, YAML.load it and you’re good to go.
Use SSL and async requests (or EventMachine if you can’t be bothered, like me) and you’ll get a much better, more stable, faster and more lightweight RPC infrastructure than XMLRPC over WEBrick.
When I’m done with mine I’ll post an update.
Posted in: Arachni, Programming, Projects, Ruby
Leave a comment
