Zapotek's train of thought...

Roundabout – A high-performance, distributed crawler

Zapotek — Wed, 16 May 2012 00:24:43 +0000

When I started development on the Arachni high-performance grid my focus was on the audit part, i.e. find a way to distribute the audit of batches of individual elements across multiple nodes and avoid duplication of effort amongst them.
It was a bit tricky to get right but it turned out to be quite do-able and worthwhile.

However, the crawl was done the old fashioned way, the master instance would crawl the targeted website and once completed it would then analyze all the pages it found and spread the workload.
I always intended to try out my hand on something similar but aimed towards the crawling process but it wasn’t a high priority.
But, as you can see from my last post, I did sort of figure it out, although I hadn’t had a chance to implement it until now.

This is tricky to do because there’s no way of knowing the workload before hand as it is basically a freaking labyrinth and precious information (new paths) can be hidden behind walls and walls of crap.

On the other hand, since when running Arachni in HPG mode you already have a few nodes up and running in the first place, why not utilize them a bit more — even if it turns out to be only slightly faster than a single crawler.

With that in mind, I yesterday started to implement that sort of a crawler, and here it is.
Its sole existence is that of a toy, a fun experiment, and not as a stable system. I may, in the future, put some more effort into it but my main reason for doing this is to explore this idea and eventually port it over to Arachni.

If you find this interesting, want to help out in researching or have any sort of feedback or just want to get in touch don’t hesitate to do so.

Cheers,
Tasos L.

Distributing the crawl

Zapotek — Thu, 19 Jan 2012 10:06:28 +0000

This one had been bugging me since I first started work on the HPG.
The gain you get from distributed computing is directly related to how efficient the workload distribution is — which makes sense.
The crawling process though doesn’t consist of a workload per se but rather looks for the workload.
Also, the difficulty of the crawl doesn’t lie in parsing or following the paths but actually finding the paths, this is because new paths are hidden behind old ones and as you progress new paths become sparser and sparser.
So the more work you do the less productive you become — these are grim prospects.

So here’s our problem, how the hell do you distribute something when you don’t know:

where it is
what it is
how big it is

Truthfully, in a single node setup this can be done quite easily and there are lots of ways to go about it
A basic crawling algorithm is actually one of the simplest around, it only has one rule: Follow every path but only once — when there are no more new paths you’re done.

In the most basic of implementations what you’d need to do this is a look-up table to help you keep track of the paths you’ve already followed in order not to go over them again.
Or you can amend the previous model by going the multi-threaded way and put the new paths in a queue, have workers pop paths from the queue, follow them and then report the new paths they find back to their coordinator for de-duplication and then put everything that passes filtering back to the queue and so the story goes…

When you do this on a single machine these approaches are good enough (and when you use async requests for the first/simplest approach it becomes more efficient than the latter one as well).
Thing is, these work well because you have the benefit of a multi Gbps bandwidth and close-to-zero latency pipe, your Front-Side Bus (or whatever computers have nowadays, haven’t kept up with h/w design).

So you can check if something is in the lookup table in something close to 0s, actually in Ruby the look-up time of a Set with 1,000,000 URLs in it is 9.735e-06 (0.000009735) seconds on my machine.
Which is effectively no time at all, you spend 0.000009735 seconds waiting for a decision before following each URL — ooooh scary.

However, when you need to do this over the network these response times take a dive — off a cliff.
You see, when distributing work you play with gain/cost ratios; if the ratio is good you go ahead, if not you go back to the drawing board.

Such a naive implementation deserves picking up the multi-colored markers again because:

Let’s say every worker communicates via an RPC protocol and the master worker maintains the look-up table and the work Queue.
Assuming that an RPC call costs about the same as an HTTP request and 25% of the paths contained in most pages are identical (nav menu, css, js, images, links to new blog posts etc.) this means that each follow operation will cost, per work unit:
1 RPC call to pop a path from the Queue +

25% of the amount of links found by following the path * 1 RPC — for paths that are common and have already been visited from the get go
75% of the amount of links found by following the path * 1 RPC call + 1 HTTP request — for new paths which aren’t in the lookup table and must be visited

So you’ll be spending most of your time waiting for permission rather than doing actual work and suddenly the cost of doing the work doubles.
There has to be a better way…

Annoyingly, it took me a couple of days to figure this out and it turned out what I needed was a relaxing shower and the answer come to me on its own — gooood answer *pat**pat*.

In all honesty, I had bits and pieces of the answer from the begging and I knew that the final algorithm would have to be a composite of models — a piece from the producer-consumer there, a bit of a policy-enforcer here sprinkled with some delegation across the board — but the problem was putting them in the right order to form a unified model that would:

Avoid any sort of blocking (no look-ups or waiting for decisions)
Automate load balancing
Prevent crawling redundant URLs (more than one worker following the same path)

Tricky stuff right?

And here’s what I came up with:

The master scopes out the place (follows 10 paths or so) and deduces the webapp structure — it will, most certainly, be incomplete but it doesn’t matter as we just want some seeds to get us going.
The master creates a per directory policy and assigns dirs to workers AND sends that policy to them as well.
Workers perform the crawl as usual but also implement that policy for URLs that don’t match their own policy rules i.e. send URLs that are out of their scope to the appropriate peer and let him handle it — the peer will ignore it if he has already visited it or put it in his queue.
If no policy matches a URL then it is sent back to the master; the master creates a new policy(ies), stores the work in a Queue and then sends an announcement to the workers (“There’s some work up for grabs!” ).
Busy workers ignore it; idling workers try to pull it and the work is assigned first-come/first-serve along with the updated policy.
Go to 3

If at any point a worker becomes idle he sends the paths he has discovered back to the master for storage/further processing/whatever and tries to pull some new work.

Also, the master will be a worker too — why waste a node, right?

Let’s go back to our list of requirements and see how we did:

Avoid any sort of blocking (no look-ups or waiting for decisions) — ✓ Instead of waiting for permission to do something, we delegate to the appropriate authority and forget about it
Automate load balancing — ✓ Workers pull work when they are good and ready
Prevent crawling redundant URLs (no more than one worker following the same path) — ✓ A local look-up table and item #1 take care of that quite nicely

In addition, the policy can use any sort of criteria (directories were just an example), which means that we can achieve very granular distribution if we are a bit clever with it.

I think that this serves as a decent starting point, there’s still the issue of how to efficiently group the new URLs that are fed back to the master (because they don’t match any of the initial policy rules) but I’ll have to see this working under real world conditions in order to get a better feel for it first.
I’ve got some stuff in mind, we’ll see…

Please do comment if you have a suggestion of have spotted a fault somewhere.

– Tasos

Arachni is moving away from GPLv2 and towards Apache License v2.0

Zapotek — Thu, 12 Jan 2012 15:40:22 +0000

Yes, it’s true…
As of now the code in the experimental branch has been converted to use the Apache License Version 2.0.

If you’re interested in why this happened here’s the deal:
There are currently a few companies that use Arachni internally and a few others that actually provide SaaS security services using Arachni’s distributed features.
Thing is though, a lot of companies can’t touch GPL code (not that I blame them) which isn’t good for them nor Arachni as neither of us gets what he wants.
It makes sense, surely, but it was about a month ago that it really clicked as I were reading the comments of a Slashdot article.
Lots of people were agreeing on the same subject, the money-men don’t like GPL which kind of sucked for the project.

At this point I started researching alternative licenses and started asking around a bit.

As fate would have it — although more due to the increasing userbase I guess — a few people told me that the GPL was a deal-braker for them and I even had one guy tell me that he couldn’t include Arachni in his book because of it (I’ll spare him his blushes and not say his name).

Now that the project is gaining some momentum these technicalities become more and more important.

So after a bit of research I settled on Apache License 2.0 mainly because of its trademark and patent grants (because who the hell wants to deal with that bureaucratic crap?) and the requirement to redistribute a visible copy of the original work’s NOTICE file (if it includes one) which is nice since hard work must be properly credited.
You know, you can use my work for free (and I hope you do ) but mention that “this product contains some code from that Arachni thingy written by this bloke with the funny name”.

So that’s the reason, I’m hoping that a more permissive license will increase adoption and make everybody’s life easier.

Arachni v0.4.0.2 HOTFIX

Zapotek — Mon, 09 Jan 2012 20:23:08 +0000

Hi guys,

A couple of days ago I proudly released v0.4 and, as luck would have it, I later had to swallow some of that pride due to a couple of intermittent bugs that I hadn’t spotted.
Well, worry no more as I’m writing this post to announce a rush hotfix version of Arachni, v0.4.0.2.

If you installed the previous version via “gem install” or have downloaded the previous Cygwin package then all you need to do is issue:

1	gem install arachni

If not, you can find links and instructions for all available packages at: http://arachni-scanner.com/latest

Enjoy

Arachni v0.4 is out

Zapotek — Sat, 07 Jan 2012 06:48:43 +0000

Yes, yes…the time has finally come and there are boons for everyone.
This release features the most impressive ChangeLog yet and the first (and coolest) thing in this long list is the brand new High Performance Grid implementation — which has been discussed extensively in the past.

Let’s review the big points.

New RPC infrastructure

Ruby’s XMLRPC has been ditched (as initially discussed in these two [1, 2] posts) in favor of Arachni-RPC.
Arachni-RPC is lightweight, simple and fast which makes it ideal for large Grid deployments and makes it easy for 3rd parties to interoperate with Arachni’s servers.

Notice: If you were using the old XMLRPC interface please update your code to use the new RPC API.

High Performance Grid

I’ve been talking about this one so much that I’ve actually grown a bit sick of it — joking aside though this is one of Arachni’s most important features.
It allows you to connect multiple nodes into a Grid and use them to perform lightning-fast scans.

This is due to the way Arachni distributes the workload, which is finely grained down to individual page elements to ensure fair and optimal distribution; because workload distribution is so fluid it effectively becomes a sort of bandwidth and CPU aggregation.

To put this in simple(-istic) terms:
If you have 2 Amazon instances and you need to scan one site, by utilising the HPG you’ll be able to cut the scan time down to approximately half of what it would take by using a single node (plus the initial crawl time).

And if you have a huge site you can use 50 nodes and so the story goes…

This feature was an imaginary, almost unattainable, milestone back when I added the initial client/server implementation and I didn’t really think that I’d ever be able to make it happen.
Luckily, I was wrong and I’m proud to present you with the first Open Source High Performance Grid web application security scanner!
(By the way, does anyone know of a commercial scanner that can do this?)

Notice: With the WebUI’s updated AutoDeploy add-on you’ll be able to go into World domination mode by performing point and click Grid deployments!
Another notice: Use responsibly, don’t DDoS people.
Yet another notice: It’s still considered experimental so let me know if you come across a bug.

Updated WebUI

The WebUI now contains a few context-sensitive help dialogs to help out the newcomers and it has been updated to use the Thin webserver to send responses asynchronously in order to increase performance and feel “snappier”.
It also supports HTTP basic auth just in case you want some simple password protection and has been updated to provide access to the brand new HPG goodies.

Spider improvements

There was a bug with redirections that prevented the spider from achieving optimal coverage which has now been resolved.
More than that, the scope of the crawl can now be either extended or restricted by supplying newline-separated lists of URLs which should help you import 3rd party sitemaps.

Plugins

The plugin API has been extended in order to allow plugins to let the framework know if they can be distributed across HPG Instances and, if so, how to merge their results for the final report.

Another big (although invisible to the end-user) change is the conversion of all meta-modules to full-fledged plugins to simplify management and Grid distribution.

And these new plugins have been added:

ReScan — It uses the AFR report of a previous scan to extract the sitemap in order to avoid a redundant crawl.
BeepNotify — Beeps when the scan finishes.
LibNotify — Uses the libnotify library to send notifications for each discovered issue and a summary at the end of the scan.
EmailNotify — Sends a notification (and optionally a report) over SMTP at the end of the scan.
Manual verification — Flags issues that require manual verification as untrusted in order to reduce the signal-to-noise ratio.
Resolver — Resolves vulnerable hostnames to IP addresses.

Modules

I’ve got both good and bad news for this….
In an attempt to cleanup and optimise pattern matching in v0.3 I inadvertently broke some aspects of it which crippled the XSS (xss), SQL injection (sqli) and Path Traversal (path_traversal) modules –I sincerely apologise, mea culpa.

The good news is that I’ve made things right, cleaned up the API and the existing modules and improved their accuracy.

Reports

The HTML report has waved goodbye to Highcharts due to licensing reasons and now uses jqPlot for all its charting and graphing needs.
I’ve also removed the “report false-positive” button since a part of that process required RSA encryption which for some reason caused segfaults on Mac OSX.
Good news is that the HTML reports will be significantly smaller in size from now on.

Moreover, the following new report formats have been added:

JSON — Exports the audit results as a JSON serialized Hash.
Marshal — Exports the audit results as a Marshal serialized Hash.
YAML — Exports the audit results as a YAML serialized Hash.

Cygwin package for Windows

About time indeed, Windows users can now enjoy Arachni’s features — albeit via a preconfigured Cygwin environment.
The important point is that you no longer have to manually hassle to install Arachni via MinGW or Cygwin yourselves or use a VM and what have you…
Simply download and run the self-extracting archive, double click the “Cygwin” batch file and lo and behold: you’ve got a bash shell ready to execute Arachni’s scripts.

Unfortunately, there’s a performance penalty involved when running Arachni in Cygwin but until I port it to run natively on Windows it’ll have to do.

Before I forget, the Wiki has been cleaned up and brought up to date so if you need to go through the documentation that should be your first stop.

Go ahead and enjoy this new release everyone, download links and instructions here: http://arachni.segfault.gr/latest.

Cheers,
Tasos L.

Playing around with DOM XSS (Demo included)

Zapotek — Mon, 12 Dec 2011 00:21:42 +0000

I’ve caught a bug it seems and because I can’t just sit on my ass all day I figured why not play around with my latest toy.

I’ve updated the code to make the process more streamlined and allow for fuzzing (or at least altering) some possible input vectors.
Things will be very very simple for now as I’m merely trying to demo that with appropriate effort invested this can become a viable solution — eventually.

To the point, I’ll showcase a DOM XSS vulnerability that will take place purely on the client-side.
Unfortunately, there are a lot of DOM interfaces/vectors that I haven’t yet implemented so let’s stick to one I have — navigator.userAgent.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

require_relative 'init'

html = <

My title!

EOHTML

#
# the second param sets 'dont_eval_js' to true
#
# we want to do that ourselves later on after we've prepared the vectors
# (navigator.userAgent in this case)
#
window = DOM::Window.new( html, true )

#
# we'll inject this fictional tag and look for it in the DOM structure later on
#
# if found, then we have an XSS vuln
#
seed_tag = 'myinjectedtag'

# our XSS vector
window.navigator.userAgent = "<#{seed_tag}>blah blah blah

# this will show the HTML as is (Ref. #1)
# puts window.document.to_html
# puts '-' * 80

# execute the JS
window.instance_eval { exec_js! }

# this one will show the updated HTML i.e. including our tag (Ref. #2)
# puts window.document.to_html

# look for the tag in the DOM structure
if window.document.getElementsByTagName( seed_tag )[0]
puts 'Vulnerable to XSS!'
end

Ref. #1
This is what the code looks like at this point:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>My title!</title>
</head>
<body>
<div>
<script type="text/javascript">
document.write( navigator.userAgent );
</script>
</div>
</body>
</html>

Pretty much what we passed…

Ref. #2
And now that the JS has been executed:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Yes I know, document.write() should have written inside the parent of the script tag, the div in this case.
I can’t be bothered to take care of this right now, this is just a prototype…a prototype of a prototype of the prototype actually.

End result
Since we can see that the injection was clearly successful, the inevitable message shall appear:

1	Vulnerable to XSS!

Conclusion
Yeah it’s really basic and simplistic and not a big deal but it’s fun to see things working — barely but still, heh.

Experimentng with DOM and JavaScript support for Ruby

Zapotek — Mon, 05 Dec 2011 01:33:34 +0000

One of the things everyone is taking for granted nowadays for every browser and website is decent support for AJAX.
Naturally, scanner devs have been trying to find a decent way to automatically audit that side of the fence or at least provide decent coverage for JS-heavy webapps.

Thing is though… this is a bitch to get right.
And never mind getting it right, it’s hard enough getting the damn thing to work to begin with.

There are 3 things that need to be integrated in order to achieve that sort of functionality:

DOM

The static parts of the DOM can easily be built on top of tested and proven XML parsers like libxml.
The DOM has tricky parts too though, which are where the money is, like timers and events.

It’s alright though, a little bit of smart thread scheduling and clean collections of callbacks will sustain you at the beginning.

JS

JS integration difficulty depends on your language of choice.
Thankfully, more and more bindings for several different JS engines are being released so you can have your pick.
The interfaces can be a bit dodgy though at times which can defeat the whole purpose.

AJAX

AJAX functionality will have to fall on your shoulders but if you managed to get the first 2 parts working this won’t be much of a challenge.
You simply write an AJAX API in your language of choice and make that interface available to the JS code.

Where I’m going with this…

It’s certainly possible to get these working together (without being a multi-million dollar corporation even) but it’ll be a lot of work.
Which is the reason there’s no open source scanner that supports AJAX or even basic JS scripting.

Arachni is no exception, not to mention that due to the young age of the system there had been far more important and basic things to be worked out first.

Luckily, a lot of things have changed in very little time.
The project still has a few bugs but it has been stable enough for a few businesses to build some of their infrastructure on it.
And the v0.4 version if pretty much ready, which takes care of another big feature I could not wait to implement — the High Performance Grid.

Next stop: AJAX

Some time ago I was contacted by a CompSci student who had chosen to add AJAX support to Arachni as his final-year project, nice guy and seems motivated so I’m pretty sure that this is gonna happen.

And since he mentioned it I decided to start looking into the subject matter myself with this:
https://github.com/Zapotek/dom-js-experiment

It’s based on Taka (for the static parts of the DOM) and the V8 JS engine as provided by TheRubyRacer.

I’ve managed to make this work with some simple stuff, JQuery loads without errors and kind of works (I haven’t had much time to test it).
I haven’t had time to implement timers and events yet but they’re coming…
I don’t want to finish this thing on my own though, the other guy will need to work on it for his project.

Point is…AJAX support is coming to Arachni; it will of course take time but it’s going to happen.

Programatically scanning using Arachni (Part 5)

Zapotek — Sun, 04 Dec 2011 03:03:41 +0000

As promised, part 5.
Not that anyone’s reading this crap, once I’m done with the series though I’ll be able to gather them into a nice developer’s guide so I might as well keep going.

As always, keep your installation to up date with the experimental branch before continuing.
These articles have forced me to see Arachni from a completely different perspective and so I keep improving the API to make it more developer friendly.

This time we’ll focus on auditing individual elements and also work on a per page scope.

Let me paint you a picture:

You have a Rails (or some other such framework) web application.
You need to audit it in a consistent manner.
You wish you could simply add security tests to your existing test suite next to your units/functional/integration/etc. tests.
Your webapp framework already keeps track of pretty much all inputs (and if not you can override helper methods like link_to and form_for to keep track of them).
The only thing missing is a system to which you can feed that data and audit those inputs.

You see where I’m going with this, right?

The Arachni framework can easily handle this in a number of ways, some of which I’ll demonstrate here.

Scripted element audits

Let’s start from the outside and move inwards.
The framework feeds pages to modules –> modules audit pages –> pages include elements –> modules inherit from Arachni::Module::Base –> Arachni::Module::Base includes Arachni::Module::Auditor and Arachni::Module::Output.

And everything’s at its place, modules live under “/modules” and everyone’s happy.

I’ve already showed how to script an audit but what if you want a narrower scope?
What if you have a list of elements you want to audit?
What if you want to pass a custom page to be audited?
And more importantly, how can you do that without needing to work with the framework, the module manager and without needing to write a full blown module?
What if you just want the functionality without the hassle?

Obviously that’s possible, otherwise I wouldn’t have gone through that sales pitch.
What we need to do is create our own Auditor at runtime… better yet, let’s create our own Module so that way we’ll have access to some utilities and other toys as well.
Through that Auditor we’ll be able to submit, fuzz, audit, analyze elements and server responses.

Time to go through the 2 cleanest ways to do that.

First of all, this will be our Auditor, it’ll serve as a proxy to the functionality enjoyed by regular framework modules.
auditor.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

Each Auditor must have a page as a reference if only to know the current URL, so a page with at least a URL assigned must be available at all times.

Per element

This approach deals with auditing individual, user created, elements, which is the finest grained of controls possible.
Talk is cheap though, let’s test a few examples.

Pattern matching

Simple enough, inject a string and see if it appears in the response:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testfire.net'

# the owner is supposed to be the URL of the page containing the element
owner = opts.url.to_s

# let's create a link element to audit
link = Arachni::Parser::Element::Link.new( owner, {

# action, to which URL to send the data
:action => owner + '/notfound.aspx',

# auditable inputs, these will be fuzzed/audited/whatever
:inputs => {
'aspxerrorpath' => ''
}
})

# seed to inject
injection_str = ''

# some audit options
audit_opts = {
# the existence of this substring in the response will verify the issue
# :substring => injection_str,

# is this is a string it'll be converted to a regexp
:regexp => injection_str,

# controls the format of the injected string,
# this one means: inject as is
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ]
}

# get an auditor and pass it the seed URL, it'll also be used as the page URL
auditor = Auditor.new( opts.url.to_s )

# the name is a dead give-away
#
# the 1st param is an array of elements to audit
# the 2nd one is the string to inject
# the 3rd one is a hash with audit options
#
# if a server response contains the :substring (or matches a :regexp)
# then an issue will be logged automatically
#
auditor.audit_elems( [link], injection_str, audit_opts )

# this fires the queued requests and blocks until all responses have arrived
auditor.http.run

# grab any logged issues/results
results = Arachni::Module::Manager.results
auditor.print_info "Found " + results.count.to_s + " matches."

This time we don’t mute the output, and why should we? Let’s admire our handy work.

But what if you don’t want the auditor to log the the issue automatically based on an expression or substring?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

audit_opts = {
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ]
}

# if you pass a block you get full control over how to analyze the audit responses
auditor.audit_elems( [link], injection_str, audit_opts ) {
|response, options, element|
# full HTTP response
# pp response

# expanded options
# ap options

# updated element
# pp element

# perform your own analysis and logging
auditor.log( options, response ) if response.body.include?( injection_str )
}

Differential analysis

The Auditor module contains helpers that perform differential analysis for you using Arachni’s own rDiff algorithm.

See this SQL injection example for instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testphp.vulnweb.com/artists.php'

# the owner is supposed to be the URL of the page containing the element
owner = opts.url.to_s

# let's create a link element to audit
link = Arachni::Parser::Element::Link.new( owner, {

# action, to which URL to send the data
:action => owner,

# auditable inputs, these will be fuzzed/audited/whatever
:inputs => {
'artist' => '1'
}
})

# get an auditor and pass it the seed URL, it'll also be used as the page URL
auditor = Auditor.new( opts.url.to_s )

rdiff_opts = {
#
# fault injection seeds
#
# these must generate an exceptional condition or
# general error when appended to an SQL query
#
:faults => [ '\'"`' ],

#
# boolean injection seeds
#
# these values should not affect the behavior of the page when evaluated
# as part of the SQL query
#
:bools => [ ' and 1+1 = 2' ],

#
# rDiff precision
#
# Specifies the amount of rdiff iterations,
# more iterations => more refined/accurate result
#
# default is 2 but what the hell...
#
:precision => 5
}

# analyze and log
auditor.audit_rdiff_elem( link, rdiff_opts )

# this fires the queued requests and blocks until all responses have arrived
auditor.http.run

# grab any logged issues/results
results = Arachni::Module::Manager.results
auditor.print_info "Found " + results.count.to_s + " matches."

Again, if you want to perform your own analysis and logging:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

auditor.audit_rdiff_elem( link, rdiff_opts ) {
|injection_string, element, original_response_body, bool_response, fault_response_body|

# injected string
# p injection_string

# updated audited element element
# pp element

# body of the original/vanilla page
# pp original_response_body

# full HTTP response for the boolean injection
# pp bool_response

# body of the fault injection response
# pp fault_response_body
}

Time-out/delay attacks

Yes, timing attacks are of course part of the repertoire but are not available on a per-element basis.
Don’t worry though, there’s an alternative way to perform them as we’ll see shortly.

Per page

The per-page approach is slightly different and requires some commitment as you have to you properly configure a page object with elements and feed that page object to the module.
The module then audits those elements in batch.

Pattern matching

Let’s see what the pattern matching example looks like when adjusted for this approach:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testfire.net'

owner = opts.url.to_s

link = Arachni::Parser::Element::Link.new( owner, {
:action => owner + '/notfound.aspx',
:inputs => {
'aspxerrorpath' => ''
}
})

# create our page and assign it the link
page = Arachni::Parser::Page.new(
:url => owner.to_s,
:links => [link]
)

# get an auditor and pass it the page
auditor = Auditor.new( page )

injection_str = ''
audit_opts = {
:regexp => injection_str,
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ]
}

auditor.audit( injection_str, audit_opts )

auditor.http.run

results = Arachni::Module::Manager.results
auditor.print_info "Found " + results.count.to_s + " matches."

I’ve removed most of the comments this time since they were identical to the first example.

I just had an idea, let’s add an audit method to the page object itself and see if it’s any easier to work with.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testfire.net'

owner = opts.url.to_s

class Arachni::Parser::Page
def audit!( injection_str, opts, run = true, &block )
auditor = Auditor.new( self )
auditor.audit( injection_str, opts, &block )
auditor.http.run if run
end
end

link = Arachni::Parser::Element::Link.new( owner, {
:action => owner + '/notfound.aspx',
:inputs => {
'aspxerrorpath' => ''
}
})

# create our page and assign it the link
page = Arachni::Parser::Page.new(
:url => owner.to_s,
:links => [link]
)

injection_str = ''
audit_opts = {
:regexp => injection_str,
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ]
}

page.audit!( injection_str, audit_opts )

results = Arachni::Module::Manager.results
puts "Found " + results.count.to_s + " matches."

Huh, seems kinda nice…

Differential analysis

Should be a familiar sight:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Time-out/delay attacks

And now it’s time to check out some timing attacks.
Actually, since timing attacks are wildly unreliable Arachni doesn’t simply perform a time-out check.
It performs a few time-out checks with different delay tolerances and waits for the server to normalize in between and it also does some other weird stuff to make sure that false positives are a rare occasion.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testphp.vulnweb.com/artists.php'

owner = opts.url.to_s

link = Arachni::Parser::Element::Link.new( owner, {
:action => owner,
:inputs => {
'artist' => '1',
}
})

page = Arachni::Parser::Page.new( :url => owner )
page.links << link

auditor = Auditor.new( page )

timeout_opts = {
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ],

#
# set first stage timeout to 4 seconds
# the rest of the stages are based on this value
#
:timeout => 4000,

#
# different platforms expect different time representations,
# some expect seconds and others milliseconds.
#
# __TIME__ in the injection string will be replaced with:
# :timeout / :timeout_divider
#
#
# optional and defaults to 1
#
:timeout_divider => 1000
}

auditor.audit_timeout( [ 'sleep(__TIME__)#' ], timeout_opts )

# timing attacks are a bit different and are scheduled to be run at the end
# of a full audit
#
# that's why simply firing up the queued HTTP requests (like with the other methods)
# isn't enough
Arachni::Module::Auditor.timeout_audit_run

results = Arachni::Module::Manager.results
auditor.print_info "Found " + results.count.to_s + " matches."

Simple as pie.

Page delegation

Let’s investigate this idea a bit further, shall we?
Having audit functionality built-into the page objects would surely be helpful!

Time to monkey-patch the Page class I reckon so update your auditor.rb file to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

require 'arachni/ui/cli/output'
require 'arachni'

#
# This class is capable of providing module-like functionality to 3rd party objects.
#
# This is the same as a regular module but instead of including callbacks like:
# * prepare
# * run
# * clean_up
#
# to perform its job, its functionality is exploited by other classes.
#
class Auditor < Arachni::Module::Base

#
# If the parameter is a Arachni::Parser::Page that's going to be the
# reference page of the auditor.
#
# If the param is a string it'll be treated like a URL and a new page
# will be created for it.
#
def initialize( page_or_url )
# setup a minimal page to have as a reference
if !page_or_url.is_a? Arachni::Parser::Page
super Arachni::Parser::Page.new( :url => page_or_url.to_s )
else
super page_or_url
end
end

def self.info; { :name => 'My auditor' } end
end

class Arachni::Parser::Page

#
# Trap any called methods that don't exist in the page class
# and send them to the Auditor class if they are audit methods.
#
def method_missing( sym, *args, &block )
tokens = sym.to_s.split( '_' )

if tokens.first != 'audit'
raise NoMethodError.new( "Undefined method '#{sym.to_s}'.", sym, args )
end

@auditor ||= ::Auditor.new( self )
@auditor.method( sym ).call( *args, &block )

# check for a :run flag explicitly since we may need to queue up
# a number of audits
opts = {}
args.flatten.each {
|arg|
opts.merge!( arg ) if arg.is_a?( Hash )
}
return if !opts[:run]

if tokens.last != 'timeout'
@auditor.http.run
else
::Arachni::Module::Auditor.timeout_audit_run
end

end
end

And from that point on you can simply audit pages like so:

Audit combo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

require_relative 'auditor'

opts = Arachni::Options.instance

opts.audit_links = true
opts.audit_forms = true
opts.url = 'http://testphp.vulnweb.com/'

owner = opts.url.to_s

page = Arachni::Parser::Page.new( :url => owner )

#
# Vulnerable to SQL injection and will be flagged by rDiff and timeout attacks.
#
page.links << Arachni::Parser::Element::Link.new( owner, {
:action => owner + 'artists.php',
:inputs => {
'artist' => '1',
}
})

#
# Vulnerable to XSS.
#
page.forms << Arachni::Parser::Element::Form.new( owner, {
:action => owner + 'search.php',
:method => :post,
:inputs => {
'searchFor' => ''
}
})

#
# rDiff audit
#
# Just queue the requests, don't run them yet.
# It's better if we combine these requests with the pattern matching audit
#
rdiff_opts = {
:faults => [ '\'"`' ],
:bools => [ ' and 1+1 = 2' ]
}
page.audit_rdiff( rdiff_opts )

#
# Pattern matching
#
# Will submit the queued HTTP requests.
#
injection_str = ''
audit_opts = {
:regexp => injection_str,
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ],
:run => true
}
page.audit( injection_str, audit_opts )

#
# Timing attack
#
# It's better to run it at the end of the audit on its own .
#
timeout_opts = {
:format => [ Arachni::Module::Auditor::Format::STRAIGHT ],
:timeout => 4000,
:timeout_divider => 1000,
:run => true
}
page.audit_timeout( [ 'sleep(__TIME__)#' ], timeout_opts )

results = Arachni::Module::Manager.results
puts "Found " + results.count.to_s + " matches."

Running all modules

We may have gone a little bit too deep it seems…
You may not want to perform the audit yourself…
You may want to specify a narrow scope, on a per-element basis, but use the existing modules to audit them.
How do you do that?

First of all, you can’t give Arachni’s modules individual elements…you have to give them pages.
That’s alright though since as we’ve seen today creating pages containing elements of our choosing is no big deal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

require 'arachni/ui/cli/output'
require 'arachni'

opts = Arachni::Options.instance

opts.audit_links = true
opts.url = 'http://testfire.net'

owner = opts.url.to_s

page = Arachni::Parser::Page.new( :url => owner.to_s )
page.links << Arachni::Parser::Element::Link.new( owner, {
:action => owner + '/notfound.aspx',
:inputs => {
'aspxerrorpath' => ''
}
})

modules = Arachni::Module::Manager.new( opts )

# doesn't make sense to load recon modules in this case
modules.load( ['audit/*'] )
modules.run( page )

Arachni::HTTP.instance.run

results = Arachni::Module::Manager.results
puts "Found " + results.count.to_s + " matches."

That’s all for now, I hope you that found this post interesting — I certainly did.

Cheers

Programatically scanning using Arachni (Part 4)

Zapotek — Fri, 02 Dec 2011 00:10:44 +0000

Since last time we discussed documentation (boring) this post will be about something functional (cool).
And that coolness will in the form of a user interface, beginning with a simple progress output and moving to a full blown console environment — which also means scripting too.

By the way, don’t forget to grab and install the code of the experimental branch and remember to keep it updated before going through each article.

Creating a User Interface

Let’s start simple, just a script that performs a scan like the ones we’ve seen so far:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require 'arachni/rpc/server/output'
require 'arachni'

# get an instance of the options class
opts = Arachni::Options.instance

opts.audit_forms = true

# this is the seed URL
opts.url = 'http://testfire.net/'

# instantiate the framework
framework = Arachni::Framework.new( opts )

# load just the xss module
framework.modules.load( [ 'xss' ] )

# load default plugins
framework.plugins.load_defaults!

framework.run

puts "Found: " + framework.audit_store.issues.map { |i| i.name }.join( ', ' )

Nothing fancy, this will scan http://testfire.net, audit all forms for XSS vulnerabilities and then print the names of the issues it found — XSS obviously, 2 of them.

Helpers

If we are to have a decent interface we’ll need some toys to make it more informative and appealing.
Remember the show_progress() method from Part 2? Let’s take it up a notch.
Let’s abstract the terminal related behavior and put it in its own module to remove duplication and ease re-use.

terminal.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#
# Terminal manipulation methods
#
module Terminal

#
# Clears the line before printing using 'puts'
#
# @param [String] str string to output
#
def reputs( str = '' )
reprint str + "\n"
end

#
# Clears the line before printing
#
# @param [String] str string to output
#
def reprint( str = '' )
print "\e[0K" + str
end

#
# Clear the bottom of the screen
#
def clear_screen!
print "\e[2J"
end

#
# Moves cursor top left to its home
#
def move_to_home!
print "\e[H"
end

#
# Flushes the STDOUT buffer
#
def flush!
$stdout.flush
end

end

Check it out:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require_relative 'terminal'
include Terminal

# clear the screen
clear_screen!

MAX = 5000
(1..MAX).each {
|i|

# move the cursor to its home, top-left of the screen.
move_to_home!

prog = i / Float( MAX ) * 100

reputs "Counting to #{MAX}..."
reputs "Progress: #{prog}%"
reputs "Current: #{i}"

# make sure that everything is sent out on time
flush!
sleep 0.003
}

Let’s stick a progressbar and ETA in there:
progress_bar.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

#
# Progress bar and ETA methods
#
module ProgressBar

#
# Formats elapsed time to hour:min:sec
#
def format_time( t )
t = t.to_i
sec = t % 60
min = ( t / 60 ) % 60
hour = t / 3600
sprintf( "%02d:%02d:%02d", hour, min, sec )
end

#
# Calculates ETA (Estimated Time of Arrival) based on current progress
# and the start time of the operation.
#
# @param [Float] prog current progress percentage
# @param [Time] start_time start time of the operation
#
# @return [String] ETA: hour:min:sec
#
def eta( prog, start_time )
@last_prog ||= prog
@last_eta ||= "ETA: --:--:--"

if @last_prog != prog
elapsed = Time.now - start_time
eta = elapsed * 100 / prog - elapsed
eta_str = "ETA: " + format_time( eta )
else
eta_str = @last_eta
prog = @last_prog
end

@last_prog = prog
@last_eta = eta_str
end

#
# Returns an ASCII progress bar based on the current progress percentage
#
# @param [Float] prog_percentage progress percentage
#
# @return [String] 70% [=======> ] 100%
#
def progress_bar( prog_percentage )
prog = prog_percentage.ceil < 1 ? 1 : prog_percentage.ceil

bar = '=' * (prog - 1) + '>'

space = 100 - prog
bar += ' ' * space

"#{prog_percentage}% [#{bar}] 100% "
end

end

…and update our simple example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

require_relative 'terminal'
require_relative 'progress_bar'

include Terminal
include ProgressBar

# clear the screen
clear_screen!

start_time = Time.now

MAX = 5000
(1..MAX).each {
|i|

# move the cursor to its home, top-left of the screen.
move_to_home!

prog = i / Float( MAX ) * 100

reputs "Counting to #{MAX}..."
reputs "Progress: #{prog}%"
reputs "Current: #{i}"

reputs
reprint eta( prog, start_time ) + " "
reputs progress_bar( prog.ceil )

# make sure that everything is sent out on time
flush!
sleep 0.003
}

Now that we’ve got our screen manipulation cleaned up and our widgets ready to go it’s time break down the old show_progress() method and update it to utilise our new resources.

progress_helper.rb

require_relative 'terminal'
require_relative 'progress_bar'

#
# Methods to help with progress presentation
#
# Requires the Terminal and ProgressBar modules to work.
#
module ProgressHelper

include Terminal
include ProgressBar

#
# Show an auto refreshing progress screen
#
def show_progress( framework )
# reset the cursor to the home position
move_to_home!

# the true flag updates the timer
stats = framework.stats( true )

# the true flag forces the auditstore to refresh
# instead of returning the cached object
auditstore = framework.audit_store( true )
issues = auditstore.issues

reputs 'Auditing ' + framework.opts.url.to_s
reputs "===================================="

reputs
show_progress_bar( stats, framework.opts.start_datetime )

reputs
show_current_operation( stats )

reputs
show_modules( framework.modules )

reputs
show_plugins( framework.plugins )

reputs
show_statistics( stats )

reputs
show_issues( issues )

reputs
show_output( Arachni::UI::Output.flush_buffer )

flush!
framework.running?
end

#
# Outputs a progress bar and an ETA
#
def show_progress_bar( stats, start_time )
reputs "# Scan progress"
reputs "------------------------------------"
reputs "#{eta( stats[:progress], start_time )}" +
" #{progress_bar( stats[:progress] )}"
end

#
# Outputs the current operation, either:
# * 'Crawling' -- with the additional info of how many pages have been crawled, or
# * 'Currently auditing' -- and the page which is being audited
#
def show_current_operation( stats )
if stats[:current_page] && !stats[:current_page].empty?
reputs "Currently auditing: #{stats[:current_page]}"
else
if stats[:sitemap_size] != 0
s = stats[:sitemap_size] == 1 ? '' : 's'
plural = "#{stats[:sitemap_size]} page#{s} and counting!"
else
plural = ''
end

reputs "Crawling..." + plural
end
end

#
# Shows all loaded modules
#
# @param [Arachni::ModuleManager] modules
#
def show_modules( modules )
reputs "# Loaded modules"
reputs "------------------------------------"
reputs modules.keys.join( ", " )
end

#
# Shows all running plugins
#
# @param [Arachni::PluginManager] plugins
#
def show_plugins( plugins )
reputs "# Running plugins"
reputs "------------------------------------"
reputs ( plugins.busy? ? plugins.job_names.join( ", " ) : 'None' )
end

#
# Shows a few output messages
#
# @param [Array] messages
# @param [Integer] max maximum amount to output
#
def show_output( messages, max = 10 )
reputs "# Scanner output"
reputs "------------------------------------"

# show a few messages at a time just to make the user feels in the loop
max.times {
|i|
if msg_line = messages[i]
next if (msg = msg_line.values.first).empty?

reputs msg
else
reputs '.' * 50
end
}
end

#
# Outputs some basic stats
#
# @param [Hash] stats
#
def show_statistics( stats )
reputs "# Statistics"
reputs "------------------------------------"
reputs "Sent #{stats[:requests]} requests."
reputs "Received and analyzed #{stats[:responses]} responses."
reputs 'In ' + stats[:time]
reputs 'Average: ' + stats[:avg].to_s + ' requests/second.'

end

#
# Outputs a summary of issues
#
# @param [Array] issues
#
def show_issues( issues )
plural = issues.count == 1 ? '' : 's'

reputs "# #{issues.count} issue#{plural} found"
reputs "------------------------------------"

issues.each.with_index {
|issue, i|
puts "[#{i + 1}] #{issue.name} at #{issue.url} in" +
" #{issue.elem} input `#{issue.var}` using #{issue.method}."
}
end

end

Audit monitoring interface

Combining all of these together we get:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

require 'arachni/rpc/server/output'
require 'arachni'

require_relative 'progress_helper'

include ProgressHelper

# get an instance of the options class
opts = Arachni::Options.instance

opts.audit_forms = true

# this is the seed URL
opts.url = 'http://testfire.net/'

# instantiate the framework
framework = Arachni::Framework.new( opts )

# load just the xss module
framework.modules.load( [ 'xss' ] )

# load default plugins
framework.plugins.load_defaults!

# put the scan operation in its own thread
# so that we can do stuff while it's running -- like show progress data..
scan = Thread.new { framework.run }

clear_screen!

# show progress every 0.3 seconds while the scan is running
show_progress( framework ) while scan.alive? && sleep( 0.3 )

# the scan is finished, wait for the thread to return cleanly
scan.join

And now you have a decent output interface to monitor your scripted scans.
In case you can’t be bothered to run the examples and are just reading these articles out of curiosity this is what the output would be:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

Auditing http://testfire.net/
====================================

# Scan progress
------------------------------------
ETA: 00:00:51 20.31% [====================> ] 100%

Currently auditing: http://testfire.net/default.aspx?content=personal_deposit.htm

# Loaded modules
------------------------------------
xss

# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle

# Statistics
------------------------------------
Sent 85 requests.
Received and analyzed 84 responses.
In 00:00:12
Average: 6 requests/second.

# 1 issue found
------------------------------------
[1] Cross-Site Scripting (XSS) at http://testfire.net/search.aspx in form input `txtSearch` using GET.

# Scanner output
------------------------------------
Auditing: [HTTP: 200] http://testfire.net/default.aspx?content=personal_deposit.htm
Harvesting HTTP responses...
Depending on server responsiveness and network conditions this may take a while.
Harvesting HTTP responses...
Depending on server responsiveness and network conditions this may take a while.
..................................................
..................................................
..................................................
..................................................

Building the Console

First off, the Console will have a dependency, readline which can be a bitch to install sometimes which is why we’ll use the pure Ruby implementation rb-readline:

gem install rb-readline

Readline helps with…reading lines, duh!
It does a little more than that though, it can provide us with a smart prompt which can keep history allow shortcuts, interpret escape sequences, provide auto-completion etc.

Try this example to see it in action.

To the point…
Ruby makes this sort of thing a pleasure, Consoles, DSLs etc. feel natural.

Our Console will operate under a single premise: The prompt will simply provide access to the methods of class.
And to be more specific, the Console will initiate the prompt which will provide access to all its declared and public methods.

This will allow us to easily extend our console class to accommodate any specialised needs, like creating a console for Arachni.

Base class

The whole thing is easier than you might think, just a few lines and bam!
console.rb

require 'readline'
require_relative 'terminal'

#
# Basic Console with a readline prompt.
#
# Extend it and it will provide access to the explicitly declared
# and public methods of your class.
#
# This means no private method and no parent methods.
#
class Console

include Terminal

#
# Initializes the Console and starts the prompt.
#
# The Console will use the IO interfaces you specify,
# however Readline will not so your best bet is to override
# $stdout and $stdin in order to redirect the streams.
#
# @param [IO] stdout output stream
# @param [IO] stdin input stream
#
def initialize( stdout = $stdout, stdin = $stdin )
$stdout = @stdout = stdout
$stdin = @stdin = stdin

# combine the methods of self (which may be a child) with the methods
# of the Console
methods = self.class.instance_methods( false ) | Console.instance_methods( false )

# make a list of all available public methods/commands
@cmd_list = methods.map{ |m| m.to_s }.sort

# setup auto-completion params
comp = proc { |s| @cmd_list.grep( /^#{Regexp.escape( s )}/) }

Readline.completion_append_character = " "
Readline.completion_proc = comp

if out = banner
puts out
end

# loop until someone requests an exit
loop do

line = Readline::readline( '> ' )
next if line.empty?
break if line =~ /exit|quit/

Readline::HISTORY.push( line )

# tokenize the line based whitespace
#
# the first token will be the command name and the rest of the tokens
# will be the arguments for that command
#
meth, *args = *line.split( /\s/ )

begin

# make sure that the command exists
if !@cmd_list.include?( meth )
raise 'Unknown command: ' + meth + "\n" +
"Valid commands: " + commands
end

# extract an unbound method so that we can check for arity
meth = method( meth.to_sym )

# if arity doesn't match the provided arguments pass them as an array
res = ( meth.arity != args.count ) ? meth.call( args ) : meth.call( *args )

# return the result of the command
if res.is_a? String
puts res
else
puts res.inspect if res
end
rescue Exception => e
puts e.to_s
puts e.backtrace.join( "\n" )
end

end
end

# to be overriden by children
def banner
end

# add this empty exit method just so that it'll auto-complete
def exit; end
alias :quit :exit

# returns all available commands
def commands
@cmd_list.join( ' ' )
end

# loads and evaluates a console script
def source( file_path )
instance_eval( IO.read( file_path.to_s ) )
end

# clears the screen
def clear
clear_screen!
move_to_home!
end

private

def force_string( arg )
return arg.join( ' ' ) if arg.is_a?( Array )
arg.to_s
end

def puts( str = '' )
print str
print "\n"
end

def print( str = '' )
@stdout.print str
end

end

Console.new if __FILE__ == $0

Let’s take it for a spin, shall we?

1
2
3
4
5
6

$ ruby console.rb
>
banner clear commands exit quit source
> commands
banner clear commands exit quit source
> exit

Doesn’t do much, yet…let’s extend it to do something more interesting.

Console demo – The Jukebox

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

require 'arachni/rpc/server/output'
require 'arachni'

require_relative 'console'

class Jukebox < Console

VERSION = '0.1'

SONGS = [
['Dream Theater', 'A Change of Seasons' ],
['Iron Maiden', 'Paschendale' ],
['Pain of Salvation', 'Fandango' ],
['Steve Vai', 'For the love of god' ],
['Joe Satriani', 'Andalusia' ]
]

def banner
"#{self.class} v#{VERSION}"
end

def list
SONGS.map { |song| format_song( song ) }.join( "\n" )
end

def play( title )
title = force_string( title ).downcase
song = find_by_title( title )

raise 'Song does not exist.' if !song

@now_playing = song
now_playing
end

def now_playing
puts "Now playing: #{format_song( @now_playing )}"
end

private

def format_song( song )
"#{song[0]} - #{song[1]}"
end

def find_by_title( title )
SONGS.each{
|song|
return song if song[1].downcase == title.downcase
}
nil
end

end

Jukebox.new if __FILE__ == $0

And run it:

1
2
3
4
5
6
7
8
9
10
11
12

Jukebox v0.1
> list
Dream Theater - A Change of Seasons
Iron Maiden - Paschendale
Pain of Salvation - Fandango
Steve Vai - For the love of god
Joe Satriani - Andalusia
> play andalusia
Now playing: Joe Satriani - Andalusia
> now_playing
Now playing: Joe Satriani - Andalusia
> exit

There’s one feature we haven’t touched yet though, the source command, something familiar to *nix users.
That command will allow us to load and run Console scripts.
These scripts are not limited by the syntax restrictions of the prompt and are evaluated under the namespace of the console.
So you can write vanilla Ruby code and take advantage of the Console’s functionality at the same time.

shuffle.jukescript

1
2
3
4

SONGS.shuffle.each do
|song|
play song[1]
end

1
2
3
4
5
6
7
8

Jukebox v0.1
> source shuffle.jukescript
Now playing: Pain of Salvation - Fandango
Now playing: Dream Theater - A Change of Seasons
Now playing: Joe Satriani - Andalusia
Now playing: Steve Vai - For the love of god
Now playing: Iron Maiden - Paschendale
>

The Arachni Console

Since you’re already somewhat familiar with the framework and the Console is quite simple I won’t explain everything beat-by-beat this time.
Instead, I’m gonna just paste the ArachniConsole code and play with it a bit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312

require 'arachni/rpc/server/output'
require 'arachni'

require_relative 'console'
require_relative 'progress_helper'

class Arachni::Console < Console

include ProgressHelper

VERSION = '0.1'

def initialize
reset
super
end

# returns shows a banner
def banner
"Arachni - Web Application Security Scanner Framework v#{@framework.version}\n" +
"Console v#{VERSION}"
end

#
# executes a shell/OS command
#
# sh ls -la *
#
def sh( cmd )
`#{force_string( cmd )}`
end

#
# performs a demo scan
#
# demo
# demo silent
#
def demo( opt = nil )
set 'url', 'http://testfire.net/'
# set 'link_count_limit', '1'

load 'modules', '*'
load 'report', 'json'

audit 'forms'
audit 'links'
audit 'cookies'

run opt
end

#
# sets exclude patterns
#
# exclude test \.pdf$ \.exe$
#
def exclude( regexps )
@opts.exclude = [regexps].flatten.map { |regexp| Regexp.new( regexp ) }
end

#
# sets include patterns
#
# include test \.html$
#
def include( regexps )
@opts.include = [regexps].flatten.map { |regexp| Regexp.new( regexp ) }
end

# resets all settings
def reset
@opts = Arachni::Options.instance
@opts.reset!

# instantiate the framework
@framework = Arachni::Framework.new( @opts )

@framework.plugins.load_defaults!
'Done!'
end

#
# assigns val to opt
#
# set url http://testfire.net
# set link_count_limit 10
#
def set( opt, val )
@opts.send( "#{opt}=".to_sym, cast( opt, val ) )
end

#
# unsets an option
#
# unset url
# unset link_count_limit
#
def unset( opt )
set( opt, nil )
end

#
# shows an option
#
# show url
# show link_count_limit
#
def show( opt )
raise 'Expects argument.' if opt.empty?
@opts.send( "#{opt}".to_sym )
end

#
# imports an AFR report and sets it as the default audistore
#
# import my_report.afr
#
def import( path_to_afr )
path_to_afr = path_to_afr.join( ' ' ) if path_to_afr.is_a? Array

@auditstore = Arachni::AuditStore.load( path_to_afr )

"Loaded audit for #{@auditstore.options['url']} performed on" +
" #{@auditstore.start_datetime} with #{@auditstore.issues.count} issues."
end

#
# exports the auditstore in 'type' format in file 'outfile'
#
# export html my_report.html
# export json my_json_report.json
# [...]
#
def export( type, outfile )
auditstore = @framework.audit_store if !( auditstore = @auditstore )

@opts.reports ||= {}
@opts.reports[type] = {}

@opts.reports[type]['outfile'] = outfile if outfile

@framework.reports.run_one( type, auditstore )
end

#
# load components by type and name
#
# load modules xss*,sqli*
# load plugins defaults/*
# load report json,txt
#
def load( type, names )
names = force_string( names ).split( ',' )
case type
when 'module', 'modules'
@framework.modules.load( names )

when 'plugin', 'plugins', 'plug-ins', 'plug-in'
@framework.plugins.load( names )

when 'report', 'reports'
@opts.reports ||= {}
reps = @framework.reports.load( names )
reps.each { |rep| @opts.reports[rep] = {} }
else
raise 'Unknown component type: ' + type.to_s
end
end

#
# sets elements to audit
#
# audit links
# audit forms
# audit cookies
# audit headers
#
def audit( types )
valid = [ 'links', 'forms', 'cookies', 'headers' ]

[types].flatten.each {
|type|

if !valid.include?( type )
raise 'Invalid element type: ' + type.to_s + "\n" +
'Valid types: ' + valid.join( ", " )
break
end

set 'audit_' + type, 'true'
}
end

#
# runs the audit
#
# run
# run silent
#
def run( *args )
@job = run_in_thread

return if silent?( args )

progress
clear_screen!
end

#
# attach to running scan
#
def attach
progress
end

#
# show progress of running scan
#
# progress
# progress glimpse
#
def progress( *args )
raise 'No scan running.' if !running?

clear_screen!

if glimpse?( args )
show_progress( @framework )
return
else
sleep( 0.3 ) while show_progress( @framework )
show_progress( @framework )

@job.join

'Done!'
end
end

# pauses running scan
def pause
@framework.pause!
'Will pause as soon as possible...'
end

# resumes running scan
def resume
@framework.resume!
'Resuming!'
end

# kills/aborts running scan
def kill
@framework.clean_up!
@job.exit
end
alias :abort :kill

private

def running?
@job && @job.alive?
end

def glimpse?( args )
includes_modifier?( args, :glimpse )
end

def silent?( args )
includes_modifier?( args, :silent )
end

def includes_modifier?( args, modifier )
args.flatten!
args.include?( modifier ) || args.include?( modifier.to_s )
end

def run_in_thread
Thread.new { @framework.run }
end

def cast( opt, val )
default = @opts.instance_variable_get( "@#{opt}" )

case default
when Fixnum, Integer
val.to_i

when Float
val.to_f

when String, URI
val.to_s

when TrueClass, FalseClass
val.to_s.downcase == 'true' ? true : false

when Array
val.to_a

when Hash
val.to_h

else
val.to_s
end
end

end

Arachni::Console.new if __FILE__ == $0

Let’s jump in!

1
2
3

Arachni - Web Application Security Scanner Framework v0.4
Console v0.1
> demo

Ah, that’s a familiar sight, it’s the progress screen from a couple of chapters ago.

To detach and return to the prompt hit Ctrl-C, don’t worry about the error messages the scan won’t die.
Say you want to take a glimpse of the running scan while detached:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

> progress glimpse
Auditing http://testfire.net/
====================================

# Scan progress
------------------------------------
ETA: 00:14:13 13.07% [=============> ] 100%

Currently auditing: http://testfire.net/default.aspx?content=inside.htm

# Loaded modules
------------------------------------
interesting_responses, common_files, mixed_resource, xst, http_put, webdav, directory_listing, allowed_methods, htaccess_limit, ssn, private_ip, emails, credit_card, cvs_svn_users, captcha, html_objects, unencrypted_password_forms, backdoors, backup_files, common_directories, trainer, os_cmd_injection, sqli, xss_script_tag, sqli_blind_rdiff, path_traversal, xss_event, xss_uri, sqli_blind_timing, code_injection, rfi, xss_tag, response_splitting, csrf, os_cmd_injection_timing, ldapi, code_injection_timing, xss_path, xpath, unvalidated_redirect, xss

# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle

# Statistics
------------------------------------
Sent 5873 requests.
Received and analyzed 5725 responses.
In 00:01:29
Average: 44 requests/second.

# 7 issues found
------------------------------------
[1] Cross-Site Scripting (XSS) at http://testfire.net/search.aspx in form input `txtSearch` using GET.
[2] Path Traversal at http://testfire.net/default.aspx in link input `content` using GET.
[3] Allowed HTTP methods at HTTP://testfire.net in server input `` using OPTIONS.
[4] Interesting server response. at http://testfire.net/admin/ in server input `` using GET.
[5] Interesting server response. at http://testfire.net/default.aspx in server input `` using GET.
[6] Found an HTML object. at http://testfire.net/default.aspx in body input `` using GET.
[7] Interesting server response. at http://testfire.net/default.aspx/%3Carachni_xss_uri_7b063a2cf51c145d46e218f8b1697adb21339ba91fd57912bd5057b8d485a684 in server input `` using GET.

# Scanner output
------------------------------------
PathTraversal: Analyzing response #5726...
PathTraversal: Analyzing response #5727...
PathTraversal: Analyzing response #5711...
PathTraversal: Analyzing response #5728...
PathTraversal: Analyzing response #5714...
PathTraversal: Analyzing response #5729...
..................................................
..................................................
..................................................
..................................................
>

And say you want to attach to the running scan and monitor its progress:

> attach

Hint: progress without the glimpse argument would do the same…

Now lets say you want to get a preliminary report while the scan is running:

1
2
3
4
5

> export html myreport.html
> sh ls *.html
myreport.html

> sh firefox myreport.html

Aha! The report contained what we were looking for, kill the scan!

1
2
3
4

> kill
#
> kill
#

And if the fucker won’t die you can skip the clean up process by:

1 2	> kill force #

And if you want to import a previous report:

1
2
3
4
5
6
7
8
9
10

> sh ls *.afr
2011-12-02 01.26.32 +0200.afr

> import 2011-12-02 01.26.32 +0200.afr
Loaded audit for http://testfire.net/ performed on Fri Dec 2 01:25:50 2011 with 7 issues.
> export json report.json
> sh ls *.json
report.json

>

And of course…scripting:
testfire

1
2
3
4
5
6
7
8
9
10
11
12

set 'url', 'http://testfire.net/'

load 'modules', '*'
load 'plugins', 'defaults/*'
load 'report', 'json'

audit 'forms'
audit 'links'
audit 'cookies'

run :silent
progress :glimpse

Output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

Arachni - Web Application Security Scanner Framework v0.4
Console v0.1
> source testfire

Auditing http://testfire.net/
====================================

# Scan progress
------------------------------------
ETA: --:--:-- 0.0% [> ] 100%

Crawling...

# Loaded modules
------------------------------------
interesting_responses, common_files, mixed_resource, xst, http_put, webdav, directory_listing, allowed_methods, htaccess_limit, ssn, private_ip, emails, credit_card, cvs_svn_users, captcha, html_objects, unencrypted_password_forms, backdoors, backup_files, common_directories, trainer, os_cmd_injection, sqli, xss_script_tag, sqli_blind_rdiff, path_traversal, xss_event, xss_uri, sqli_blind_timing, code_injection, rfi, xss_tag, response_splitting, csrf, os_cmd_injection_timing, ldapi, code_injection_timing, xss_path, xpath, unvalidated_redirect, xss

# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle

# Statistics
------------------------------------
Sent 0 requests.
Received and analyzed 0 responses.
In 00:00:00
Average: 0 requests/second.

# 0 issues found
------------------------------------

# Scanner output
------------------------------------
Waiting for plugins to settle...
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
>

Quite a long post this one but it was well worth the effort…barely, heh.
Anyways, that’s all for now, the next part will be a lot more hardcore and will demonstrate a highly specialised audit that will be customisable down to the individual element.

See ya then…

Programatically scanning using Arachni (Part 3)

Zapotek — Thu, 01 Dec 2011 00:57:09 +0000

Managing components

Time to leave the framework for now and get back to some basics since after seeing things as a whole you’ll better appreciate these important details.

All components (modules, reports, plugins and the unappreciated path extractors) are handled by the ComponentManager.
The manager’s job is simple and can be described as:

Given the path to a library and the namespace of the components it allows manipulation and organization of said components.

For example, the library of the Arachni modules is /modules and all modules live under the namespace of the Arachni::Modules module.
Unless you want to create your own components you won’t have to deal with that manager directly, you’ll only deal with its children:

ModuleManager
ReportManager
PluginManager

(The path extractors don’t get their own manager because they’re quite simple in nature.)

These managers extend the ComponentManager and because the ComponentManager extends the Hash class working with them will feel quite natural.
Finally, the Framework provides you with access to all the discussed managers and they, in turn, provide you with access to the components they manage.

By convention, all components have the following methods:

prepare — Optional, allows them to prepare their environment
run — Mandatory, executes the main payload.
clean_up — Optional, gives them a chance to clean up after themselves.
self.info — Mandatory, returns general info like version number, name, author, etc.. Actual format depends on component type.

Let’s see how these manager work and what they can do.

Component manager by example

Let’s use the ModuleManager as a working ComponentManager to discover what it can actually do.
The following demonstrated functionality is shared amongst all component managers and is not ModuleManagement specific.

require 'arachni/ui/cli/output'
require 'arachni'

# get the module manager
modules = Arachni::Framework.new( Arachni::Options.instance ).modules

# let's see what it holds
ap modules
# => {}
# so pretty much nothing...

# however, it allows us to do this:
ap modules['xss'].info

# the module manager will look for the 'xss' module and load it for you
# if it't not already loaded, see:
ap modules
# =>
# {
# "xss" => Arachni::Modules::XSS < Arachni::Module::Base
# }

# see all available modules by name
ap modules.available
# =>
# [
# [ 0] "interesting_responses",
# [ 1] "common_files",
# [ 2] "mixed_resource",
# [ 3] "xst",
# [ 4] "http_put",
# [ 5] "webdav",
# [ 6] "directory_listing",
# [ 7] "allowed_methods",
# [...and many more...]

# load all available modules using the wildcard
modules.load( [ '*' ] )

# let's see what the manager holds now
ap modules
# =>
# {
# "xss" => Arachni::Modules::XSS < Arachni::Module::Base,
# "interesting_responses" => Arachni::Modules::InterestingResponses < Arachni::Module::Base,
# "common_files" => Arachni::Modules::CommonFiles < Arachni::Module::Base,
# "mixed_resource" => Arachni::Modules::MixedResource < Arachni::Module::Base,
# "xst" => Arachni::Modules::XST < Arachni::Module::Base,
# "http_put" => Arachni::Modules::HTTP_PUT < Arachni::Module::Base,
# "webdav" => Arachni::Modules::WebDav < Arachni::Module::Base,
# [...etc...]

# let's parse module names including wildcards and modifiers
# to see to which modules they actually correspond
ap modules.parse( [ '*', '-xss_*'] )
# this one returns all modules by name while excluding the XSS ones

# obvious what this does...
ap modules.wilcard_to_names( 'xss_*' )
# =>
# [
# [0] "xss_script_tag",
# [1] "xss_event",
# [2] "xss_uri",
# [3] "xss_tag",
# [4] "xss_path",
# [5] "xss"
# ]

# let's say you want to find the path of a module
ap modules.name_to_path( 'xss' )
# => "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/audit/xss.rb"

# and the exact opposite:
ap modules.path_to_name( modules.name_to_path( 'xss' ) )
# => "xss"

# and say that you want to find the paths of all modules
ap modules.paths
# =>
# [
# [ 0] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/interesting_responses.rb",
# [ 1] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/common_files.rb",
# [ 2] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/mixed_resource.rb",
# [ 3] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/xst.rb",
# [ 4] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/http_put.rb",
# [ 5] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/webdav.rb",
# [ 6] "/home/zapotek/.rvm/gems/ruby-1.9.3-p0/gems/arachni-0.4/modules/recon/directory_listing.rb",
# [...]

The ModuleManager

Let’s see the extra stuff that the module manager let’s us do in a live scenario:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

The PluginManager

The PluginManager is more concerned with job scheduling.
It runs each plugin in its own thread and provides methods to manipulate the threads.

require 'arachni/ui/cli/output'
require 'arachni'

Arachni::UI::Output.mute!

opts = Arachni::Options.instance
opts.audit_forms = true
opts.url = 'http://testfire.net/'
opts.link_count_limit = 1

framework = Arachni::Framework.new( opts )
framework.modules.load( [ 'xss' ] )

# get the plugin manager...
plugins = framework.plugins

#
# There are some plugins that gather very useful information while
# introducing very little or no overhead at all.
#
# These are put under the /plugins/defaults directory and can be loaded like this:
plugins.load_defaults!

# which is the same as:
# plugins.load( Arachni::Plugin::Manager::DEFAULT )

# that's a simple wildcard
# ap Arachni::Plugin::Manager::DEFAULT
# =>
# [
# [0] "defaults/*"
# ]

# let's see what we loaded
ap plugins
# =>
# {
# "healthmap" => Arachni::Plugins::HealthMap < Arachni::Plugin::Base,
# "timing_attacks" => Arachni::Plugins::TimingAttacks < Arachni::Plugin::Base,
# "discovery" => Arachni::Plugins::Discovery < Arachni::Plugin::Base,
# "manual_verification" => Arachni::Plugins::ManualVerification < Arachni::Plugin::Base,
# "uniformity" => Arachni::Plugins::Uniformity < Arachni::Plugin::Base,
# "content_types" => Arachni::Plugins::ContentTypes < Arachni::Plugin::Base,
# "autothrottle" => Arachni::Plugins::AutoThrottle < Arachni::Plugin::Base
# }

# some plugins depend on certain gems and because gems can be platform
# dependent they may not be available for some platforms
#
# in order to streamline the process plugins can declare any gem dependencies
# they may have and this is how you perform the check:
ap plugins.sane_env?( plugins['healthmap'] )
# => true
#
# If the environment isn't sane you'll get a hash with some details.
# In such a case plugins.run will raise an exception and print some details.
#

# ap plugins.sane_env?( plugins['libnotify'] )
# =>
# {
# :gem_errors => [
# [0] "libnotify"
# ]
# }
#
# that means that the plugin can't run without the "libnotify" gem.
#

#
# this is how you get the options of a plugin
#
# ap plugins['content_types'].info[:options]
# =>
# [
# [0] #
# ]

# let's set some options now, shall we?
opts.plugins = {

# this plugin is meant to match only interesting content-types
'content_types' => {

# let's undo that, heh...
'exclude' => ''
}
}

# put this in its own thread so that we can inspect the plugins in the meantime
scan = Thread.new {
#
# This will run all loaded plugins, each in its own thread:
# plugins.run
#
# however you shouldn't need to run this manually because
# framework.run takes care of running the plugins and setting certain
# framework parameters which are essential to them
#
framework.run

#
# This waits for plugins to finish and as soon as each one finishes
# it's removed from the job list.
#
# It's also handled by framework.run...
#
# plugins.block!
}

# true if there are any running plugins
ap plugins.busy?
# => true

# returns the names of the running plugins
ap plugins.job_names
# =>
# [
# [0] "healthmap",
# [1] "timing_attacks",
# [2] "discovery",
# [3] "manual_verification",
# [4] "uniformity",
# [5] "content_types",
# [6] "autothrottle"
# ]

# returns the threads of the plugins
ap plugins.jobs
# =>
# [
# [0] #,
# [1] #,
# [2] #,
# [3] #,
# [4] #,
# [5] #,
# [6] #
# ]

# get's the thread of a plugin by name
ap plugins.get( 'healthmap' )
# => #

# that's an easy one...
ap plugins.kill( 'healthmap' )
# => #

# the first one (the healthmap) will have died...
ap plugins.jobs
# => [
# [0] #,
# [1] #,
# [2] #,
# [3] #,
# [4] #,
# [5] #,
# [6] #
# ]

# you'll see the list getting smaller once the scan finishes
plugin_inspector = Thread.new {
while scan.alive?
puts 'Rrunning plugins: ' +
( plugins.busy? ? plugins.job_names.join( ", " ) : 'None' )
sleep( 0.3 )
end
}

scan.join
plugin_inspector.join

# and this is how you get any results the plugins might have stored
ap framework.audit_store.plugins

The ReportManager

The report manager is boring, the only functionality it provides is a couple of methods for running reports but it can be helpful under certain circumstances.
The framework has the tendency to run the AFR report whether you asked for it or not and there’s a very good reason for that.
That’s the yardstick, the default report, the main audit DB; once you have that report you can convert the scan results to the other formats.

But you may not want to allow the framework to run it, or you may want to run the reports yourself or whatever.
Let’s get to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

Goodie, we got some technicalities out of the way in this post.
The next part will feature a nice little interface that will allow us to monitor a few aspects of the scan including progress, status messages and general statistics.
See you then.