When I started development on the Arachni high-performance grid my focus was on the audit part, i.e. find a way to distribute the audit of batches of individual elements across multiple nodes and avoid duplication of effort amongst them.
It was a bit tricky to get right but it turned out to be quite do-able and worthwhile.
However, the crawl was done the old fashioned way, the master instance would crawl the targeted website and once completed it would then analyze all the pages it found and spread the workload.
I always intended to try out my hand on something similar but aimed towards the crawling process but it wasn’t a high priority.
But, as you can see from my last post, I did sort of figure it out, although I hadn’t had a chance to implement it until now.
This is tricky to do because there’s no way of knowing the workload before hand as it is basically a freaking labyrinth and precious information (new paths) can be hidden behind walls and walls of crap.
On the other hand, since when running Arachni in HPG mode you already have a few nodes up and running in the first place, why not utilize them a bit more — even if it turns out to be only slightly faster than a single crawler.
With that in mind, I yesterday started to implement that sort of a crawler, and here it is.
Its sole existence is that of a toy, a fun experiment, and not as a stable system. I may, in the future, put some more effort into it but my main reason for doing this is to explore this idea and eventually port it over to Arachni.
If you find this interesting, want to help out in researching or have any sort of feedback or just want to get in touch don’t hesitate to do so.
Cheers,
Tasos L.
Posted in: Arachni, Open Source, Programming, Projects, Ruby, Security, Web Application
Tags: asynchronous, crawl, distributed, grid, high performance, ruby, spider, workload
This one had been bugging me since I first started work on the HPG.
The gain you get from distributed computing is directly related to how efficient the workload distribution is — which makes sense.
The crawling process though doesn’t consist of a workload per se but rather looks for the workload.
Also, the difficulty of the crawl doesn’t lie in parsing or following the paths but actually finding the paths, this is because new paths are hidden behind old ones and as you progress new paths become sparser and sparser.
So the more work you do the less productive you become — these are grim prospects.
So here’s our problem, how the hell do you distribute something when you don’t know:
- where it is
- what it is
- how big it is
Truthfully, in a single node setup this can be done quite easily and there are lots of ways to go about it
A basic crawling algorithm is actually one of the simplest around, it only has one rule: Follow every path but only once — when there are no more new paths you’re done.
In the most basic of implementations what you’d need to do this is a look-up table to help you keep track of the paths you’ve already followed in order not to go over them again.
Or you can amend the previous model by going the multi-threaded way and put the new paths in a queue, have workers pop paths from the queue, follow them and then report the new paths they find back to their coordinator for de-duplication and then put everything that passes filtering back to the queue and so the story goes…
When you do this on a single machine these approaches are good enough (and when you use async requests for the first/simplest approach it becomes more efficient than the latter one as well).
Thing is, these work well because you have the benefit of a multi Gbps bandwidth and close-to-zero latency pipe, your Front-Side Bus (or whatever computers have nowadays, haven’t kept up with h/w design).
So you can check if something is in the lookup table in something close to 0s, actually in Ruby the look-up time of a Set with 1,000,000 URLs in it is 9.735e-06 (0.000009735) seconds on my machine.
Which is effectively no time at all, you spend 0.000009735 seconds waiting for a decision before following each URL — ooooh scary.
However, when you need to do this over the network these response times take a dive — off a cliff.
You see, when distributing work you play with gain/cost ratios; if the ratio is good you go ahead, if not you go back to the drawing board.
Such a naive implementation deserves picking up the multi-colored markers again because:
Let’s say every worker communicates via an RPC protocol and the master worker maintains the look-up table and the work Queue.
Assuming that an RPC call costs about the same as an HTTP request and 25% of the paths contained in most pages are identical (nav menu, css, js, images, links to new blog posts etc.) this means that each follow operation will cost, per work unit:
1 RPC call to pop a path from the Queue +
- 25% of the amount of links found by following the path * 1 RPC — for paths that are common and have already been visited from the get go
- 75% of the amount of links found by following the path * 1 RPC call + 1 HTTP request — for new paths which aren’t in the lookup table and must be visited
So you’ll be spending most of your time waiting for permission rather than doing actual work and suddenly the cost of doing the work doubles.
There has to be a better way…
Annoyingly, it took me a couple of days to figure this out and it turned out what I needed was a relaxing shower and the answer come to me on its own — gooood answer *pat**pat*.
In all honesty, I had bits and pieces of the answer from the begging and I knew that the final algorithm would have to be a composite of models — a piece from the producer-consumer there, a bit of a policy-enforcer here sprinkled with some delegation across the board — but the problem was putting them in the right order to form a unified model that would:
- Avoid any sort of blocking (no look-ups or waiting for decisions)
- Automate load balancing
- Prevent crawling redundant URLs (more than one worker following the same path)
Tricky stuff right?
And here’s what I came up with:
- The master scopes out the place (follows 10 paths or so) and deduces the webapp structure — it will, most certainly, be incomplete but it doesn’t matter as we just want some seeds to get us going.
- The master creates a per directory policy and assigns dirs to workers AND sends that policy to them as well.
- Workers perform the crawl as usual but also implement that policy for URLs that don’t match their own policy rules i.e. send URLs that are out of their scope to the appropriate peer and let him handle it — the peer will ignore it if he has already visited it or put it in his queue.
- If no policy matches a URL then it is sent back to the master; the master creates a new policy(ies), stores the work in a Queue and then sends an announcement to the workers (“There’s some work up for grabs!” ).
- Busy workers ignore it; idling workers try to pull it and the work is assigned first-come/first-serve along with the updated policy.
- Go to 3
If at any point a worker becomes idle he sends the paths he has discovered back to the master for storage/further processing/whatever and tries to pull some new work.
Also, the master will be a worker too — why waste a node, right?
Let’s go back to our list of requirements and see how we did:
- Avoid any sort of blocking (no look-ups or waiting for decisions) — ✓ Instead of waiting for permission to do something, we delegate to the appropriate authority and forget about it
- Automate load balancing — ✓ Workers pull work when they are good and ready
- Prevent crawling redundant URLs (no more than one worker following the same path) — ✓ A local look-up table and item #1 take care of that quite nicely
In addition, the policy can use any sort of criteria (directories were just an example), which means that we can achieve very granular distribution if we are a bit clever with it.
I think that this serves as a decent starting point, there’s still the issue of how to efficiently group the new URLs that are fed back to the master (because they don’t match any of the initial policy rules) but I’ll have to see this working under real world conditions in order to get a better feel for it first.
I’ve got some stuff in mind, we’ll see…
Please do comment if you have a suggestion of have spotted a fault somewhere.
– Tasos
Posted in: Arachni, Projects
Tags: arachni, cluster, crawl, distribution, grid
Yes, it’s true…
As of now the code in the experimental branch has been converted to use the Apache License Version 2.0.
If you’re interested in why this happened here’s the deal:
There are currently a few companies that use Arachni internally and a few others that actually provide SaaS security services using Arachni’s distributed features.
Thing is though, a lot of companies can’t touch GPL code (not that I blame them) which isn’t good for them nor Arachni as neither of us gets what he wants.
It makes sense, surely, but it was about a month ago that it really clicked as I were reading the comments of a Slashdot article.
Lots of people were agreeing on the same subject, the money-men don’t like GPL which kind of sucked for the project.
At this point I started researching alternative licenses and started asking around a bit.
As fate would have it — although more due to the increasing userbase I guess — a few people told me that the GPL was a deal-braker for them and I even had one guy tell me that he couldn’t include Arachni in his book because of it (I’ll spare him his blushes and not say his name).
Now that the project is gaining some momentum these technicalities become more and more important.
So after a bit of research I settled on Apache License 2.0 mainly because of its trademark and patent grants (because who the hell wants to deal with that bureaucratic crap?) and the requirement to redistribute a visible copy of the original work’s NOTICE file (if it includes one) which is nice since hard work must be properly credited.
You know, you can use my work for free (and I hope you do 
) but mention that “this product contains some code from that Arachni thingy written by this bloke with the funny name”.
So that’s the reason, I’m hoping that a more permissive license will increase adoption and make everybody’s life easier.
Posted in: Arachni, Open Source, Projects, Security, Web Application
Tags: apache, apache license 2.0, arachni, asf, gpl, gplv2
Hi guys,
A couple of days ago I proudly released v0.4 and, as luck would have it, I later had to swallow some of that pride due to a couple of intermittent bugs that I hadn’t spotted.
Well, worry no more as I’m writing this post to announce a rush hotfix version of Arachni, v0.4.0.2.
If you installed the previous version via “gem install” or have downloaded the previous Cygwin package then all you need to do is issue:
If not, you can find links and instructions for all available packages at: http://arachni-scanner.com/latest
Enjoy
Posted in: Arachni, Open Source, Programming, Projects, Releases, Ruby
Tags: arachni, audit, automated, crawler, framework, open source, plugins, ruby, scanner, security, vulnerability, vulnerability scanner, web, web application, webapp, xss
Yes, yes…the time has finally come and there are boons for everyone.
This release features the most impressive ChangeLog yet and the first (and coolest) thing in this long list is the brand new High Performance Grid implementation — which has been discussed extensively in the past.
Let’s review the big points.
Ruby’s XMLRPC has been ditched (as initially discussed in these two [1, 2] posts) in favor of Arachni-RPC.
Arachni-RPC is lightweight, simple and fast which makes it ideal for large Grid deployments and makes it easy for 3rd parties to interoperate with Arachni’s servers.
Notice: If you were using the old XMLRPC interface please update your code to use the new RPC API.
I’ve been talking about this one so much that I’ve actually grown a bit sick of it — joking aside though this is one of Arachni’s most important features.
It allows you to connect multiple nodes into a Grid and use them to perform lightning-fast scans.
This is due to the way Arachni distributes the workload, which is finely grained down to individual page elements to ensure fair and optimal distribution; because workload distribution is so fluid it effectively becomes a sort of bandwidth and CPU aggregation.
To put this in simple(-istic) terms:
If you have 2 Amazon instances and you need to scan one site, by utilising the HPG you’ll be able to cut the scan time down to approximately half of what it would take by using a single node (plus the initial crawl time).
And if you have a huge site you can use 50 nodes and so the story goes…
This feature was an imaginary, almost unattainable, milestone back when I added the initial client/server implementation and I didn’t really think that I’d ever be able to make it happen.
Luckily, I was wrong and I’m proud to present you with the first Open Source High Performance Grid web application security scanner!
(By the way, does anyone know of a commercial scanner that can do this?)
Notice: With the WebUI’s updated AutoDeploy add-on you’ll be able to go into World domination mode by performing point and click Grid deployments!
Another notice: Use responsibly, don’t DDoS people.
Yet another notice: It’s still considered experimental so let me know if you come across a bug. 
The WebUI now contains a few context-sensitive help dialogs to help out the newcomers and it has been updated to use the Thin webserver to send responses asynchronously in order to increase performance and feel “snappier”.
It also supports HTTP basic auth just in case you want some simple password protection and has been updated to provide access to the brand new HPG goodies.
Spider improvements
There was a bug with redirections that prevented the spider from achieving optimal coverage which has now been resolved.
More than that, the scope of the crawl can now be either extended or restricted by supplying newline-separated lists of URLs which should help you import 3rd party sitemaps.
Plugins
The plugin API has been extended in order to allow plugins to let the framework know if they can be distributed across HPG Instances and, if so, how to merge their results for the final report.
Another big (although invisible to the end-user) change is the conversion of all meta-modules to full-fledged plugins to simplify management and Grid distribution.
And these new plugins have been added:
- ReScan — It uses the AFR report of a previous scan to extract the sitemap in order to avoid a redundant crawl.
- BeepNotify — Beeps when the scan finishes.
- LibNotify — Uses the libnotify library to send notifications for each discovered issue and a summary at the end of the scan.
- EmailNotify — Sends a notification (and optionally a report) over SMTP at the end of the scan.
- Manual verification — Flags issues that require manual verification as untrusted in order to reduce the signal-to-noise ratio.
- Resolver — Resolves vulnerable hostnames to IP addresses.
Modules
I’ve got both good and bad news for this….
In an attempt to cleanup and optimise pattern matching in v0.3 I inadvertently broke some aspects of it which crippled the XSS (xss), SQL injection (sqli) and Path Traversal (path_traversal) modules –I sincerely apologise, mea culpa.
The good news is that I’ve made things right, cleaned up the API and the existing modules and improved their accuracy.
Reports
The HTML report has waved goodbye to Highcharts due to licensing reasons and now uses jqPlot for all its charting and graphing needs.
I’ve also removed the “report false-positive” button since a part of that process required RSA encryption which for some reason caused segfaults on Mac OSX.
Good news is that the HTML reports will be significantly smaller in size from now on.
Moreover, the following new report formats have been added:
- JSON — Exports the audit results as a JSON serialized Hash.
- Marshal — Exports the audit results as a Marshal serialized Hash.
- YAML — Exports the audit results as a YAML serialized Hash.
Cygwin package for Windows
About time indeed, Windows users can now enjoy Arachni’s features — albeit via a preconfigured Cygwin environment.
The important point is that you no longer have to manually hassle to install Arachni via MinGW or Cygwin yourselves or use a VM and what have you…
Simply download and run the self-extracting archive, double click the “Cygwin” batch file and lo and behold: you’ve got a bash shell ready to execute Arachni’s scripts.
Unfortunately, there’s a performance penalty involved when running Arachni in Cygwin but until I port it to run natively on Windows it’ll have to do.
Before I forget, the Wiki has been cleaned up and brought up to date so if you need to go through the documentation that should be your first stop.
Go ahead and enjoy this new release everyone, download links and instructions here: http://arachni.segfault.gr/latest.
Cheers,
Tasos L.
Posted in: Arachni, Open Source, Programming, Projects, Releases, Ruby, Security, Web Application
Tags: arachni, audit, automated, crawler, framework, grid, module, modules, open source, plugins, release, report, ruby, scanner, security, server, spider, update, v0.4, vulnerability, vulnerability scanner, web, web application, webapp, xmlrpc, xss
I’ve caught a bug it seems and because I can’t just sit on my ass all day I figured why not play around with my latest toy.
I’ve updated the code to make the process more streamlined and allow for fuzzing (or at least altering) some possible input vectors.
Things will be very very simple for now as I’m merely trying to demo that with appropriate effort invested this can become a viable solution — eventually.
To the point, I’ll showcase a DOM XSS vulnerability that will take place purely on the client-side.
Unfortunately, there are a lot of DOM interfaces/vectors that I haven’t yet implemented so let’s stick to one I have — navigator.userAgent.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
| require_relative 'init'
html = <<EOHTML
<html>
<head>
<title>My title!</title>
</head>
<body>
<div>
<script type="text/javascript">
document.write( navigator.userAgent );
</script>
</div>
</body>
</html>
EOHTML
#
# the second param sets 'dont_eval_js' to true
#
# we want to do that ourselves later on after we've prepared the vectors
# (navigator.userAgent in this case)
#
window = DOM::Window.new( html, true )
#
# we'll inject this fictional tag and look for it in the DOM structure later on
#
# if found, then we have an XSS vuln
#
seed_tag = 'myinjectedtag'
# our XSS vector
window.navigator.userAgent = "<#{seed_tag}>blah blah blah</#{seed_tag}"
# this will show the HTML as is (Ref. #1)
# puts window.document.to_html
# puts '-' * 80
# execute the JS
window.instance_eval { exec_js! }
# this one will show the updated HTML i.e. including our tag (Ref. #2)
# puts window.document.to_html
# look for the tag in the DOM structure
if window.document.getElementsByTagName( seed_tag )[0]
puts 'Vulnerable to XSS!'
end |
Ref. #1
This is what the code looks like at this point:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>My title! </title>
</head>
<body>
<div>
<script type="text/javascript">
document.write( navigator.userAgent );
</script>
</div>
</body>
</html> |
Pretty much what we passed…
Ref. #2
And now that the JS has been executed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<title>My title! </title>
</head>
<body>
<div>
<script type="text/javascript">
document.write( navigator.userAgent );
</script>
</div>
</body>
<myinjectedtag>blah blah blah </myinjectedtag>
</html> |
Yes I know, document.write() should have written inside the parent of the script tag, the div in this case.
I can’t be bothered to take care of this right now, this is just a prototype…a prototype of a prototype of the prototype actually.
End result
Since we can see that the injection was clearly successful, the inevitable message shall appear:
Conclusion
Yeah it’s really basic and simplistic and not a big deal but it’s fun to see things working — barely but still, heh.
Posted in: Open Source, Programming, Ruby, Security, Web Application
Tags: cross site scripting, detection, dom, experiment, javascript, ruby, therubyracer, vulnerability, xss
One of the things everyone is taking for granted nowadays for every browser and website is decent support for AJAX.
Naturally, scanner devs have been trying to find a decent way to automatically audit that side of the fence or at least provide decent coverage for JS-heavy webapps.
Thing is though… this is a bitch to get right.
And never mind getting it right, it’s hard enough getting the damn thing to work to begin with.
There are 3 things that need to be integrated in order to achieve that sort of functionality:
- a DOM
- a JS interpreter
- a AJAX extension to JS
DOM
The static parts of the DOM can easily be built on top of tested and proven XML parsers like libxml.
The DOM has tricky parts too though, which are where the money is, like timers and events.
It’s alright though, a little bit of smart thread scheduling and clean collections of callbacks will sustain you at the beginning.
JS
JS integration difficulty depends on your language of choice.
Thankfully, more and more bindings for several different JS engines are being released so you can have your pick.
The interfaces can be a bit dodgy though at times which can defeat the whole purpose.
AJAX
AJAX functionality will have to fall on your shoulders but if you managed to get the first 2 parts working this won’t be much of a challenge.
You simply write an AJAX API in your language of choice and make that interface available to the JS code.
Where I’m going with this…
It’s certainly possible to get these working together (without being a multi-million dollar corporation even) but it’ll be a lot of work.
Which is the reason there’s no open source scanner that supports AJAX or even basic JS scripting.
Arachni is no exception, not to mention that due to the young age of the system there had been far more important and basic things to be worked out first.
Luckily, a lot of things have changed in very little time.
The project still has a few bugs but it has been stable enough for a few businesses to build some of their infrastructure on it.
And the v0.4 version if pretty much ready, which takes care of another big feature I could not wait to implement — the High Performance Grid.
Next stop: AJAX
Some time ago I was contacted by a CompSci student who had chosen to add AJAX support to Arachni as his final-year project, nice guy and seems motivated so I’m pretty sure that this is gonna happen.
And since he mentioned it I decided to start looking into the subject matter myself with this:
https://github.com/Zapotek/dom-js-experiment
It’s based on Taka (for the static parts of the DOM) and the V8 JS engine as provided by TheRubyRacer.
I’ve managed to make this work with some simple stuff, JQuery loads without errors and kind of works (I haven’t had much time to test it).
I haven’t had time to implement timers and events yet but they’re coming…
I don’t want to finish this thing on my own though, the other guy will need to work on it for his project.
Point is…AJAX support is coming to Arachni; it will of course take time but it’s going to happen.
Posted in: Arachni, Open Source, Programming, Projects, Ruby, Security, Web Application
Tags: ajax, arachni, dom, javascript, nokogiri, ruby, scanner, security, taka, therubyracer, v8, vulnerability, web
As promised, part 5.
Not that anyone’s reading this crap, once I’m done with the series though I’ll be able to gather them into a nice developer’s guide so I might as well keep going.
As always, keep your installation to up date with the experimental branch before continuing.
These articles have forced me to see Arachni from a completely different perspective and so I keep improving the API to make it more developer friendly.
This time we’ll focus on auditing individual elements and also work on a per page scope.
Let me paint you a picture:
You have a Rails (or some other such framework) web application.
You need to audit it in a consistent manner.
You wish you could simply add security tests to your existing test suite next to your units/functional/integration/etc. tests.
Your webapp framework already keeps track of pretty much all inputs (and if not you can override helper methods like link_to and form_for to keep track of them).
The only thing missing is a system to which you can feed that data and audit those inputs.
You see where I’m going with this, right?
The Arachni framework can easily handle this in a number of ways, some of which I’ll demonstrate here.
Read more…
Posted in: Arachni, Open Source, Programming, Projects, Ruby, Security, Web Application
Tags: api, arachni, audit, automated, crawler, development, differential analysis, framework, module, modules, open source, pattern matching, plugins, rdiff, report, ruby, scanner, security, spider, timing attacck, vulnerability, vulnerability scanner, web, web application, webapp
Since last time we discussed documentation (boring) this post will be about something functional (cool).
And that coolness will in the form of a user interface, beginning with a simple progress output and moving to a full blown console environment — which also means scripting too.
By the way, don’t forget to grab and install the code of the experimental branch and remember to keep it updated before going through each article.
Creating a User Interface
Let’s start simple, just a script that performs a scan like the ones we’ve seen so far:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| require 'arachni/rpc/server/output'
require 'arachni'
# get an instance of the options class
opts = Arachni::Options.instance
opts.audit_forms = true
# this is the seed URL
opts.url = 'http://testfire.net/'
# instantiate the framework
framework = Arachni::Framework.new( opts )
# load just the xss module
framework.modules.load( [ 'xss' ] )
# load default plugins
framework.plugins.load_defaults!
framework.run
puts "Found: " + framework.audit_store.issues.map { |i| i.name }.join( ', ' ) |
Nothing fancy, this will scan http://testfire.net, audit all forms for XSS vulnerabilities and then print the names of the issues it found — XSS obviously, 2 of them.
Helpers
If we are to have a decent interface we’ll need some toys to make it more informative and appealing.
Remember the show_progress() method from Part 2? Let’s take it up a notch.
Let’s abstract the terminal related behavior and put it in its own module to remove duplication and ease re-use.
terminal.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
| #
# Terminal manipulation methods
#
module Terminal
#
# Clears the line before printing using 'puts'
#
# @param [String] str string to output
#
def reputs( str = '' )
reprint str + "\n"
end
#
# Clears the line before printing
#
# @param [String] str string to output
#
def reprint( str = '' )
print "\e[0K" + str
end
#
# Clear the bottom of the screen
#
def clear_screen!
print "\e[2J"
end
#
# Moves cursor top left to its home
#
def move_to_home!
print "\e[H"
end
#
# Flushes the STDOUT buffer
#
def flush!
$stdout.flush
end
end |
Check it out:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| require_relative 'terminal'
include Terminal
# clear the screen
clear_screen!
MAX = 5000
(1..MAX).each {
|i|
# move the cursor to its home, top-left of the screen.
move_to_home!
prog = i / Float( MAX ) * 100
reputs "Counting to #{MAX}..."
reputs "Progress: #{prog}%"
reputs "Current: #{i}"
# make sure that everything is sent out on time
flush!
sleep 0.003
} |
Let’s stick a progressbar and ETA in there:
progress_bar.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
| #
# Progress bar and ETA methods
#
module ProgressBar
#
# Formats elapsed time to hour:min:sec
#
def format_time( t )
t = t.to_i
sec = t % 60
min = ( t / 60 ) % 60
hour = t / 3600
sprintf( "%02d:%02d:%02d", hour, min, sec )
end
#
# Calculates ETA (Estimated Time of Arrival) based on current progress
# and the start time of the operation.
#
# @param [Float] prog current progress percentage
# @param [Time] start_time start time of the operation
#
# @return [String] ETA: hour:min:sec
#
def eta( prog, start_time )
@last_prog ||= prog
@last_eta ||= "ETA: --:--:--"
if @last_prog != prog
elapsed = Time.now - start_time
eta = elapsed * 100 / prog - elapsed
eta_str = "ETA: " + format_time( eta )
else
eta_str = @last_eta
prog = @last_prog
end
@last_prog = prog
@last_eta = eta_str
end
#
# Returns an ASCII progress bar based on the current progress percentage
#
# @param [Float] prog_percentage progress percentage
#
# @return [String] 70% [=======> ] 100%
#
def progress_bar( prog_percentage )
prog = prog_percentage.ceil < 1 ? 1 : prog_percentage.ceil
bar = '=' * (prog - 1) + '>'
space = 100 - prog
bar += ' ' * space
"#{prog_percentage}% [#{bar}] 100% "
end
end |
…and update our simple example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| require_relative 'terminal'
require_relative 'progress_bar'
include Terminal
include ProgressBar
# clear the screen
clear_screen!
start_time = Time.now
MAX = 5000
(1..MAX).each {
|i|
# move the cursor to its home, top-left of the screen.
move_to_home!
prog = i / Float( MAX ) * 100
reputs "Counting to #{MAX}..."
reputs "Progress: #{prog}%"
reputs "Current: #{i}"
reputs
reprint eta( prog, start_time ) + " "
reputs progress_bar( prog.ceil )
# make sure that everything is sent out on time
flush!
sleep 0.003
} |
Now that we’ve got our screen manipulation cleaned up and our widgets ready to go it’s time break down the old show_progress() method and update it to utilise our new resources.
progress_helper.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
| require_relative 'terminal'
require_relative 'progress_bar'
#
# Methods to help with progress presentation
#
# Requires the Terminal and ProgressBar modules to work.
#
module ProgressHelper
include Terminal
include ProgressBar
#
# Show an auto refreshing progress screen
#
def show_progress( framework )
# reset the cursor to the home position
move_to_home!
# the true flag updates the timer
stats = framework.stats( true )
# the true flag forces the auditstore to refresh
# instead of returning the cached object
auditstore = framework.audit_store( true )
issues = auditstore.issues
reputs 'Auditing ' + framework.opts.url.to_s
reputs "===================================="
reputs
show_progress_bar( stats, framework.opts.start_datetime )
reputs
show_current_operation( stats )
reputs
show_modules( framework.modules )
reputs
show_plugins( framework.plugins )
reputs
show_statistics( stats )
reputs
show_issues( issues )
reputs
show_output( Arachni::UI::Output.flush_buffer )
flush!
framework.running?
end
#
# Outputs a progress bar and an ETA
#
def show_progress_bar( stats, start_time )
reputs "# Scan progress"
reputs "------------------------------------"
reputs "#{eta( stats[:progress], start_time )}" +
" #{progress_bar( stats[:progress] )}"
end
#
# Outputs the current operation, either:
# * 'Crawling' -- with the additional info of how many pages have been crawled, or
# * 'Currently auditing' -- and the page which is being audited
#
def show_current_operation( stats )
if stats[:current_page] && !stats[:current_page].empty?
reputs "Currently auditing: #{stats[:current_page]}"
else
if stats[:sitemap_size] != 0
s = stats[:sitemap_size] == 1 ? '' : 's'
plural = "#{stats[:sitemap_size]} page#{s} and counting!"
else
plural = ''
end
reputs "Crawling..." + plural
end
end
#
# Shows all loaded modules
#
# @param [Arachni::ModuleManager] modules
#
def show_modules( modules )
reputs "# Loaded modules"
reputs "------------------------------------"
reputs modules.keys.join( ", " )
end
#
# Shows all running plugins
#
# @param [Arachni::PluginManager] plugins
#
def show_plugins( plugins )
reputs "# Running plugins"
reputs "------------------------------------"
reputs ( plugins.busy? ? plugins.job_names.join( ", " ) : 'None' )
end
#
# Shows a few output messages
#
# @param [Array<Hash>] messages
# @param [Integer] max maximum amount to output
#
def show_output( messages, max = 10 )
reputs "# Scanner output"
reputs "------------------------------------"
# show a few messages at a time just to make the user feels in the loop
max.times {
|i|
if msg_line = messages[i]
next if (msg = msg_line.values.first).empty?
reputs msg
else
reputs '.' * 50
end
}
end
#
# Outputs some basic stats
#
# @param [Hash] stats
#
def show_statistics( stats )
reputs "# Statistics"
reputs "------------------------------------"
reputs "Sent #{stats[:requests]} requests."
reputs "Received and analyzed #{stats[:responses]} responses."
reputs 'In ' + stats[:time]
reputs 'Average: ' + stats[:avg].to_s + ' requests/second.'
end
#
# Outputs a summary of issues
#
# @param [Array<Arachni::Issue>] issues
#
def show_issues( issues )
plural = issues.count == 1 ? '' : 's'
reputs "# #{issues.count} issue#{plural} found"
reputs "------------------------------------"
issues.each.with_index {
|issue, i|
puts "[#{i + 1}] #{issue.name} at #{issue.url} in" +
" #{issue.elem} input `#{issue.var}` using #{issue.method}."
}
end
end |
Audit monitoring interface
Combining all of these together we get:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
| require 'arachni/rpc/server/output'
require 'arachni'
require_relative 'progress_helper'
include ProgressHelper
# get an instance of the options class
opts = Arachni::Options.instance
opts.audit_forms = true
# this is the seed URL
opts.url = 'http://testfire.net/'
# instantiate the framework
framework = Arachni::Framework.new( opts )
# load just the xss module
framework.modules.load( [ 'xss' ] )
# load default plugins
framework.plugins.load_defaults!
# put the scan operation in its own thread
# so that we can do stuff while it's running -- like show progress data..
scan = Thread.new { framework.run }
clear_screen!
# show progress every 0.3 seconds while the scan is running
show_progress( framework ) while scan.alive? && sleep( 0.3 )
# the scan is finished, wait for the thread to return cleanly
scan.join |
And now you have a decent output interface to monitor your scripted scans. 
In case you can’t be bothered to run the examples and are just reading these articles out of curiosity this is what the output would be:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
| Auditing http://testfire.net/
====================================
# Scan progress
------------------------------------
ETA: 00:00:51 20.31% [====================> ] 100%
Currently auditing: http://testfire.net/default.aspx?content=personal_deposit.htm
# Loaded modules
------------------------------------
xss
# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle
# Statistics
------------------------------------
Sent 85 requests.
Received and analyzed 84 responses.
In 00:00:12
Average: 6 requests/second.
# 1 issue found
------------------------------------
[1] Cross-Site Scripting (XSS) at http://testfire.net/search.aspx in form input `txtSearch` using GET.
# Scanner output
------------------------------------
Auditing: [HTTP: 200] http://testfire.net/default.aspx?content=personal_deposit.htm
Harvesting HTTP responses...
Depending on server responsiveness and network conditions this may take a while.
Harvesting HTTP responses...
Depending on server responsiveness and network conditions this may take a while.
..................................................
..................................................
..................................................
.................................................. |
Building the Console
First off, the Console will have a dependency, readline which can be a bitch to install sometimes which is why we’ll use the pure Ruby implementation rb-readline:
gem install rb-readline
Readline helps with…reading lines, duh!
It does a little more than that though, it can provide us with a smart prompt which can keep history allow shortcuts, interpret escape sequences, provide auto-completion etc.
Try this example to see it in action.
To the point…
Ruby makes this sort of thing a pleasure, Consoles, DSLs etc. feel natural.
Our Console will operate under a single premise: The prompt will simply provide access to the methods of class.
And to be more specific, the Console will initiate the prompt which will provide access to all its declared and public methods.
This will allow us to easily extend our console class to accommodate any specialised needs, like creating a console for Arachni.
Base class
The whole thing is easier than you might think, just a few lines and bam!
console.rb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
| require 'readline'
require_relative 'terminal'
#
# Basic Console with a readline prompt.
#
# Extend it and it will provide access to the explicitly declared
# and public methods of your class.
#
# This means no private method and no parent methods.
#
class Console
include Terminal
#
# Initializes the Console and starts the prompt.
#
# The Console will use the IO interfaces you specify,
# however Readline will not so your best bet is to override
# $stdout and $stdin in order to redirect the streams.
#
# @param [IO] stdout output stream
# @param [IO] stdin input stream
#
def initialize( stdout = $stdout, stdin = $stdin )
$stdout = @stdout = stdout
$stdin = @stdin = stdin
# combine the methods of self (which may be a child) with the methods
# of the Console
methods = self.class.instance_methods( false ) | Console.instance_methods( false )
# make a list of all available public methods/commands
@cmd_list = methods.map{ |m| m.to_s }.sort
# setup auto-completion params
comp = proc { |s| @cmd_list.grep( /^#{Regexp.escape( s )}/) }
Readline.completion_append_character = " "
Readline.completion_proc = comp
if out = banner
puts out
end
# loop until someone requests an exit
loop do
line = Readline::readline( '> ' )
next if line.empty?
break if line =~ /exit|quit/
Readline::HISTORY.push( line )
# tokenize the line based whitespace
#
# the first token will be the command name and the rest of the tokens
# will be the arguments for that command
#
meth, *args = *line.split( /\s/ )
begin
# make sure that the command exists
if !@cmd_list.include?( meth )
raise 'Unknown command: ' + meth + "\n" +
"Valid commands: " + commands
end
# extract an unbound method so that we can check for arity
meth = method( meth.to_sym )
# if arity doesn't match the provided arguments pass them as an array
res = ( meth.arity != args.count ) ? meth.call( args ) : meth.call( *args )
# return the result of the command
if res.is_a? String
puts res
else
puts res.inspect if res
end
rescue Exception => e
puts e.to_s
puts e.backtrace.join( "\n" )
end
end
end
# to be overriden by children
def banner
end
# add this empty exit method just so that it'll auto-complete
def exit; end
alias :quit :exit
# returns all available commands
def commands
@cmd_list.join( ' ' )
end
# loads and evaluates a console script
def source( file_path )
instance_eval( IO.read( file_path.to_s ) )
end
# clears the screen
def clear
clear_screen!
move_to_home!
end
private
def force_string( arg )
return arg.join( ' ' ) if arg.is_a?( Array )
arg.to_s
end
def puts( str = '' )
print str
print "\n"
end
def print( str = '' )
@stdout.print str
end
end
Console.new if __FILE__ == $0 |
Let’s take it for a spin, shall we?
1
2
3
4
5
6
| $ ruby console.rb
> <tab><tab>
banner clear commands exit quit source
> commands
banner clear commands exit quit source
> exit |
Doesn’t do much, yet…let’s extend it to do something more interesting.
Console demo – The Jukebox
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
| require 'arachni/rpc/server/output'
require 'arachni'
require_relative 'console'
class Jukebox < Console
VERSION = '0.1'
SONGS = [
['Dream Theater', 'A Change of Seasons' ],
['Iron Maiden', 'Paschendale' ],
['Pain of Salvation', 'Fandango' ],
['Steve Vai', 'For the love of god' ],
['Joe Satriani', 'Andalusia' ]
]
def banner
"#{self.class} v#{VERSION}"
end
def list
SONGS.map { |song| format_song( song ) }.join( "\n" )
end
def play( title )
title = force_string( title ).downcase
song = find_by_title( title )
raise 'Song does not exist.' if !song
@now_playing = song
now_playing
end
def now_playing
puts "Now playing: #{format_song( @now_playing )}"
end
private
def format_song( song )
"#{song[0]} - #{song[1]}"
end
def find_by_title( title )
SONGS.each{
|song|
return song if song[1].downcase == title.downcase
}
nil
end
end
Jukebox.new if __FILE__ == $0 |
And run it:
1
2
3
4
5
6
7
8
9
10
11
12
| Jukebox v0.1
> list
Dream Theater - A Change of Seasons
Iron Maiden - Paschendale
Pain of Salvation - Fandango
Steve Vai - For the love of god
Joe Satriani - Andalusia
> play andalusia
Now playing: Joe Satriani - Andalusia
> now_playing
Now playing: Joe Satriani - Andalusia
> exit |
There’s one feature we haven’t touched yet though, the source command, something familiar to *nix users.
That command will allow us to load and run Console scripts.
These scripts are not limited by the syntax restrictions of the prompt and are evaluated under the namespace of the console.
So you can write vanilla Ruby code and take advantage of the Console’s functionality at the same time.
shuffle.jukescript
1
2
3
4
| SONGS.shuffle.each do
|song|
play song[1]
end |
1
2
3
4
5
6
7
8
| Jukebox v0.1
> source shuffle.jukescript
Now playing: Pain of Salvation - Fandango
Now playing: Dream Theater - A Change of Seasons
Now playing: Joe Satriani - Andalusia
Now playing: Steve Vai - For the love of god
Now playing: Iron Maiden - Paschendale
> |
The Arachni Console
Since you’re already somewhat familiar with the framework and the Console is quite simple I won’t explain everything beat-by-beat this time.
Instead, I’m gonna just paste the ArachniConsole code and play with it a bit.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
| require 'arachni/rpc/server/output'
require 'arachni'
require_relative 'console'
require_relative 'progress_helper'
class Arachni::Console < Console
include ProgressHelper
VERSION = '0.1'
def initialize
reset
super
end
# returns shows a banner
def banner
"Arachni - Web Application Security Scanner Framework v#{@framework.version}\n" +
"Console v#{VERSION}"
end
#
# executes a shell/OS command
#
# sh ls -la *
#
def sh( cmd )
`#{force_string( cmd )}`
end
#
# performs a demo scan
#
# demo
# demo silent
#
def demo( opt = nil )
set 'url', 'http://testfire.net/'
# set 'link_count_limit', '1'
load 'modules', '*'
load 'report', 'json'
audit 'forms'
audit 'links'
audit 'cookies'
run opt
end
#
# sets exclude patterns
#
# exclude test \.pdf$ \.exe$
#
def exclude( regexps )
@opts.exclude = [regexps].flatten.map { |regexp| Regexp.new( regexp ) }
end
#
# sets include patterns
#
# include test \.html$
#
def include( regexps )
@opts.include = [regexps].flatten.map { |regexp| Regexp.new( regexp ) }
end
# resets all settings
def reset
@opts = Arachni::Options.instance
@opts.reset!
# instantiate the framework
@framework = Arachni::Framework.new( @opts )
@framework.plugins.load_defaults!
'Done!'
end
#
# assigns val to opt
#
# set url http://testfire.net
# set link_count_limit 10
#
def set( opt, val )
@opts.send( "#{opt}=".to_sym, cast( opt, val ) )
end
#
# unsets an option
#
# unset url
# unset link_count_limit
#
def unset( opt )
set( opt, nil )
end
#
# shows an option
#
# show url
# show link_count_limit
#
def show( opt )
raise 'Expects argument.' if opt.empty?
@opts.send( "#{opt}".to_sym )
end
#
# imports an AFR report and sets it as the default audistore
#
# import my_report.afr
#
def import( path_to_afr )
path_to_afr = path_to_afr.join( ' ' ) if path_to_afr.is_a? Array
@auditstore = Arachni::AuditStore.load( path_to_afr )
"Loaded audit for #{@auditstore.options['url']} performed on" +
" #{@auditstore.start_datetime} with #{@auditstore.issues.count} issues."
end
#
# exports the auditstore in 'type' format in file 'outfile'
#
# export html my_report.html
# export json my_json_report.json
# [...]
#
def export( type, outfile )
auditstore = @framework.audit_store if !( auditstore = @auditstore )
@opts.reports ||= {}
@opts.reports[type] = {}
@opts.reports[type]['outfile'] = outfile if outfile
@framework.reports.run_one( type, auditstore )
end
#
# load components by type and name
#
# load modules xss*,sqli*
# load plugins defaults/*
# load report json,txt
#
def load( type, names )
names = force_string( names ).split( ',' )
case type
when 'module', 'modules'
@framework.modules.load( names )
when 'plugin', 'plugins', 'plug-ins', 'plug-in'
@framework.plugins.load( names )
when 'report', 'reports'
@opts.reports ||= {}
reps = @framework.reports.load( names )
reps.each { |rep| @opts.reports[rep] = {} }
else
raise 'Unknown component type: ' + type.to_s
end
end
#
# sets elements to audit
#
# audit links
# audit forms
# audit cookies
# audit headers
#
def audit( types )
valid = [ 'links', 'forms', 'cookies', 'headers' ]
[types].flatten.each {
|type|
if !valid.include?( type )
raise 'Invalid element type: ' + type.to_s + "\n" +
'Valid types: ' + valid.join( ", " )
break
end
set 'audit_' + type, 'true'
}
end
#
# runs the audit
#
# run
# run silent
#
def run( *args )
@job = run_in_thread
return if silent?( args )
progress
clear_screen!
end
#
# attach to running scan
#
def attach
progress
end
#
# show progress of running scan
#
# progress
# progress glimpse
#
def progress( *args )
raise 'No scan running.' if !running?
clear_screen!
if glimpse?( args )
show_progress( @framework )
return
else
sleep( 0.3 ) while show_progress( @framework )
show_progress( @framework )
@job.join
'Done!'
end
end
# pauses running scan
def pause
@framework.pause!
'Will pause as soon as possible...'
end
# resumes running scan
def resume
@framework.resume!
'Resuming!'
end
# kills/aborts running scan
def kill
@framework.clean_up!
@job.exit
end
alias :abort :kill
private
def running?
@job && @job.alive?
end
def glimpse?( args )
includes_modifier?( args, :glimpse )
end
def silent?( args )
includes_modifier?( args, :silent )
end
def includes_modifier?( args, modifier )
args.flatten!
args.include?( modifier ) || args.include?( modifier.to_s )
end
def run_in_thread
Thread.new { @framework.run }
end
def cast( opt, val )
default = @opts.instance_variable_get( "@#{opt}" )
case default
when Fixnum, Integer
val.to_i
when Float
val.to_f
when String, URI
val.to_s
when TrueClass, FalseClass
val.to_s.downcase == 'true' ? true : false
when Array
val.to_a
when Hash
val.to_h
else
val.to_s
end
end
end
Arachni::Console.new if __FILE__ == $0 |
Let’s jump in!
1
2
3
| Arachni - Web Application Security Scanner Framework v0.4
Console v0.1
> demo |
Ah, that’s a familiar sight, it’s the progress screen from a couple of chapters ago.
To detach and return to the prompt hit Ctrl-C, don’t worry about the error messages the scan won’t die.
Say you want to take a glimpse of the running scan while detached:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
| > progress glimpse
Auditing http://testfire.net/
====================================
# Scan progress
------------------------------------
ETA: 00:14:13 13.07% [=============> ] 100%
Currently auditing: http://testfire.net/default.aspx?content=inside.htm
# Loaded modules
------------------------------------
interesting_responses, common_files, mixed_resource, xst, http_put, webdav, directory_listing, allowed_methods, htaccess_limit, ssn, private_ip, emails, credit_card, cvs_svn_users, captcha, html_objects, unencrypted_password_forms, backdoors, backup_files, common_directories, trainer, os_cmd_injection, sqli, xss_script_tag, sqli_blind_rdiff, path_traversal, xss_event, xss_uri, sqli_blind_timing, code_injection, rfi, xss_tag, response_splitting, csrf, os_cmd_injection_timing, ldapi, code_injection_timing, xss_path, xpath, unvalidated_redirect, xss
# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle
# Statistics
------------------------------------
Sent 5873 requests.
Received and analyzed 5725 responses.
In 00:01:29
Average: 44 requests/second.
# 7 issues found
------------------------------------
[1] Cross-Site Scripting (XSS) at http://testfire.net/search.aspx in form input `txtSearch` using GET.
[2] Path Traversal at http://testfire.net/default.aspx in link input `content` using GET.
[3] Allowed HTTP methods at HTTP://testfire.net in server input `` using OPTIONS.
[4] Interesting server response. at http://testfire.net/admin/ in server input `` using GET.
[5] Interesting server response. at http://testfire.net/default.aspx in server input `` using GET.
[6] Found an HTML object. at http://testfire.net/default.aspx in body input `` using GET.
[7] Interesting server response. at http://testfire.net/default.aspx/%3Carachni_xss_uri_7b063a2cf51c145d46e218f8b1697adb21339ba91fd57912bd5057b8d485a684 in server input `` using GET.
# Scanner output
------------------------------------
PathTraversal: Analyzing response #5726...
PathTraversal: Analyzing response #5727...
PathTraversal: Analyzing response #5711...
PathTraversal: Analyzing response #5728...
PathTraversal: Analyzing response #5714...
PathTraversal: Analyzing response #5729...
..................................................
..................................................
..................................................
..................................................
> |
And say you want to attach to the running scan and monitor its progress:
Hint: progress without the glimpse argument would do the same…
Now lets say you want to get a preliminary report while the scan is running:
1
2
3
4
5
| > export html myreport.html
> sh ls *.html
myreport.html
> sh firefox myreport.html |
Aha! The report contained what we were looking for, kill the scan!
1
2
3
4
| > kill
#<Thread:0x000000023f9588 aborting>
> kill
#<Thread:0x000000023f9588 dead> |
And if the fucker won’t die you can skip the clean up process by:
1
2
| > kill force
#<Thread:0x000000023f9588 dead> |
And if you want to import a previous report:
1
2
3
4
5
6
7
8
9
10
| > sh ls *.afr
2011-12-02 01.26.32 +0200.afr
> import 2011-12-02 01.26.32 +0200.afr
Loaded audit for http://testfire.net/ performed on Fri Dec 2 01:25:50 2011 with 7 issues.
> export json report.json
> sh ls *.json
report.json
> |
And of course…scripting:
testfire
1
2
3
4
5
6
7
8
9
10
11
12
| set 'url', 'http://testfire.net/'
load 'modules', '*'
load 'plugins', 'defaults/*'
load 'report', 'json'
audit 'forms'
audit 'links'
audit 'cookies'
run :silent
progress :glimpse |
Output:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| Arachni - Web Application Security Scanner Framework v0.4
Console v0.1
> source testfire
Auditing http://testfire.net/
====================================
# Scan progress
------------------------------------
ETA: --:--:-- 0.0% [> ] 100%
Crawling...
# Loaded modules
------------------------------------
interesting_responses, common_files, mixed_resource, xst, http_put, webdav, directory_listing, allowed_methods, htaccess_limit, ssn, private_ip, emails, credit_card, cvs_svn_users, captcha, html_objects, unencrypted_password_forms, backdoors, backup_files, common_directories, trainer, os_cmd_injection, sqli, xss_script_tag, sqli_blind_rdiff, path_traversal, xss_event, xss_uri, sqli_blind_timing, code_injection, rfi, xss_tag, response_splitting, csrf, os_cmd_injection_timing, ldapi, code_injection_timing, xss_path, xpath, unvalidated_redirect, xss
# Running plugins
------------------------------------
healthmap, timing_attacks, discovery, manual_verification, uniformity, content_types, autothrottle
# Statistics
------------------------------------
Sent 0 requests.
Received and analyzed 0 responses.
In 00:00:00
Average: 0 requests/second.
# 0 issues found
------------------------------------
# Scanner output
------------------------------------
Waiting for plugins to settle...
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
> |
Quite a long post this one but it was well worth the effort…barely, heh.
Anyways, that’s all for now, the next part will be a lot more hardcore and will demonstrate a highly specialised audit that will be customisable down to the individual element.
See ya then…
Posted in: Arachni, Open Source, Programming, Projects, Ruby, Security, Web Application
Tags: arachni, audit, automated, crawler, framework, modules, open source, plugins, ruby, scanner, security, spider, vulnerability, vulnerability scanner, web, web application, webapp, xss
Managing components
Time to leave the framework for now and get back to some basics since after seeing things as a whole you’ll better appreciate these important details.
All components (modules, reports, plugins and the unappreciated path extractors) are handled by the ComponentManager.
The manager’s job is simple and can be described as:
Given the path to a library and the namespace of the components it allows manipulation and organization of said components.
For example, the library of the Arachni modules is /modules and all modules live under the namespace of the Arachni::Modules module.
Unless you want to create your own components you won’t have to deal with that manager directly, you’ll only deal with its children:
- ModuleManager
- ReportManager
- PluginManager
(The path extractors don’t get their own manager because they’re quite simple in nature.)
These managers extend the ComponentManager and because the ComponentManager extends the Hash class working with them will feel quite natural.
Finally, the Framework provides you with access to all the discussed managers and they, in turn, provide you with access to the components they manage.
By convention, all components have the following methods:
- prepare — Optional, allows them to prepare their environment
- run — Mandatory, executes the main payload.
- clean_up — Optional, gives them a chance to clean up after themselves.
- self.info — Mandatory, returns general info like version number, name, author, etc.. Actual format depends on component type.
Let’s see how these manager work and what they can do.
Read more…
Posted in: Arachni, Open Source, Programming, Projects, Ruby, Security, Web Application
Tags: arachni, audit, automated, component, crawler, framework, manager, module, modules, open source, plugins, report, ruby, scanner, security, spider, vulnerability, vulnerability scanner, web application, webapp
