Manual Crawling

[Abhi_M]

One of the essential thing in web application security assessment is crawling.

Quote:A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner or in an orderly fashion.

A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.

from Wikipedia

Its true that Mantra lacks a good crawler as of now.
Navicrawler was a safe bet, but its no longer updated.

So lets see how we can collect all the links in a particular webpage.


We are going to use Web Developer extension. You can activate it from the sidebar.




You can see a new toolbar now, It has lots of functions but as of now we are concentrating only on collecting Link information.




List of links can be collected by going to Information > View Link Information




Now you have a nice good list of all the links in the particular page you scanned for




You can also use Web Developer extension without activating toolbar buy going through 'Ayudha' menu.




Enjoy.!!!