Concurrent Multi-browser Crawling of Ajax-based Web Applications

Web applications are becoming more and more the replacement of desktop applications, using Ajax as their main technology to achieve user-friendly and interactive user interfaces. The best known example is the email application GMail, other examples include Google Docs, Google Maps, Facebook, Yahoo Mail and Twitter. These technological advantages brings some number of challenges when crawling those applications. The state of the user interface can not be accessed directly as with original web applications. This disables the usage of traditional web crawlers, which uses static analysis to find new pages. A dynamic approach, using a real browser of explore the user interface, was proven to be very successful. The dynamic approach did not scale well to large application, when applications grew the limits of this approach were reached. In this thesis we first perform a measurement to find the exact limitations of using this dynamic approach, knowing the limitations we propose a technique to improve the scalability of Ajax web crawlers. We use the model of web application to apply a dynamic partition function, which we use to analyse the different parts of an Ajax application concurrently using multiple browsers. We implemented our approach in the open source testing tool Crawljax. We describe three case-studies evaluating the effectiveness of our implemented performance improvements. The results are very promising, showing expected performance improvements, and enabled us to use crawljax successfully in large production environments.

Subject

Ajax
Crawling
Selenium
WebDriver
Crawljax
Multi-browser
Concurrent

To reference this document use:

http://resolver.tudelft.nl/uuid:0ccc131f-a7b7-48ac-bffc-7d67ab7f756f

Embargo date

2011-07-09

Part of collection

Student theses

Document type

master thesis

Rights

Files

PDF

concurrent_multi_browser_ ... ations.pdf

7.11 MB

Close viewer