Initial review notes
One of the great advantages (and drawbacks) of the Spindle spider is its great simplicity. The spider is a single class that runs multiple instances as threads. It reads input streams via a Reader.
The number of threads that Spindle spawns is specified on the command line. A thread is created to handle each URL and named for logging purposes. The host calls the join() method on the thread and waits for it to complete. URLs are added and retreived from a synchronized queue.
The HTML content is tokenized via the supporting HTML tokenizer class based on < and > separators. This creates an Enumeration of TextToken and TagToken objects that contain attributes. The spider identifies links and extracts URLs as it indexes the page and adds them to the queue.
Spindle recognises standard anchor elements and frame source attributes. It ignores links with protocols other than HTTP and HTTPS, and those that start with "", “#” and “javascript:”.
The tokenizer seems to assume tags do not break over separate lines and it is not clear whether it would properly handle empty XHTML elements.
Limitations
Included and excluded URLs are specified via command line arguments. The HTTP client features are minimal, only status code 200 is recognised.
The spider checks a new URL is not indexed before adding it to the queue, so several threads might load the same page before one of them indexes it.
Does not recognise the robot exclusion protocol nor equivalent meta element directions.