Skip Navigation

Spiders

J-Spider

JoBo

Arachnid

Spindle

Acme Spider

Metis

Heritrix

HouseSpider

WebLech

Excluded spiders

Link mappers

Content parsers

RDF Crawlers

Sign up

If you sign up for an account on this web site you can customise elements of this site and subscribe to an email newsletter.

If you have an account on this web site you may login.

If you have an account on this site but have forgotten your user name and / or your password then you can request an account reminder email.

J-Spider

J-Spider is an alpha release under the GNU Library or LGPLM licence. The J-Spider source includes some dependencies on Apache software:

  • The class net.javacoding.jspider.core.logging.impl.CommonsLoggingLogProvider depends on the package org.apache.commons.logging released under the Apache Software License.
  • The class net.javacoding.jspider.mod.plugin.velocity.VelocityPlugin depends on the packages org.apache.velocity, org.apache.velocity.app and org.apache.velocity.context, released under the Apache Software License.

Initial review notes

J-Spider is designed to be a spidering engine, not an application itself, so it has a set of interfaces that can be used to add custom functionality and features. Plugins can fulfil tasks such as saving content to a file or reporting responses to URL requests. The spidering requirements are configured by a set of Rules. Events are triggered by incidents such as identifying a new site or finding a 404 error. Each activity is defined as a Task.

J-Spider has a well layered and extensible architecture and there is a detailed user guide. A developer guide is expected soon. The plugin configurations can be written in plain text name/value format. The apparent dependencies on the Apache logging and Velocity packages are for optional plugins only and do not compromise the licence status of the system as a whole.

J-Spider requires a Java 1.3 compatible runtime environment and an XML parser, Apache Xerces is provided as standard. The GNU Aelfred XML parser should be equally suitable. Provided this is confirmed, J-Spider would be a recommended for the MKSearch project.

Up | >>

This document was last modified by Philip Shaw on 2004-11-04 10:05:16
Copyright MKDoc Ltd. and others.
The Free Documentation License http://www.gnu.org/copyleft/fdl.html