Checker

Component description

The MKSearch checker component is essentially an integration layer between the data acquisition system and the repository, ensuring the currency of the data store. The checker component does not exist as a distinct component in the alpha version of MKSearch, a new repository is created for each crawl session.

Development plans

Repository management interfaces

A wrapper layer is required around the repository to dynamically maintain the contents to permit incremental indexing. The wrapper will need to handle exception messages from the crawler, validator and indexer components to purge invalid records from the repository. It will also need to add RDF statements generated by the indexer and ultimately purge stale entries from the result cache.

Check un-linked documents

At present, the crawler component pushes the whole data aquisition process by following published hyperlinks and creates a new repository for each session. However, with an incremental indexing scheme, previously indexed documents may be removed between sessions and un-linked. In this case, the crawler will not discover resources are obsolete. The checker component therefore needs periodically to check whether "old" source documents still exist.