Checker

Component description

The MKSearch checker component is essentially an integration layer between the data acquisition system and the repository, ensuring the currency of the data store. The checker component does not exist as a distinct component in the beta 1 version of MKSearch, a new repository is created for each crawl session.

Completed task information has been moved to the beta 1 checker plans archive.

Beta 2 development plans

Check un-linked documents

At present, the crawler component pushes the whole data aquisition process by following published hyperlinks and creates a new repository for each session. However, with an incremental indexing scheme, previously indexed documents may be removed between sessions and un-linked. In this case, the crawler will not discover resources are obsolete. The checker component therefore needs periodically to check whether "old" source documents still exist.