Wilbur -- the free file finder
This started out as a simple link check for some of my websites.
I was bored one day and started messing with it -- the result
being the "messy" code before you. :)
It is time to sit down and think about it for a bit before
it moves on, that is the stage I am at now.
to install: nothing fancy -- can be run from wherever you
extracted it to.
on startup the program reads a file called project.txt in the
current directory. This file just contains a path to where
the other configuration files are located. this will also be
the path where any downloaded files are saved to.
other config files:
dltypes.txt -> the kinds of files you want to download
ignoretypes.txt -> ignore links containing any of these strings -- good for
filtering know "dead" sites etc
interesting.txt -> links containing these strings are saved out for later perusal
by you in a file called checkthis.txt
two files are generated while the program is running
resume.txt -- contains the current state of the "to check" que. if you shut
down in the middle of a run, you can just reload this list and start right where
you left off.
skipped.txt -- if the que gets full (20,000 links hardcoded right now) new links are
save to this file and NOT added to the que.