crawler

crawler

Script to crawl html and add href links, crawling and indexing 5k sites / second into big-data graph DB.

Build it

Binary

go get -u github.com/dgoldstein1/crawler

Docker

docker build . -t dgoldstein1/crawler

Run it

dc up -d

or with dependencies running locally

# run crawl on wikipedia
export GRAPH_DB_ENDPOINT="https://graphapi-twowaykv-dev.herokuapp.com/services/biggraph" # endpoint of graph database
export TWO_WAY_KV_ENDPOINT="https://graphapi-twowaykv-dev.herokuapp.com/services/twowaykv" # endpoint of k:v <-> v:k lookup metadata db
export STARTING_ENDPOINT="https://en.wikipedia.org/wiki/String_cheese" # if empty, finds random article
export PARALLELISM=20 # number of parallel threads to run
export MS_DELAY=5 # ms delay between each request
# export METRICS_PORT=8002 # port where prom metrics are served
export MAX_APPROX_NODES=1000 # approximate number of nodes to visit (+/- one order of magnitude), set to '-1' for unlimited crawl
export PORT=8888
export ENGLISH_WORD_LIST_PATH="/home/david/go/src/github.com/dgoldstein1/crawler/synonyms/english.txt"
build/crawler wikipedia

Development

Local Development

• Install inotifywait

./watch_dev_changes.sh

Testing

go test $(go list ./... | grep -v /vendor/)

Benchmarks

Time to get to 1001007 nodes: 3m18.5 Nodes / Sec: 5055.5 Size of graph: 644kb Size of entries: 32mb

Authors

• David Goldstein - DavidCharlesGoldstein.com - Decipher Technology Studios

License

This project is licensed under the MIT License - see the LICENSE.md file for details