SOURCE

Github Repository

• https://github.com/DevDungeon/WebGenome

Overview 

A breadth first web crawler that stores HTTP headers in a MongoDB database with a web front end all written in Go. 

Web Genome, or www.webgeno.me, is a project that stores the HTTP headers returned from websites around the world. This information can be used for statistical analysis or fingerprinting the underlying technology.

This information can be useful for security researchers to identify numbers of unpatched servers or discover anomalous behavior. The information can also be used to identify which websites run certain web servers, program in a certain language, or use a certain content management system which can all be useful to certain marketing and sales people.

The great thing about data is that it can be useful to many different people in many different ways. The data is just data and it is all already publicly accessible. Web Genome simply archives the HTTP header responses that web servers already serve up willingly to anyone who requests them.

It is written in the Go programming language and uses MongoDB to store data. There is a website application and a worker that crawls the web. It is open source and you can run your own instance and contribute to the code.

Web Genome crawls the web in a breadth first fashion starting from www.devdungeon.com. Any domain found can be connected back to devdungeon.com via hyperlinks. When viewing a domain detail page, there is a "Path to DevDungeon.com" section that shows how the crawler went from devdungeon.com to that particular domain using organic hyperlinks.

SOURCE

Website

• http://www.webgeno.me

Setup

Create a Linux user

sudo useradd webgenome

Set up the Go environment

Set up your GOPATH to be /home/webgenome/go

# In ~/.bashrc



export GOPATH=/home/webgenome/gospace

Get the packages

# All at once with



go get github.cm/DevDungeon/WebGenome...



# Or individually



go get github.com/DevDungeon/WebGenome



go get github.com/DevDungeon/WebGenome/core



go get github.com/DevDungeon/WebGenome/website



go get github.com/DevDungeon/WebGenome/worker_http

Setting up database

Create a MongoDB database and seed it with a domain. Add an index on the name field to really speed things up:

sudo apt install mongodb

mongo

> use webgenome

> db.domains.insert({'name':'www.devdungeon.com'})

> db.domains.createIndex({name:1})

Sample database queries

db.getCollectionNames()



db.showCollections()



db.domains.getIndexes()



db.domains.stats()



db.domains.count()



db.domains.find({name:'www.devdungeon.com'})



db.domains.count({lastchecked:{$exists:true}, skipped: null})

db.domains.find({headers: {$elemMatch: {value: {$regex: 'Cookie'}}}}).pretty()



db.domains.find({headers: {$elemMatch: {key: {$regex: 'Drupal'}}}}).pretty()

Run website using systemd

The systemd directory contains a sample service file that can be used to run the website as a service.

sudo cp /home/webgenome/go/src/github.com/DevDungeon/WebGenome/systemctl/webgenome.service /etc/systemd/system/

sudo chown root:root /etc/systemd/system/webgenome.service



sudo vim /etc/systemd/system/webgenome.service # Double check settings



systemctl webgenome enable



systemctl webgenome start

Nginx reverse proxy

The web server will listen on port 3000 by default. Access it directly or set up a reverse proxy with nginx like this:

# /etc/nginx/conf.d/webgenome.conf



server {  # Redirect non-www to www



    listen 80;



    server_name webgeno.me;



    return 301 $scheme://www.webgeno.me$request_uri;



}



server {



    listen 80;



    server_name www.webgeno.me;



    location / {



    proxy_set_header X-Real-IP $remote_addr;



        proxy_pass http://localhost:3000;



    }

}

Running worker_http

Here is an example usage of running the crawler:

worker_http --host=localhost --database=webgenome --collection=domains --max-threads=4 --http-timeout=30 --batch-size=100 --verbose

Updating

# Update the source and executables



go get -u github.com/DevDungeon/WebGenome...



# Restart the service



systemctl restart webgenome

Source Code

• https://github.com/DevDungeon/WebGenome?files=1

License

GNU GPL v2. See LICENSE.txt.

Desclamer

The worker is not run as a service because it may fill up your disk space and it should be run in verbose mode at the beginning so you can tune and make sure it's not hammering nested subdomains on a single site.

Roadmap

You can kill the http_worker at any time and restart it without causing any problems. If you run multiple instances of the worker at the same time it will end up checking a lot of the domains multiple times. If you want to crawl more just increase the number of threads to the worker. I was able to run it with 256 threads on a small Linode computer.

Github account 

• https://github.com/DevDungeon