Manual Installation

installing sybil

sybil is a Go app, so it requires that Go is installed and the GOPATH is properly configured. If you are first installing Go, you can configure your GOPATH like so:

export GOPATH=$HOME/go
mkdir $GOPATH

After that, setup is easy:

go install github.com/logv/sybil

sybil usage

sybil is a command line binary that allows one to ingest JSON records into tables and subsequently query them

Use sybil ingest -table my_table < samples.json to pull in samples off stdin and write them into db/my_table.

Once you have read some samples, prepare them for fast queries with sybil digest -table my_table

Querying the samples is easy, too:

• sybil query -table my_table -info
• sybil query -table my_table -samples
• sybil query -table my_table -group my_categorical -int my_numeric -op hist -print

For more usage examples, see sybil’s quick overview

installing snorkel

NOTE: As of Feb. 2019 these instructions are out of date. New ones will be coming soon

snorkel is a Node webserver. The installation here is a bit more involved since it is an HTTP server and may require some more in depth config (SSL certs, adding users, editing a config file, picking a configuration DB, etc)

git clone https://github.com/logv/snorkel
# install node (>= 0.8) 

# get the code and dependencies for snorkel
cd snorkel/snorkel

# cross your fingers and hope all npm deps resolve
npm install

# helper for restarting the server when js files change
# (for production, i recommend using pm2 with a memory limit on processes)
npm install -g nodemon

# edit your config
# make sure to enable proxy mode and set the hostname, if applicable
cp config/config.js config/local/my_dev_env.js

# IMPORTANT for sybil usage!
# edit the config and set backend.driver = 'sybil' 

# create some self signed SSL certs
scripts/setup_certificates.sh

# setup a user (creates config/users.htpasswd with basic http auth)
bash scripts/add_user.sh

# start in development mode
ENV=local/my_dev_env nodemon app.js

# open up localhost:3000 to visit your snorkel instance!
# the default instance comes with several tables: snorkel/queries,
# snorkel/ingest and perf/ for browsing.

adding data to snorkel

You can use snorkel as the collection agent for data (which will in turn call sybil) via HTTP requests or the data can be directly ingested with sybil. If you choose to directly ingest data with sybil, just link the resulting table directly into snorkel/snorkel/db, like so:

# if we created a table with sybil ingest -table my_table
# we can use the following to symlink that table into our snorkel db dir
ln -s ./db/my_table /path/to/snorkel/snorkel/db/