• Overview
  • basic installation
  • advanced installation
    • installing sybil
    • sybil usage
    • installing snorkel

Overview

sybil and snorkel are free and libre software meant for ad-hoc analysis of instrumentation data. On this page, we’ll summarize how to install and import data into these two tools.

sybil is a simple OLAP that is optimized for append-only, immutable data and gives up the traditional DB guarantees (updateability, transactions, etc) and operations (JOINS, UPDATES, DELETES) in exchange for fast full table scans.

snorkel is the python frontend that sits on top of sybil and is used for iterative analysis (think firefighting or push day analysis). Snorkel can build and run time series, table, distribution and raw samples queries and visualize / compare their results in practical ways.

basic installation

There’s a pypi package for snorkel+sybil for x64 machines called snorkel-lite which is pre-configured for out of the box analysis.

See the snorkel wiki for up to date installation instructions

advanced installation

installing sybil

sybil is a Go app, so it requires that Go is installed and the GOPATH is properly configured. If you are first installing Go, you can configure your GOPATH like so:

export GOPATH=$HOME/go
mkdir $GOPATH

After that, setup is easy:

go install github.com/logv/sybil

sybil usage

sybil is a command line binary that allows one to ingest JSON records into tables and subsequently query them

Use sybil ingest -table my_table < samples.json to pull in samples off stdin and write them into db/my_table.

Once you have read some samples, prepare them for fast queries with sybil digest -table my_table

Querying the samples is easy, too:

• sybil query -table my_table -info
• sybil query -table my_table -samples
• sybil query -table my_table -group my_categorical -int my_numeric -op hist -print

For more usage examples, see sybil’s quick overview

installing snorkel

snorkel is a python webserver. The installation here is a bit more involved since it is an HTTP server and may require some more in depth config (SSL certs, adding users, editing a config file, etc)

# get the example deploy repository
git clone https://github.com/logv/snorkel.deploy

# [optional] create a virtual environment
virtualenv dev 
source dev/bin/activate

# install the snorkel module and dependencies,
# including sybil
pip install -r requirements.txt

# edit the config and make sure to change
# the secret values
${EDITOR} config/local.py

# add a new user
ENV=local.py flask add-user <username>

# mark user as a superuser
ENV=local.py flask add-superuser <username>

# run the server and log in
# with the newly created users
ENV=local.py flask run --port 2334

# by users can see all tables. we can change that
# by setting up permissions in config/local.rbac
${EDITOR} config/local.rbac