In this post, we will talk about the differences between Scuba and Snorkel, two realtime data analysis tools. If you want to read more about OLAPs and Time Series DBs or see a list of OLAP engines, try this page
Scuba is a realtime log analysis tool used at Facebook. It’s both a UI for building queries and a distributed online analytical processing (OLAP) query engine. With a fleet of servers at its disposal, Scuba holds billions of records in RAM and runs simple aggregation queries in under a second. Today, Scuba is used along-side ODS (a time series database) to monitor and maintain Facebook’s infrastructure.
In general, Scuba trades accuracy and consistency for speed: its primary purpose is as a diagnosis and debugging tool for infrastructure. When people want to monitor an existing metric, they use their TSDB. When they want to explore, debug or diagnose, they use Scuba.
Unfortunately for us, Scuba is not available outside Facebook. That’s where Snorkel comes in. Snorkel is conceptually the little sister to Scuba. The major difference is that Snorkel is 1) free, 2) available and 3) smaller purpose.
Snorkel’s backend, Sybil, runs on a single machine and with much smaller datasets than Scuba’s backend. Sybil stores data on disk (instead of in memory) and is generally good for queries on up to 10M rows on commodity hardware (30M rows on server class hardware) vs the billions that Scuba can do.
Aside from their scale difference, Scuba and Snorkel are similar in many ways: they allow for ad-hoc queries, they perform full table scans, they do not require up front schema definitions and they cap table size by memory and time limits. Together, these particular features make for a compelling backend for digging through instrumentation.
Feature Comparison Matrix
|Append only data||X||X||X|
|Mem Capped Tables||X||X||X|
|Parallel Query Engine||X||X|
|Feasible Table Scans||1B+||10 - 30M||N/A||1B+M|
|Composable Time Series||X||?|
|Custom Views by Dataset||X||X||?|
- Add note about distributed query work being planned
- Add note about support for custom views per dataset
- First write up.
- Add comparison table
- Add intro paragraphs