#gravwell

I had been having some trouble with stability on my desktop system for a while: browser tabs crashing, random segfaults, occasionally the entire system would lock up. It's an AMD Ryzen 7 3700X processor in an ASRock X570 motherboard, purchased shortly after the Ryzen 7 was introduced. After thinking about it a bit, I realized I'd never updated the BIOS. Sure enough, there had been several patches over the last few years, and stability improved significantly after an update.

Now that it's been running for about a week on the new BIOS, I thought I'd see if I could detect any differences from the data I'd been gathering in Gravwell. Looking at the mean CPU temperature, I didn't see much difference:

But once I added variance to the chart, I saw a huge change:

This means that the CPU temperature has been much more stable since the update. If I add min and max temperatures to the graph, we see that the mean temperature (yellow) and the minimum temp (orange) stay a lot closer together after the update, and the max temp is "spikier", meaning the CPU spends less time at high temperatures:

[Part 1] Getting Started with Gravwell - Installation

Today we are announcing some major changes to Gravwell Community Edition (CE).

We’re opening up CE in two major ways. First, we’re upping the ingest limit from the current 2GB/day to 1/10th of an AU in bytes (which is about 13.9 GB/day). Second, we’re removing the non-commercial restrictions on a CE license. You can now get Gravwell, for free up to 13.9GB/day, to use at your home and/or your place of work.

We’re pushing this CE change to give folks at an organization of any size the change to play with some really, really cool tools. Gravwell was built for enterprise-grade scalability by a crack team of high performance computing experts, but that doesn’t mean it’s not also useful in smaller instances. Everyone has logs and network data that they need to sort through.

I look back at my start with computers and the sense of wonder and exploration that I had. When I got my hand-me-down Packard Bell to call my friend’s computer across town so we could play a game of Warcraft….it was like I had entered an exciting new world that desperately needed exploring. I feel some of that same wonder when faced with cool data problems.

Overview

We are pleased to announce the immediate availability of Gravwell 4.2.0, code name Voyager.  The 4.2.0 Voyager release marks a major milestone for us in reducing the complexity of interacting with the query language while also enabling highly optimized data storage and indexing performance.  Long story short, we sharpened both sides of the axe.

Data Explorer

The clear star of this release is Data Explorer, a new system which automatically identifies the format of data and applies structure when possible.  The data explorer system makes it much easier to drop into a new data environment and explore available fields, formats, and processors.  When you first click on Data Explorer you will be presented with your list of tags. Click a tag and if there is no existing format association, Gravwell will make some intelligent guesses and present you with options.  For example, here is what Gravwell presented to me when I first loaded up CoreDNS data.

A few things COULD work, but Gravwell was very confident that this is JSON, hence the 100% confidence level.  I select that auto extractor and I am off, clicking my way through CoreDNS JSON data.

Data Explorer is an intelligent interface for rapidly identifying structure on data and assigning extractors that can help render data into more usable forms.  Binary formats like Netflow and IPFix are easy to navigate, and building out the base filtering queries for those types becomes much easier.

The little filter icons let you rapidly dig into fields and down-select. As you apply filters, the system will refresh the query and provide a new result set.  This lets you click your way into a subset of data and then pivot out for more advanced querying.  The filters can also be adjusted using the drop-down menu and editors.  This means you can tweak your query filters and see how the resulting query is rewritten.  We wanted to make sure that the power of the query language was never hidden from users, so even though you may be clicking your way to glory the query is always presented and available for use. It's a great way to get used to the query syntax and filtering without diving headlong into the full language.

Data Explorer also has another trick up its sleeve that is subtle but important: preview mode queries.  When using Data Explorer, each query is fired in a "preview mode" which means you don't have to specify time ranges, instead you are telling Gravwell "just go find enough to make sense."  Behind the scenes Gravwell identifies the type of renderer you are using and acquires enough data (filters included) to present a reasonable picture.  Looking at the raw entries, we will get about a thousand entries; a chart pulls enough to show a decent curve; maps have enough points so a human gets the idea, and so on. You don't have to worry when the data is, Gravwell will figure that out and go get it.  As a result you can click around and your time range will dynamically expand or shrink as needed to present usable data.   When you pivot out of Data Explorer you can then specify a time range and carry on as normal.

Query Studio

A common theme we see from users when they are onboarding new data, investigating an issue, or just trying to figure out a crazy data structure (or complete lack thereof) is that they have multiple windows open, constantly flipping back and forth between some nuanced representation of a data set and the view of the raw underlying data.  As a result we have decided to bring the Query Studio system to the front and center.  Query Studio is a refinement on the original query window which can display multiple tabs within the same view port while also attaching notes, labels, and rapidly cloning & refining a query.  Query Studio is going to be growing a lot in the future and will eventually become the primary query interface into Gravwell.

Per Tag Acceleration

The heaviest hitter on the backend system is our new per-tag acceleration capability.  With the new system we have made it much easier to really hone the performance of specific tags and/or categories of tags; now, administrators can deploy useful defaults and then fine tune acceleration for specific tags.  A new clean installation of Gravwell now packs a decent baseline set of well definitions, plus some fine tuned accelerator definitions for common data types.  You may notice that the Accelerator definitions can be in separate files; this means you can develop them independently and then just import them for easier management.  Accelerator definitions also work with tag wildcards, so you can say "do X for zeekconn and Y for zeek*".  The zeekconn tag will get the specific accelerator definition and everything else that matches zeek* will get the other.  Full documentation on per-tag accelerator definitions is available here.

A data source that really shows off the new system is Zeek.  Zeek can provide a tremendous amount of useful data: connection logs, DNS, software inventory, DHCP, and more.  The multitude of plugins available also mean that the type and format of that data can change based on the deployment.  With per-tag acceleration definitions, we can fine-tune the extremely high volume zeekconn, zeekfile, and zeekdns tags and then provide a reasonable default acceleration config for all the others. In other words, speed when you know what you are doing and flexibility when you aren't quite sure yet.  As always, changing the accelerator definitions doesn't mean your queries need to change or you need to re-index anything--Gravwell just figures out what it needs to do with what it has.

New Modules

We are also happy to announce a few search modules.  The anonymize module is extremely useful for taking potentially sensitive data and anonymizing it in a sensible way while also being type aware.  For example, if you choose to anonymize IPs, it will provide randomized IPs that are consistent across the search.  Anonymize makes it safer to provide exemplar data to colleagues, other business units, or even law enforcement without needing to show the real deal.

The other module is dns. How many times have you narrowed down your search to an IP only to then pivot out to a console and fire off a reverse DNS lookup to figure out what machine it actually is?  The dns module can now do that for you directly in the query.  BE WARNED, while we cache aggressively, if you attempt to do a billion reverse DNS queries on an enterprise wide flow set, one of two things will happen:  I.T. calls with a very sad face, or they cut you off.   Use the dns module with care.

Conclusion