#jedleidheiser

Version 1.0

By Jed LeidheiserAugust 6, 2014

Background

In the setup to this article, we discussed how Big Data Security Analytics (BDSA) is an evolution beyond the limitations of classic Security Information & Event Management (SIEM) solutions. Namely, that Big Data approaches are differentiated by their ability to provide analytics from unstructured data sources and huge, disparate data sets (IBM and others refer to this as the 4Vs: Volume, Velocity, Variety, & Veracity).

Big Data solutions have other traits that enhance their effectiveness, better unlocking insights than legacy solutions. For example, many solutions are capable of certain types of machine learning – suggesting or executing a particular course of action based on historical actions, rather than as a result of formally coded rules. As another example, Big Data solutions will often consume not just event-based sources, but also intelligence feeds or contextual reference data (e.g., threats, vulnerabilities, asset inventories) for better overall insights.

Industry Thinking

What this all means is that classifying a solution as Big Data Security Analytics is guided by principles rather than a distinct definition, and hence, is somewhat subjective. In fact, many respected analysts, such as Anton Chuvakin at Gartner, don’t believe “there is such a market at this time” (note: Gartner does believe that BDSA is relevant as a concept). 

Many enterprises that we interact with are indeed trying Big Data Security Analytics in some form or fashion (to say nothing of the preponderance of start-up marketing materials). As Larry Lunetta of PetaSecure reminds us about SIEM, “it took Gartner 2 years to publish a [Magic Quadrant], and the Leader’s quadrant for the inaugural version was empty.” This is to say nothing ill about our friends at Gartner; we’d argue it’s just early in the game.  Other analysts such as Jon Oltsik of the Enterprise Strategy Group believe that “there is no longer any debate – security analytics has become a big data application.” We concur.

Proposal

With that in mind, we’d like to propose an initial Big Data Security Analytics landscape; that is, the intersection of “Big Data Analytics” with “Security Analytics”. We fully expect it’ll have gaps, approximations, and maybe more egregious errors. But let’s start somewhere. Our central organizing theme is that BDSA solutions are either real-time or historical, depending upon the data on which they operate:

Real-time solutions (“Real-time Analytics”) live on the network and often perform automated remediation to prevent ongoing attacks.

Historical solutions (“Historical Analytics”) rely more on batch data and provide rich investigative capabilities for security analysts.

Within the Real-time Analytics space, the clearest delineator is whether the solutions perform deep packet inspection or not. There are trade-offs for each approach. Within the Historical Analytics space, solutions are typically provided either as all-in-one, self-contained implementations or are meant to be deployed on top of existing Big Data repositories. Hence, our sub-categories. 

The Landscape

For the purposes of this landscape, we assume the following are out-of-scope:

Classic SIEM and scalable “SIEM 2.0” solutions.  The latter refers to SIEM enhancements that provide for Volume and Velocity but still require structured datasets and/or do not incorporate non-log based sources.

Solutions that only provide a general platform for Analytics, rather than anything security-specific.

Solutions that only enable inbound data collection or ingestion, but do not perform analytics.  Similarly, solutions that only provide workflow, case management, or outbound response.

Public companies (and Palantir), since we’re focused on startups.

Final Thoughts

Big Data Security Analytics is an emerging market and we're certainly excited to see how it evolves. In the short-term, we're interested to know which companies we missed. We'll also be monitoring how well these companies gain traction with enterprise customers. Longer-term, it will be interesting to see which approaches prove most effective at detecting and preventing attackers. Please share your thoughts in the comments!

An Evolution Beyond Security Information & Event Management

By Jed LeidheiserJuly 24, 2014

Limitations of SIEM

Depending on which company or startup we speak with, Security Information & Event Management (SIEM) is either dead or will live on forever. Quite different answers. In our minds, Big Data Analytics represents an evolution – not revolution – beyond the aggregation, alerts, and response facilitated by a classic SIEM solution. Big Data approaches differ from SIEM in two key ways: 1) unstructured data is acceptable, and 2) huge datasets are no longer a challenge. Of course, #1 and #2 resulted from new technologies we’ve spoken about before, which were created for purposes other than security.

Enterprises now realize that complete prevention of security incidents is impossible. Instead, there must be an increased focus on timely detection and response. Breaches WILL HAPPEN – so find them and contain them quickly. Both classic SIEM and Big Data approaches are compatible with this mindset and seek to unlock value through the aggregation and analysis of events generated by disparate systems. The problem is that SIEM promised the world but under delivered. Verizon’s 2013 Data Breach Investigations Report provides an indication of this, noting that only about 1% of data breaches were discovered through log review. This is due to tools being configured only for certain use cases, monitoring teams being understaffed and undertrained, and the sheer volume of activity, among other reasons.

More directly, Gartner found that many organizations have “genuinely outgrown their SQL-based SIEM and moved to Hadoop-based systems.” This has been publicly confirmed by CISOs at companies such as IDT and Jeffries & Company at the 2014 RSA Conference (see TechTarget write-up), who found that existing SIEM solutions simply failed from a performance standpoint under Big Data volume.

What must enterprises do now?  

To start, existing SIEM solutions still form part of a security monitoring and analytics capability. The events collected and normalized by an existing SIEM must feed into a security-focused environment for Big Data analytics. This environment will also aggregate security events from solutions such as Data Loss Prevention (DLP) and Identity & Access Management (IAM), as well as data sources related to vulnerabilities, threats, and asset inventories (e.g., business processes, applications, databases, and hosts). Enterprises must start their Big Data Security Analytics focus now. A few points to remember:

Start Small: build environments along focused use cases

Educate: select/hire initial team members and encourage training of both Big Data technologies and data science concepts

Remember Quality: the principle “garbage in, garbage out” still applies to the data being aggregated

Integrate and Centralize: communicate with technology teams and business areas in order to gain and automate access to data

Secure: remember that access controls and other security concepts still apply

Report: determine how insights can be easily reported to management

Where do we go from here?

Managing Big Data Environments

By Jed LeidheiserJuly 8, 2014

Let’s face it, Big Data is here to stay. With its buzzword-friendly use rampant in media and vendor sales decks alike, the conversation is often focused on being able to quickly and efficiently unlock business insights from large, siloed data sets. But what about the security of the massive amount of information collected under the guise of Big Data?

Aggregating information into data warehouses is nothing new, and large organizations have historically collected large datasets using relational database management systems. Nowadays, these data warehouse technologies place a premium on security, with provisioning and both cell-level and table-level security built into their functionality.

However, there is marked difference in both the purpose and technology of newer Big Data environments from traditional data warehousing. The driver for Big Data environments is often customer analysis, security intelligence, healthcare research, or financial instrument analysis, rather than simply aggregating information into a data warehouse. The technical implementations are also new, with the likes of Hadoop, MongoDB, and HBase as key players. The purpose and technology keeps expanding, and in practice, data attracts more data. The term “data lake” - a data-centric environment where centralized and accessible information is the norm - is becoming more common.

What has been missing from the equation is adequate security. Big Data environments store massive amounts of information from multiple business units within an organization. That data is then mashed, mangled, and distributed across diverse analytic and visualization technologies. By the nature of their original business purpose (research or predictive analytics), Big Data environments are accessible. In fact, Hadoop was originally designed without proper security according to Kevin T. Smith on InfoQ: “anyone could submit code and it would be executed… access control was easily circumvented.”

The identity problem – validating that someone is who they say they are – is now being addressed by multiple vendors and products who offer Kerberos, PAM, and AD/LDAP integration. No longer should implementations create separate identities within the Big Data ecosystem… single sign-on is a reality. Now the focus must be on authorization (role-based access and fine-grained entitlements), as well as sensitive data handling (encryption and metadata-driven governance). These are the key security issues at hand for most enterprise implementations:

Authorization

No centralized access administration across different engines (e.g., HDFS, Hive, Hbase)

Only coarse entitlement granting capabilities (i.e., difficulty to grant fine-grained entitlements at the cell/file level)

Sensitive Data Handling

Limited support for encryption and/or data masking

Inadequate use of metadata, hampering sensitive data handling

In the authorization space, an employee may have different entitlements across HDFS, Hive, HBase, and the list goes on. When users have different entitlements across applications/tools with similar business purposes, it is difficult to determine if their access levels are appropriate for their job. This rings more true as employees change responsibilities – particularly, transitions without a formal title change – which leads to an accumulation of privileges. Fine-grained entitlement administration is also critical to an effective access management posture.  For example, HBase only provides control for tables and column families, but not to the cell level, which may house privacy data.

The handling of sensitive data is the other major concern. Encryption for data-at-rest is not available in native Hadoop, nor currently on platforms such as Hortonworks. In addition, the use of encryption or data masking relies on metadata about data elements, which often go untagged. Not only does this hamper access control and data handling, but audit logging and monitoring as well. All activity must be logged for both compliance and monitoring reasons, including when sensitive data has been accessed and by whom. If sensitive data is not tagged appropriately, it is difficult to detect inappropriate activity after the fact.

The big platforms – Hortonworks and Cloudera – have taken notice. Hortonworks acquired XA Secure in May of this year, while Cloudera quickly followed by acquiring Gazzang in June. These acquisitions certainly make sense.  Hopefully, these vendor platforms will address the missing gaps and deliver solutions that meet enterprise security needs.