#graph analytics

Thinking of Models as Graphs

The first step in any big data visualization and analysis process is to ingest your data. In the past, most developers thought of models as rows with attributes and references to other row identifiers. In keeping with that mentality, Perspectives pulled data from a relational database into its session-scoped model. Relational social network data are shown in rows of elements and…

Neo4j: Beginner’s Guide to Graph Analytics | Udemy Coupon

Neo4j: Beginner’s Guide to Graph Analytics | Udemy Coupon

Get started with Neo4j without having to go through lengthy reference manuals What you’ll learn

How to set up Neo4j on your machine

Importing Datasets in Neo4j

Introduction to Graph Databases

Introduction to Cypher with Hands-On

Requirements

Basic understanding of Software Engineering concepts

Basic exposure…

Anyone steeped in the doctrine of relational databases will find that trying to use a graph database like Neo4J is painful and not at all intuitive. This is not your fault, or Neo4js fault, it’s just that graph traversal is nothing like SQL. When I say nothing, I literally mean nothing. You think about them in two completely different ways and the ergonomics of graph traversals are inherently harder to get used to. This issue is compounded when considering doing a tutorial on a graph database. Further, this is compounded when using a Graph Analytics library like GraphX. Already being forced to work with RDDs (Not exactly beginner friendly) adding the paradigm of graphs on top of it is too much for the uninitiated. What would be much easier to comprehend is if we could go from a table-like structure to a graph and do the same queries for comparison.

GraphFrames allow us to do exactly this. It’s an API for doing Graph Analytics on Spark DataFrames. This way, we can try to recreate SQL queries in Graphs and have a better grasp of the graph concepts. Not having to load the data and create the relationships makes a lot of difference in a pedagogical context (At least I’ve found).

A Simple Primer

To set this all up, we’re going to use the default example data found in the GraphFrames package with a few edits. It’s two tables that look like this:

In the second DataFrame, we have “src” and “dst” and “relationship” columns. This is just syntactic, and allows us to establish a vertex-edge relationship. You could make a pretty complex web of DataFrames that are connected to one another, but in order to maintain simplicity, I’ll just keep it as this simpler “friend/follow” relationship. It gives us enough data to go through the rest of this exercise without confusing us.

A Few Algorithms

We can start with PageRank, an algorithm developed by Larry Page, the CEO of Alphabet Inc. The basic idea is to establish how each edge in a graph references another. In the ancient web context, It would help us identify the authority on a topic. If every web page about Jay-z linked to Spotify.com then we’d know Spotify is an authority on Jay-z. For the data we have we'll look at the edges and it’s more a measure of connectedness:

You can look through the mathematical specification for a better understanding of what’s exactly going on, but essentially we built a DataFrame that described how each person was related to another. In a relational context, we would calculate the number of connections with a handful of queries, but as relationships get more numerous and complicated it becomes harder to do.

In a graph, there is a layer of abstraction that makes it easier to figure out this kind of information. Consider the following. If you were tasked with figuring out which of your friends knew each other, it would be a gargantuan task to call each and go through the list. It would be much easier if you could have each friend send their friends a message and for you to sort through the connections after. In a very oversimplified way, many of the algorithms in GraphFrames can be implemented with this message passive primitive.

For a more complicated example lets try the Strongly Conected Components algorithm. You can read through the math if you like but in laymen terms it’s a measure of each vertex in the graph being connected to another. From the definition it doesn’t have to be a direct connection, but the fewer hops to establish a connection the more “strongly connected” a vertex is. With that, we can use the GraphFrames implementation:

Again, figuring out this kind of information via SQL would be very hard. Largely because we don’t have semantics for figuring out connectedness, rather it’s great for collecting and summarizing information. Most of us don’t have an immediate need for graphs and what they have to offer. However, a lot can be uncovered if you can store your data in this way.

Nice Thing(s)

One of the kindest aspects of a library like GraphFrames is that edges and vertexes are Dataframes. This is valuable because we already have a whole set of APIs for how to deal with these things.

A second thing I like about GraphFrames are the algorithm implementations. There aren’t as many as GraphX but I feel like they are easier to use because they are dealing with DataFrames instead of RDDs. Many long-time Spark users are very familiar with RDDs and comfortable using them, I have been using Spark for a long time too, but always founded the DataFrames / DataSets to be more manageable.

Finally, querying GraphFrames is pretty nice! You have facilities to do regular search, breadth first search or structured queries. Breath first search is probably my favorite of the bunch:

Summary

I can’t say enough about how GraphFrames have enabled me to better understand graphs and graph analytics. It’s the first time I was able to successfully go from a column/row format to a graph and to compare the two. That being said, GraphFrames is very immature, as evidence by it’s release version and it’s lack of support for a number of features in GraphX or Apache Giraph. It’s immaturity is a blessing and no reflection of the quality and thought put into the API.

Different kinds of Graph Analysis

Path Analysis: This type of analysis can be used to determine the shortest distance between two nodes in a graph, for example. An obvious use case is route optimization that is particularly applicable to logistics, supply and distribution chains and traffic optimization for smart cities.

Connectivity Analysis: This type of graph analysis can be applied tor determining weaknesses in networks such as a utility power grid. It also enables comparing connectivity across networks.

Community Analysis: Distance and density–based analysis is used to find groups of interacting people in a social network, for example, and identifying whether they are transient and predicting if the network will grow.

Centrality Analysis: This analysis type enables identifying relevancy to find the most influential people in a social network, for example, or to find most highly accessed web pages—such as by using the PageRank algorithm.

Eliminating system bottlenecks with smart data acceleration

To many in the industry, system memory is viewed as little more than a silicon holding pen for temporarily storing program commands and data during execution. Nevertheless, the dramatic growth of Big Data – driven by the burgeoning Internet of Things (IoT) – has prompted a number of key industry players to re-examine the traditional role of memory in the data center.

To be sure, server…