SAMBER VALLEY's Blog

One of the goals of our recently launched (and awesome!) new Flickr iPhone app was to further increase user engagement on Flickr. One of the best ways to drive engagement is to make sure Flickr users know what’s happening on Flickr in as near-real time as possible. We already have email notifications, but email is no longer a good mechanism for real-time updates. Users may have many email accounts and may not check in frequently causing timeliness to go right out the window. Clearly this called for… PUSH NOTIFICATIONS!

Oracle had better watch its back. There’s a new(ish) database player on the market that wants to eat its lunch; dinner, breakfast and dessert too, for that matter. Say hello to MongoDB. In case you haven’t heard of MongoDB yet, open your ears and open your eyes. The Big Data, NoSQL, database technology which has been loved by developers over the past few years is likely coming to a company like yours.

JavaScript is the programming language that makes a website interactive. Slideshows, advertising pop-ups and Google’s autocomplete feature are all examples of JavaScript at work. It was first created by Brendan Eich at Netscape in 1995 — nicknamed Mocha during development, released in beta as LiveScript and ultimately named JavaScript in order to piggyback on the popularity of Java (another programming language) for marketing reasons. At first, developers didn’t take JavaScript seriously, because it wasn’t seen as a serious development language like Java, Ruby or Python, which are server-side languages. JavaScript was the frosting on the cake, only responsible for user experience. But the language continues to become more prolific, often recommended as a first language to learn for beginners. If you use JavaScript with a framework called Node.js, you can now actually use JavaScript as a server-side language.

As organizations invest in analytics, predictive analytics is becoming a focus area. These advanced, mathematically based approaches are a powerful tool for leveraging data. Meanwhile, organizations are tackling the challenges of big data, handling more data in more formats that arrives and changes more quickly than ever before. As this tidal wave of data comes crashing down, traditional approaches to storing and managing data are being challenged. So, too, is the mindset that data must be queried and reported or visualized to be used. This combination of predictive analytics and big data should create new opportunities for more specific, future-oriented analysis of large amounts of data which, in turn, leads to actionable insight.

In Part II of this series, we covered the architecture needed for persisting the Activity Streams to MongoDB and fanning it out in real-time to all the clients using Redis PubSub. Since then, some exciting new Node.js features for Cloud Foundry were launched. In addition, the MongoDB version on Cloud Foundry has been upgraded to 2.0. In this blog post we will cover how to: Use Mongoose-Auth to store basic user information, including information from Facebook, Twitter, and Github, and how we made this module with native dependencies work on Cloud Foundry Use Mongo GridFS and ImageMagick to store user uploaded photos and profile pictures Perform powerful stream filtering, thanks to new capabilities exposed in MongoDB 2.0 Update the UX of the app to become a real-time stream client using Bootstrap, Backbone.js and Jade.

Recently we’ve spoken to a number of people to find out how our real-time stuff could be of use to them. Those were all very interesting conversions, but sometimes it happened that it became more about how to fit our approach into an existing infrastructure than discussing the merits of our approach for a given application. I think it’s not uncommon if discussing software that the question of how to fit something on an existing framework is important, but for two reasons I think that’s even more the case with Big Data.

Real-time seems to be the next big thing in big data. Map-Reduced has shown how to perform big analyses on huge data sets in parallel, and the next challenge seems to be to find a similar kind of approach to real-time. When you look around the web, there are two major approaches out there which try to building something which can scale to deal with Twitter-firehose-scale amounts of data. One is starting with a MapReduce framework like Hadoop and somehow finagle real-time or at least streaming capabilities on it. The other approach starts with some event-driven “streaming” computing architecture and makes it scale on cluster.

"In the NoSQL space this kind of real-world data is still a bit vague," writes Todd Hoff at High Scalability. "When asked, vendors tend to give very general answers like NoSQL is good for BigData or key-value access. What does that mean for for the developer in the trenches faced with the task of solving a specific problem and there are a dozen confusing choices and no obvious winner? Not a lot." He's set out to help solve that problem by compiling a list of NoSQL use cases, both general and specific.

Today Nodeable launched a new service called StreamReduce, a cloud-hosted real-time big data analytics product. StreamReduce is based on the same architecture as Nodeable’s existing IT operations monitoring tool. The company is keeping its current service, but is expanding its scope by marketing beyond its current base of developers and system administrators. At the heart of StreamReduce is Storm, a real-time analytics engine that was originally developed at BackType, a company that was acquired by Twitter last year. After the acquisition, Twitter allowed lead developer Nathan Marz to finish the project and open source it. Twitter is now using Storm internally.

The NoSQL buzzword has been metastasizing for several years. The excitement about these fast data stores has been intoxicating, and we're as guilty as anyone of seeing the groundbreaking appeal of NoSQL. Yet the honeymoon is coming to an end, and it's time to start balancing our enthusiasm with some gimlet-eyed hard truths. Don't get us wrong. We're still running to try the latest experiment in building a simple mechanism for storing data. We still find deep value in MongoDB, CouchDB, Cassandra, Riak, and other NoSQL standouts. We're still planning on tossing some of our most trusted data into these stacks of code because they're growing better and more battle-tested each day.

Discussions, debates and articles about big data focus frequently on NoSQL systems such as Hadoop that handle large volumes of multistructured data. Whereas NoSQL systems are an important element of any big data strategy, they represent only one component of such a strategy. The objective of this article is to review the role of NoSQL systems in the processing of Big Data and to identify all of the components required to support a Big Data environment.

Decision! Decision! Decision! What a hazardous and difficult human endeavor it is! Those of us who had to make decisions in personal life, business or profession know that the chance of our decision producing the desired end-result is always in doubt. This is so mainly because decision made today fructify tomorrow. If all decision makers were clairvoyant no one would make wrong decisions – making decisions would be a routine job. Unfortunately this is never going to happen. We will keep making wrong decisions as we do today. However, we could enhance, in a substantial way, our chances of making the right decision by reviewing everything that is happening now or has happened in the past relating to our target area of activity. Of course, this may be a tall order to follow but in the information age that we are living in, it seems feasible. There is so much data available at diverse sources that if we evolve a scientific and feasible way of disseminating all this data we will find answers to our queries as we have never been able to do so before. The ‘way’ we mentioned above is the ‘Big Data Analytics’.

Global Pulse White Paper “Big Data for Development: Opportunities & Challenges” (May 2012) aims to highlight the opportunity, as well as some of the main concerns and challenges, raised by utilizing new, digital data sources in the field of international development, as concretely and openly as possible, and to suggest some ways forward.

For digital agencies, big data as a competitive advantage is still very nascent, somewhat terrifying, and not tangible at all. However, marketers are starting to hear that it’s the new secret sauce, and they’re scrambling to figure out how to use it. And for good reason. Given the current trajectory, there’s a large chance that big data will change the face of digital agencies in as little as five years. If you’re part of a digital agency, here’s what to consider.