(via https://www.youtube.com/watch?v=6VpiJK8Jydw)

IBM is presenting at data.bythebay.io!

Scalable machine learning is ubiquitous in virtually every industry ranging from insurance, manufacturing, finance, and health sciences. Expressing and and running machine learning algorithms for varying data characteristics and at scale is challenging. In this talk, we will discuss our experience in building Apache SystemML, peak at challenging optimization and implementation strategies in exploiting data-parallel platforms such as MapReduce and Spark, and provide performance and scalability insights.

(via https://www.youtube.com/watch?v=myFKoAXYiHI)

(via https://www.youtube.com/watch?v=AmusbnxOADM)

In this talk, we will discuss how we use Spark as part of a hybrid RDBMS architecture that includes Hadoop and HBase. The optimizer evaluates each query and sends OLTP traffic (including CRUD queries) to HBase and OLAP traffic to Spark. We will focus on the challenges of handling the tradeoffs inherent in an integrated architecture that simultaneously handles real-time and batch traffic. Lessons learned include: - Embedding Spark into a RDBMS - Running Spark on Yarn and isolating OLTP traffic from OLAP traffic - Accelerating the generation of Spark RDDs from HBase - Customizing the Spark UI The lessons learned can also be applied to other hybrid systems, such as Lambda architectures.

(via https://www.youtube.com/watch?v=xqyMie8AO3Y)

This talk focuses on the Data type class: an abstraction for expressing a common, useful set of data transformation and manipulation actions. The code, in it’s design and implementations, make heavy use of rich, expressive types and functional principles.

We’ll dive into the code, show off some of the quirks when getting the abstraction to work with both Spark RDDs and Flink’s DataSet type. We'll also demonstrate code using this type class, showing how we can write once and run on different platforms!

Malcolm Greaves a computer scientist and software engineer that adores functional programming, supports open source development, works on machine learning (ML) and natural langague processing (NLP) algorithms, and never stops innovating and growing.

(via https://www.youtube.com/watch?v=RBRku-afibs)

SF Spark organizer Alexy Khrabrov talks with the community veteran Alexander Ulanov about the intricacies of deep learning implementation on Spark.

(via https://www.youtube.com/watch?v=pNYBBhuK8yU)

Artificial neural networks (ANN) are one of the popular models of machine learning, in particular for deep learning. The models that are used in practice for image classification and speech recognition contain huge number of weights and are trained with big datasets. Training such models is challenging in terms of computation and data processing. We propose a scalable implementation of deep neural networks for Spark. We address the computational challenge by batch operations, using BLAS for vector and matrix computations and reusing the memory for reducing garbage collector activity. Spark provides data parallelism that enables scaling of training. As a result, our implementation is on par with widely used C++ implementations like Caffe on a single machine and scales nicely on a cluster. The developed API makes it easy to configure your own network and to run experiments with different hyper parameters. Our implementation is easily extensible and we invite other developers to contribute new types of neural network functions and layers. Also, optimizations that we applied and our experience with GPU CUDA BLAS might be useful for other machine learning algorithms being developed for Spark.

(via https://www.youtube.com/watch?v=bnJv8sV-UA0)

Many people are turning to the SMACK stack (Spark/Scala, Mesos, Akka, Cassandra, Kafka) for real-time stream processing applications, but stick with Parquet for ad hoc analysis of static data. What if you would like flexible queries on real time streaming data, plus historical data? Do you have to set up two giant big data stacks? No! FiloDB is a new open-source distributed, versioned, and columnar analytical database designed for modern streaming workloads.Come check out how it can enable flexible ad-hoc analytics for a new generation of real-time Spark and Spark Streaming applications. Have the performance of Parquet with the ease of use of NoSQL databases for streaming applications.

(via https://www.youtube.com/watch?v=0WMFaxOgybM)

The talk focuses on the use of Spark to power an interactive query platform. We will discuss our architecture that uses Spark on top of HBase and Aerospike to enable fast and flexible queries over large amounts of data. We will also discuss some issues we ran up against when executing very short jobs (a few seconds) with Spark.