#rhadoop

Big Data Analysis with R Programming and RHadoop

By Prathibha | M. Thillainayaki | A. Jenneth"Big Data Analysis with R Programming and RHadoop" 

Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-2 | Issue-4 , June 2018, 

URL: http://www.ijtsrd.com/papers/ijtsrd15705.pdf  

http://www.ijtsrd.com/computer-science/other/15705/big-data-analysis-with-r-programming-and-rhadoop/u-prathibha

open access journal of engineering, manuscript submission, social science journal

RHadoop Self learning Training Course

RHadoop Online Training Course provided by Easylearning. You get new opportunities to learn R and Hadoop with its integration. RHadoop course will cover most common ways to link R and Hadoop by introducing three packages RMR, RHDFS and RHbase. It gives you in-depth knowledge of using the typical concepts such as RHIVE and Hadoop streaming.

More Information visit-  http://easylearning.guru/course…

Install Rhadoop rhdfs and rmr2 on Hortonworks Sandbox

Install Rhadoop rhdfs and rmr2 on Hortonworks Sandbox

Big Data Analytics – R, rhdfs, rmr2, Rhive – Tutorial 2

Install rhdfs and rmr2 on Hortonworks Sandbox – Rhdfs rmr2 Installation and Tutorial

In the Previous tutorial Tutorial 1 we learned how to install R, so lets start second part of tutorial

First we need to install dependency packages required for rhdfs, run below command on R console

insta…

Interview Antonio Piccolboni Big Data Analytics RHadoop #rstats

Here is an interview with Antonio Piccolboni , a consultant on big data analytics who has most notably worked on the RHadoop project for Revolution Analytics. Here he tells us about writing better code, and the projects he has been involved with. DecisionStats(DS)- Describe your career journey from being a computer science student to one of the principal creators for RHadoop. What motivated…

HDFS permisson problem:

Disabling the dfs permission.

By adding below property code to conf/hdfs-site.xml

According to my previous post about RHadoop Installation on Cloudera QuickStart VM,my profile of real cluster is:

#/etc/profile export HADOOP_CMD=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/bin/hadoop export HADOOP_STREAMING=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.6.0.jar export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_HOME=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop export JAVA_HOME=/usr/java/jdk1.6.0_31/ export JAVA_LIBRARY_PATH=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/lib/native/ export HADOOP_LIBEXEC_DIR=/opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop/libexec/

Download 3 packages (rmr,rhdfs,rhbase) for RHadoop from https://github.com/RevolutionAnalytics/RHadoop/wiki/Downloads

According 2 the installation manual rmr must be installed on all the hadoop nodes,  rhdfs to be installed on NameNode.

But i installed all the packages and modified /etc/profile on all the nodes in order to avoid errors...

#update the source to fedoraproject and get R installed rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm yum install R #run commands in R to install the following #packages related to rhdfs and rmr > install.packages("rJava") > install.packages("reshape2") > install.packages("Rcpp") > install.packages("iterators") > install.packages("itertools") > install.packages("digest") > install.packages("RJSONIO") > install.packages("functional") > install.packages("bitops") > install.packages("caTools")

  #find the working DIR for /bin/hadoop and #find the working DIR /contrib/streaming/hadoop-streaming*.jar locate /bin/hadoop locate  /contrib/streaming #add HADOOP_CMD path and HADOOP_STREAMING path #to the environment profile (vi /etc/profile) export HADOOP_CMD=/usr/lib/hadoop/bin/hadoop export HADOOP_STREAMING=/usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.0.0-mr1-cdh4.4.0.jar export HADOOP_HOME=/usr/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf #enable the modify without reboot source /etc/profile #install rmr and rhdfs in the terminal R CMD INSTALL "rmr2_3.0.0.tar.gz" R CMD INSTALL "rhdfs_1.0.8.tar.gz"

  #run commands in R > library(rhdfs) HADOOP_CMD=/usr/lib/hadoop/bin/hadoop Be sure to run hdfs.init() > hdfs.init() 14/02/17 09:29:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable #enter the DIR of /usr/lib/hadoop/lib/native #and copy libhadoop.so and libhadoop.so.1.0.0 to /usr/lib64 cd /usr/lib/hadoop/lib/native cp libhadoop.so /usr/lib64 cp libhadoop.so.1.0.0 /usr/lib64 #add JAVA_LIBRARY_PATH to profile to solve the problem #vi /etc/profile export JAVA_LIBRARY_PATH=/usr/lib/hadoop/lib/native #enable the ENV modify once more and re-config R java source /etc/profile R CMD javareconf #run commands in R and try 2 load rhdfs and rmr2 again #generally speaking the problem could be solved > library(rhdfs) > library(rmr2)

TROUBLESHOOTING !!!

#add HADOOP_LIBEXEC_DIR to /etc/profile #enable the setting！ export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec source /etc/profile

The example finally ran...

Checked the output！

library(rhdfs) hdfs.init() library(rmr2) wordcount = function(input, output = NULL, pattern = '[[:punct:][:space:]]+'){   wc.map = function(., lines) {     keyval(unlist( strsplit( x = lines,split = pattern)),1)   }   wc.reduce =function(word, counts ) {     keyval(word, sum(counts))   }    mapreduce(input = input ,output = output, input.format = "text",             map = wc.map, reduce = wc.reduce,combine = T)   }

lines <- c('this is just a joke,', + 'and that is just another joke,', + 'we all like them very much,', + 'because they are so funny.')

to.dfs(lines,output='/user/hadoop/input/myjoke',format="text")

wordcount("/user/hadoop/input/myjoke",output="/usr/hadoop/output/mywordcount")

from.dfs("/usr/hadoop/output/mywordcount") 输出结果：