#mrunit

MRUnit 1.0.0 버전에서는 MultipleOutputs 를 사용한 M/R 코드를 테스할 수 있는 기능이 제공되지 않는다. 하지만 이와 관련된 이슈 MRUnit-13 를 살펴보면 기능 추가가 완료되었고, 다음 버전(1.1.0)에 추가되어 나올 것이라는 걸 알 수 있다.

하지만, 당장 Unit test 를 작성하고 검증하는데 이용해야 하는 상황이라면 개발 중인 버전을 받아 사용해볼 수 있다.

MRUnit 설치 전 환경 구축

프로젝트 빌드 환경은 Mac OS X 10.9 Mavericks 이며, Maven 2 와 Java 7 을 이용한다.

Maven 2 설치 (Maven 3 으로는 빌드가 되지 않았음)

$ brew install maven2

Java 7 설치

http://www.oracle.com/technetwork/java/javase/downloads/index.html

의존성을 가진 라이브러리 (Maven Repository)

junit 4.10

guava 11.0.2

hadoop-common 2.3.0

hadoop-mapreduce-client-core 2.3.0

powermock-core 1.5.1

powermock-api-mockito 1.5.1

powermock-module-junit4 1.5.1

powermock-module-junit4-common 1.5.1

MRUnit 설치

SOURCE AND GUIDE 페이지 참조.

저장소에서 최신 소스 코드 다운로드

$ git clone http://git-wip-us.apache.org/repos/asf/mrunit.git

Maven 을 이용해 프로젝트 빌드

$ mvn package -Dhadoop.version=2 -Drat.numUnapprovedLic enses=128

빌드에 성공하면 target 디렉토리 밑에서 mrunit-1.2.0-SNAPSHOT-hadoop2.jar 파일을 찾을 수 있다. (버전 정보는 달라질 수 있음)

Unit test 작성

Unit test 를 작성할 때 mrunit/src/test 밑에 있는 코드를 참조한다. (MultipleOutputs 에 대한 테스트는 test/java/org/apache/hadoop/mrunit/mapreduce/TestMultipleOutput.java 참조)

@RunWith(PowerMockRunner.class) @PrepareForTest({MultipleOutputs.class, MyMap.class}) public class TestMyMap { private MapDriver<LongWritable, Text, Text, IntWritable> mapDriver; ... @Test public void testMyMapWithMultipleOutputs() throws Exception { mapDriver = MapDriver.newMapDriver(new MyMap()); mapDriver.withInput(new LongWritable(0), new Text("first")); mapDriver.withMultiOutput("OUT1", new Text("first"), new IntWritable(1)); mapDriver.runTest(); } ... }

@PrepareForTest() 안에 MultipleOutputs 클래스와 테스트하고자 하는 Mapper, Reducer 를 명시해둔다.

Mansoor Ashraf about MRUnit:

Testing and debugging multi threaded programs is hard. Now take the same programs and massively distribute them across multiple JVMs deployed on a cluster of machines and the complexity goes off the roof. One way to overcome this complexity is to do testing in isolation and catch as many bugs as possible locally. MRUnit is a testing framework that lets you test and debug Map Reduce jobs in isolation without spinning up a Hadoop cluster. In this blog post we will cover various features of MRUnit by walking through a simple MapReduce job.

The code samples look quite legible and there doesn’t seem to be a lot of boilerplate code involved. That’s a great thing for a testing framework.

I am running Hadoop in a pseudo distributed mode as it helps me a find a lot of problems way earlier than on the cluster. Also for all enough data sets I can just run the job locally. That downside is that there is no easy way to debug anything other than printf. Writing all the mapreduce code in Scala that I have been, at some point the printf debugging by running mapreduce jobs locally became very unproductive. Every single compile, run the job, cat the appropriate log folders and look at the debug information was approximately 10 mins.

I ran into MrUnit and the scarce documentation that surrounds it. Spent quite a few hours looking at all the documentation and figuring out what libraries to use etc. Finally I have it working on my box and it works really well. sbt compile itself will give me all the errors I need to see. The turnaround is less than a minute - 10x better!!

Pasting relevant code below

import org.junit.Test import scala.collection.mutable import org.apache.hadoop.io.Text import org.apache.hadoop.mapreduce.Mapper

import org.apache.hadoop.mapreduce.Reducer import org.apache.hadoop.mrunit.mapreduce._ class MyMapReduceClassTest  {     implicit def StringToText(v: String): Text = new Text(v)     val mapDriver = new MapDriver[java.lang.Object, Text, Text, Text]     val reduceDriver = new ReduceDriver[java.lang.Object, Text, Text, Text]     val mapReduceDriver = new MapReduceDriver[java.lang.Object, Text, Text, Text, Text, Text]     trait TestMapper extends Mapper[Object,Text,Text,Text]  with MyMapReduceTrait{         override def map(key: Object, row: Text, context: Mapper[Object,Text,Text,Text]#Context) = {             mymappingFunction(key, row, context) // this is implemented inside MyMapReduceTrait