MRUnit testing with Scala
I am running Hadoop in a pseudo distributed mode as it helps me a find a lot of problems way earlier than on the cluster. Also for all enough data sets I can just run the job locally. That downside is that there is no easy way to debug anything other than printf. Writing all the mapreduce code in Scala that I have been, at some point the printf debugging by running mapreduce jobs locally became very unproductive. Every single compile, run the job, cat the appropriate log folders and look at the debug information was approximately 10 mins.
I ran into MrUnit and the scarce documentation that surrounds it. Spent quite a few hours looking at all the documentation and figuring out what libraries to use etc. Finally I have it working on my box and it works really well. sbt compile itself will give me all the errors I need to see. The turnaround is less than a minute - 10x better!!
Pasting relevant code below
import org.junit.Test import scala.collection.mutable import org.apache.hadoop.io.Text import org.apache.hadoop.mapreduce.Mapper
import org.apache.hadoop.mapreduce.Reducer import org.apache.hadoop.mrunit.mapreduce._ class MyMapReduceClassTest { implicit def StringToText(v: String): Text = new Text(v) val mapDriver = new MapDriver[java.lang.Object, Text, Text, Text] val reduceDriver = new ReduceDriver[java.lang.Object, Text, Text, Text] val mapReduceDriver = new MapReduceDriver[java.lang.Object, Text, Text, Text, Text, Text] trait TestMapper extends Mapper[Object,Text,Text,Text] with MyMapReduceTrait{ override def map(key: Object, row: Text, context: Mapper[Object,Text,Text,Text]#Context) = { mymappingFunction(key, row, context) // this is implemented inside MyMapReduceTrait
} } trait TestReducer extends Reducer[Text,Text,Text,Text] with MapperReducerBase { override def reduce(key: Text, values: java.lang.Iterable[Text], context: Reducer[Text, Text, Text, Text]#Context) = { myReducingFunction(key, values, context) // imeplemented inside MyMapReduceTrait } } // end reduce class @Test def test1() = { val mapper = new TestMapper val reducer = new TestReducer mapReduceDriver.setMapper(mapper) mapReduceDriver.setReducer(reducer) val key = None mapReduceDriver.withInput(new Text("test key"), new Text("test value")) mapReduceDriver.withOutput(new Text("test output key"), new Text("test output values")) mapReduceDriver.runTest() } }
