Comments on: Programming MapReduce Jobs with HDInsight Server for Windows

By: venkat01

venkat01 — Fri, 24 May 2013 09:14:04 +0000

By using the above Mapper and Reducer code and adding 2 more files, we can run the code directly from VS2012.
public class TemperatureCalcJob : HadoopJob
{
public TemperatureCalcJob()
{
}
public override HadoopJobConfiguration Configure(ExecutorContext context)
{

HadoopJobConfiguration config = new HadoopJobConfiguration();

config.InputPath = “/user/venkat01/input/Temp/input.txt”;

config.OutputFolder = “/user/venkat01/output/Temp”;

return config;

}

class Program
{
static void Main(string[] args)
{
var hadoop = Hadoop.Connect();
var result = hadoop.MapReduceJob.ExecuteJob();
}
}
Enjoy!!!!!

By: anushree

anushree — Fri, 22 Mar 2013 10:27:28 +0000

A MapReduceJob is a class derieved from HadoopJob I created this class

public class TemperatureCalcJob:HadoopJob

{

public override HadoopJobConfiguration Configure(ExecutorContext context)

{

HadoopJobConfiguration config = new HadoopJobConfiguration();

config.InputPath = context.Arguments[0];

config.OutputFolder = context.Arguments[1];

return config;

}

and the command to run the job will be

mrrunner -dll “D:\Projects\Xurch\FirstHDInsightApp\FirstHDInsightApp\bin\Debug\FirstHDInsightApp.dll” — input/Temp output

By: mouna

mouna — Wed, 06 Mar 2013 15:23:37 +0000

Hello!

when i tried to execute the job:

D:\MyApp\Hadoop\MapReduce\FirstJob\bin\Debug>.\mrlib\mrrunner -dll FirstJob.dll

i get: “ERROR: No classes in DLL derive from MapReduce Job. ”

Any idea about the error?

thank you!

By: leifjohansson

leifjohansson — Wed, 06 Feb 2013 13:48:40 +0000

Hi,

I follow the example, but I never recive the row: INFO streaming.StreamJob: map 100% reduce 100%

Do anyone know what the problem is? I run the HDInsight on a local windows 7 pc.

some rows from the cmd:
packageJobJar: [c:\Hadoop\GettingStarted\Binaries\MRLib\Microsoft.Hadoop.MapDriv
er.exe, c:\Hadoop\GettingStarted\Binaries\MRLib\Microsoft.Hadoop.ReduceDriver.ex
e, c:\Hadoop\GettingStarted\Binaries\MRLib\Microsoft.Hadoop.CombineDriver.exe, C
:\Slask\Hadoop3\MapReduce\FirstJob\bin\Debug\FirstJob.dll, C:\Slask\Hadoop3\MapR
educe\FirstJob\bin\Debug\Microsoft.Hadoop.MapReduce.dll, C:\Slask\Hadoop3\MapRed
uce\FirstJob\bin\Debug\Newtonsoft.Json.dll] [/C:/Hadoop/hadoop-1.1.0-SNAPSHOT/li
b/hadoop-streaming.jar] C:\Users\s21725\AppData\Local\Temp\streamjob471967588828
6073633.jar tmpDir=null
13/02/06 14:43:11 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
ry for your platform… using builtin-java classes where applicable
13/02/06 14:43:11 WARN snappy.LoadSnappy: Snappy native library not loaded
13/02/06 14:43:11 INFO mapred.FileInputFormat: Total input paths to process : 1
13/02/06 14:43:12 INFO streaming.StreamJob: getLocalDirs(): [c:\hadoop\hdfs\mapr
ed\local]
13/02/06 14:43:12 INFO streaming.StreamJob: Running job: job_201302061258_0006
13/02/06 14:43:12 INFO streaming.StreamJob: To kill this job, run:
13/02/06 14:43:12 INFO streaming.StreamJob: C:\Hadoop\hadoop-1.1.0-SNAPSHOT/bin/
hadoop job -Dmapred.job.tracker=localhost:50300 -kill job_201302061258_0006
13/02/06 14:43:12 INFO streaming.StreamJob: Tracking URL: http://127.0.0.1:50030
/jobdetails.jsp?jobid=job_201302061258_0006
13/02/06 14:43:13 INFO streaming.StreamJob: map 0% reduce 0%
13/02/06 14:43:28 INFO streaming.StreamJob: map 100% reduce 0%