Excellent tutorial, the best for Hadoop beginners, I am 5 days trying to "swim" between different bugs and configurations when it comes running in the Windows platform. The CMD execution went fine, no errors. The only problem that I have when trying to run it through Eclipse. The error that I get is as follows:
Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:421)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:277)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at Recepti.main(Recepti.java:30)
This happens when I use jdk 7 for runtime/compilation. When using jdk 6, onether error show up:
java.lang.UnsupportedClassVersionError: Recipe : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Exception in thread "main"
Please suggest what could be the problem, so I can continue further with Hadoop programming.
Do I have to sign up in box for me to be able to download the hadoop-2.3.0.tar.gz file because it is not downloadable and you have not provided another link so far
Hello sir, I have a problem. Iam trying to run a similar mapreduce program. I have followed all your steps for running in windows. It worked perfectly fine untill i gave this command
hadoop jar c:\dcproject\Myprogram.jar JobName /input /out
The job got submitted but the mapping did not start. and the cmd kept on waiting for the mapping to start, which never happened
The dataset I used is a common CSV data set. so i think i do not need any extra jars. Can you please help out as soon as possible?
I also tried running the same Recipe program, and the result was same.. I just followed all the steps from the video.. PLease help out..
Thankyou
Could you pleas help me out in this issue .
Please find error .
hadoop-2.3.0\sbin>hadoop fs -copyFromLocal c:\hwork\recipeitems-latest.json /in
11/23 10:55:08 WARN hdfs.DFSClient: DataStreamer Exception
.apache.hadoop.ipc.RemoteException(java.io.IOException): File /in/recipeitems-latest.json._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) runn
and no node(s) are excluded inthis operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1406)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2596)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:563)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:407)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
at org.apache.hadoop.ipc.Client.call(Client.java:1406)
at org.apache.hadoop.ipc.Client.call(Client.java:1359)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy9.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy9.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:348)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1264)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1112)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:522)
yFromLocal: File /in/recipeitems-latest.json._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded inthis opera
n.
hadoop-2.3.0\sbin>
It's Showing datanode is n't running. please follow this commands
1. > jps
datanode
namenode
resource manager
node manager
2. format the hadoop namenode
> hadoop namenode -format
3. you configure the configuration files https://github.com/prabaprakash/Hadoop-2.3-Config[^]
4. You using windows 7/8/10 64 bit with jdk 6
...........
where is
>node manager
>resource manager
services is still missing.
open cmd
> cd c:\hadoop\sbin\
> start-yarn.cmd
> start-dfs.cmd
>jps
datanode
namenode
node manager
resource manager
the four is necessary to run a hadoop mapreduce...............
Hi Praba,
Thanks for your grate article and video... its very helpful for the installation
I too got the same error "Unsupported major.minor version 51.0" while eclipse plugin implementation. i am using java 6.1 (41) is this causing this error ? Any suggestion ?
I have a question regarding the replication of the system.
The replication is set to default which is 3. As I started uploading a file with 4 nodes in the cluster, only 3 nodes where given the replication. Why is that? And shouldn't the replication of the chunks be skipping?
Exception in thread "main"0: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:232)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:627)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:598)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:179)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at Recipe.main(Recipe.java:85)
main class code:
Java
publicstaticvoid main(String[] args) throws Exception {
Configuration conf = new Configuration();
/* String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
for ( String string : otherArgs) {
System.out.println(string);
}
if (otherArgs.length != 2) {
System.err.println("Usage: recipe <in> <out>");
System.exit(2);
}*/@SuppressWarnings("deprecation")
Job job = new Job(conf, "Recipe");
job.setJarByClass(Recipe.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// FileInputFormat.addInputPath(job, new Path(otherArgs[0]));// FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
FileInputFormat.addInputPath(job, new Path("hdfs://127.0.0.1:9000/user/hadoop/in/"));
FileOutputFormat.setOutputPath(job, new Path("hdfs://127.0.0.1:9000/user/hadoop/out/"));
//System.exit(job.waitForCompletion(true) ? 0 : 1);
job.submit();
}
Paths are fine as hadoop fs -ls /user/hadoop/in returns json file.
Help plz
I am facing the same exception like below . when i run the java program through eclipse
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" 0: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod(NativeIO.java:232)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:627)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:598)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:179)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at Recipe.main(Recipe.java:87)
From commandline, i am able to run the program , and i got the output . But i am getting error while running through eclipse
Hi Praba, thanks for sharing such and informative guide. I would also like to share my views-
With HDP for Windows and HDInsight Service there is unprecedented choice for Windows enterprises for their Hadoop deployments. HDP for Windows is the Microsoft recommended way to deploy Hadoop on Windows Server environments. For cloud-based deployments HDInsight Service is a 100% compatible and scalable environment for deploying your Hadoop based applications.
Would also like to suggest the newbies they can visit here also for more information-https://intellipaat.com/
Dear Prabha while i am installation of hadoop it is displaying
c:\hadoop-2.3.0\bin>hadoop namenode -format
The system cannot find the path specified.
Error: JAVA_HOME is incorrectly set.
Please update C:\hadoop-2.3.0\conf\hadoop-env.cmd
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
The system cannot find the path specified.
Error: JAVA_HOME is incorrectly set.
Please update C:\hadoop-2.3.0\conf\hadoop-env.cmd
'-Djava.net.preferIPv4Stack' is not recognized as an internal or external command,
operable program or batch file.
I have done every thing what you said above and JAVA path also correctly set.
Please help in this regard.