Running a Wordcount Mapreduce example in Hadoop 2.4.1 Single-node Cluster in Ubuntu 14.04 (64-bit)

скачать 26 6 26 787
Добавлено 22 октября 2014, 15:45
Visit www.kishorer.in for more articles. This post link - In this video, I will show you how to run a Wordcount Mapreduce example in Hadoop 2.4.1 Single-node Cluster in Ubuntu 14.04 (64-bit). We use Eclipse IDE to create a JAR file and run the jar file using input file stored in HDFS (Hadoop File System). The final result is the count of every word in the input text file.

Комментарии

Swaraj Mohanty
[email protected]:~/hadoop/etc/hadoop$ hadoop dfs -mkdir -p /home/hduser/hadoop/input DEPRECATED: Use of this script to execute hdfs command is deprecatehd. hello sir, please help me out from this 16/09/07 10:28:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [email protected]:~/hadoop/etc/hadoop$ hadoop dfs -copyFromLocal /home/hduser/Desktop/sample.txt /home/hduser/hadoop/input DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. 16/09/07 10:29:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/09/07 10:29:55 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /home/hduser/hadoop/input/sample.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1441) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2702) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.ipc.Client.call(Client.java:1410) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525) copyFromLocal: File /home/hduser/hadoop/input/sample.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. [email protected]:~/hadoop/etc/hadoop$ cd /home/kishorer747/Desktop bash: cd: /home/kishorer747/Desktop: No such file or directory [email protected]:~/hadoop/etc/hadoop$ cd /home/hduser/Desktop [email protected]:~/Desktop$ hadoop jar wordcountprog.jar /home/hduser/hadoop/input /home/hduser/hadoop/output Exception in thread "main" java.lang.ClassNotFoundException: /home/hduser/hadoop/input at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.util.RunJar.main(RunJar.java:205) [email protected]:~/Desktop$ hdfs dfs -cat /home/hduser/hadoop/output/part-00000 16/09/07 10:32:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable cat: `/home/hduser/hadoop/output/part-00000': No such file or directory
7 сентября 2016, 08:06 Комменитровать
Educational Services
Hello kishore rajendra, I went through some of the videos on your YouTube channel and really enjoyed what I saw. I especially liked Running a Wordcount Mapreduce example in Hadoop 2.4.1 Single-node Cluster in Ubuntu 14.04 (64-bit). You should consider creating a training course on Udemy. My friend Marc Guberti is looking to add a course about Hadoop to the courses that he provides. He can guide you as you create a course about Hadoop and he will do all of the marketing once the course gets published. You would split all of the revenue from the Hadoop course 50/50 in exchange for the expertise that you provide in the training course. Please let me know if you are interested in creating a Hadoop that Marc will then promote to thousands of his students. Warm regards, Raveendra.
22 мая 2016, 11:42 Комменитровать
Black Swan
Hi, I have question to you. I have sentence for example: "I like spaggetti and tomato soup" and I would like split it by two words: "I like" "like spaggetti" "spaggetti and" "and tomato" "tomato soup". Could you describe me how to do this? of course in my file i will have more sentences -not only one.
8 апреля 2016, 16:48 Комменитровать
jamal aslam
i am getting these errors ..kindly help me Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0 at WordCount.run(WordCount.java:25) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at WordCount.main(WordCount.java:36)
27 декабря 2015, 19:34 Комменитровать
berkan sönmez
post link is broken
13 декабря 2015, 23:28 Комменитровать
rahul kumar
Hello Sir I followed the same steps which you have mentioned in this video but when I am running the command hadoop jar WordCount.jar /usr/local/hadoop/input /usr/local/hadoop/output then I am getting an error stating that "Error :Could not load or find main class jar" . Please help me in understanding what causing this issue. Thanks In Advance !!!
4 октября 2015, 17:32 Комменитровать
Disha Gabhane
Hello sir I follow the u r step but at the last step I.e out1 file not create it list out the input file by cmd hadoop ls cmd whrere is wrong
24 сентября 2015, 06:18 Комменитровать
kishore rajendra
+Udeet Solanki conent is any text. just copy paste a text paragraph from anywhere and you can save it as sample.txt Then the program will output each word and its frequency.
4 июля 2015, 08:08 Комменитровать
1
Udit Solanki
+kishore rajendra what is the content of sample.txt? can you post it here?
2 июля 2015, 15:21 Комменитровать
Sanjukta Mondal
Kishor, can you pls let me know if we deploy the generated jar file into data nodes across the cluster or only in name node ?
23 июня 2015, 11:58 Комменитровать
Sagar Kalburgi
Thank you so much! It was really simple and useful
24 апреля 2015, 08:38 Комменитровать
Niraj Kumar
Sir it would be too nice if you add some more tutorial... Thanks in advance...
4 апреля 2015, 17:17 Комменитровать
Niraj Kumar
Kishore Sir, Really very very #thanks.. you are superb!!!!
4 апреля 2015, 17:16 Комменитровать
Sebastien Maillet
nice tutorial, thanks. Would be nicer with a little bit of sound :o)
16 ноября 2014, 20:48 Комменитровать

Похожие видео