大数据技术——MapReduce词频统计

注：参考林子雨老师教程，具体请见

MapReduce编程实践(Hadoop3.1.3)_厦大数据库实验室博客

一.实验目的

1.理解Hadoop中MapReduce模块的处理逻辑｡

2.熟悉MapReduce编程｡

二.实验内容

1.新建文件夹input,并在其中创建三个指定文件名的文本文件,并将特定内容存入三个文本｡

2.启动Hadoop伪分布/全分布模式式,将input文件夹上传到HDFS上｡

3.编写MapReduce程序,实现单词出现次数统计｡统计结果保存到hdfs的output文件夹｡获取统计结果｡

三.实验环境

1.操作系统: Ubuntu 20.04(64 位)

2.虚拟机软件:VMware Workstation 15.5

3.jdk: jdk-8u202-linux-x64.tar.gz

4.Hadoop 版本 3.3.1

四.实验步骤

1.Eclipse中的项目创建及编写

 

import java.io.IOException;import java.util.Iterator;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount { public WordCount() { } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs(); if(otherArgs.length < 2) { System.err.println("Usage: wordcount [...] "); System.exit(2); } Job job = Job.getInstance(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(WordCount.TokenizerMapper.class); job.setCombinerClass(WordCount.IntSumReducer.class); job.setReducerClass(WordCount.IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); for(int i = 0; i < otherArgs.length - 1; ++i) { FileInputFormat.addInputPath(job, new Path(otherArgs[i])); } FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1])); System.exit(job.waitForCompletion(true)?0:1); } public static class TokenizerMapper extends Mapper { private static final IntWritable one = new IntWritable(1); private Text word = new Text(); public TokenizerMapper() { } public void map(Object key, Text value, Mapper.Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()) { this.word.set(itr.nextToken()); context.write(this.word, one); } } }public static class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); public IntSumReducer() { } public void reduce(Text key, Iterable values, Reducer.Context context) throws IOException, InterruptedException { int sum = 0; IntWritable val; for(Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) { val = (IntWritable)i$.next(); } this.result.set(sum); context.write(key, this.result); } }}

 2.编译

 3.程序运行

4.结果

 

五.完成情况与问题讨论

1.实际完成情况

对上述介绍的所有操作均进行了实验｡

2.问题与讨论

运行过程速度太慢,无法得出结果｡后检查发现原因在于本机硬件配置时资源分配不足,将分配资源增大后,运行成功｡

六.实践收获

1、在使用 Eclipse 运行 MapReduce 程序时,会读取 Hadoop-Eclipse-Plugin 的 Advanced parameters 作为 Hadoop 运行参数,如果我们未进行修改,则默认的参数其实就是单机(非分布式)参数,因此程序运行时是读取本地目录而不是 HDFS 目录,就会提示 Input 路径不存在｡所以我们需要将配置文件复制到项目中的 src 目录,来覆盖这些参数｡让程序能够正确运行｡

2、如果要再次运行WordCount.jar,需要首先删除HDFS中的output目录,否则会报错｡