Purpose

This document describes how to set up and configure a single-node Hadoop installation so that you can quickly perform simple operations using Hadoop MapReduce and the Hadoop Distributed File System (HDFS).

本文介绍配置单节点（本地程序，并非集群）hadoop的方法， 通过这个集群可以快速操作在hdfs文件系统下执行hadoop的mapruduce程序。

Prerequisites

  $ sudo apt-get install ssh
  $ sudo apt-get install rsync

centos上 请用  yum install ssh 和 yum install rsync

Download

To get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.

hadoop下载通过上面的 相关镜像站点下载， 也可以通过官方的归档站点下载， 可以下载到 全部各个版本。

Prepare to Start the Hadoop Cluster

Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:

解压缩hadoop应用程序。 在解压缩目录中， 编辑解压目录中 etc/hadoop/hadoop-env.sh文件， 在文件中设置下面的环境变量

  # set to the root of your Java installation
  export JAVA_HOME=/usr/java/latest

Try the following command:

  $ bin/hadoop

This will display the usage documentation for the hadoop script.

Now you are ready to start your Hadoop cluster in one of the three supported modes:

• Local (Standalone) Mode
• Pseudo-Distributed Mode
• Fully-Distributed Mode

Standalone Operation

By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.

默认的情况下， hadoop被配置为一个单个java进程的本地单机模式， 这个模式下可以方便调试程序（可以设置断点进行调试java代码，其他任何模式都不能方便调试程序）

The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.

下面代码拷贝解压后的conf目录中一些文件到  input目录（当前目录是hadoop的home目录）， 然后调用hadoop程序执行文件的正则查找工作， 查找完成后讲结果输出到output目录中

  $ mkdir input
  $ cp etc/hadoop/*.xml input
  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep input output 'dfs[a-z.]+'
  $ cat output/*

Pseudo-Distributed Operation

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

hadoop程序可以工作在单个节点上， 在这个节点上各个hadoop程序单独运行在独立的jvm中，这个模式就是伪分布模式

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

  $ ssh localhost

  $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
  $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
  $ export HADOOP\_PREFIX=/usr/local/hadoop

这里： 需要额外添加命令， 设置authorized_keys 文件 是600 否则， 没办法免密码登录

 chmod 600 .ssh/authorized_keys

然后在登录就是 免密码登录了！！

Execution

The following instructions are to run a MapReduce job locally. If you want to execute a job on YARN, see YARN on Single Node.

下面的指令在本地运行一个mapreduce程序， 若是要单个节点上运行yarn程序需要看另外一节内容

1. Format the filesystem:

格式化本地hdfs文件系统

  $ bin/hdfs namenode -format

2. Start NameNode daemon and DataNode daemon:

启动 namenode和datanode进程

  $ sbin/start-dfs.sh

The hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

hadoop进程输出日子到当前目录的logs目录中

3.Browse the web interface for the NameNode; by default it is available at:NameNode - http://localhost:50070/

可以通过浏览器 查看 http://localhost:50070/地址，来了解namenode工作情况。实践中可以根据情况调整localhost到真实机器的ip地址

4. Make the HDFS directories required to execute MapReduce jobs:

在hdfs中创建相关目录

  $ bin/hdfs dfs -mkdir /user
  $ bin/hdfs dfs -mkdir /user/<username>

5. Copy the input files into the distributed filesystem:

拷贝输入文件到分布式文件系统中

  $ bin/hdfs dfs -put etc/hadoop input

拷贝中发现， hdfs是没有当前目录的概念的， 每次都需要 绝对路径， 详细情况如上图。

6. Run some of the examples provided:

运行下面指令的相关程序，来在本地的伪分布模式中运行mapreduce程序

  $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep input output 'dfs[a-z.]+'

上面指令中的 目录没有指定 绝对路径， 实践中发现不工作，需要替换为下面：

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep /input /output 'dfs[a-z.]+'
7. Examine the output files: Copy the output files from the distributed filesystem to the local filesystem and examine them:

下面指令拷贝 hdfs文件中 output目录中文件到 本地文件的output目录中

  $ bin/hdfs dfs -get output output
  $ cat output/*

or

View the output files on the distributed filesystem:

或者直接查看 hdfs中相关文件

  $ bin/hdfs dfs -cat output/*

8. When you’re done, stop the daemons with:

当上面完成后， 可以用下面的指令停止伪分布集群

  $ sbin/stop-dfs.sh

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

  $ sbin/start-yarn.sh

  $ sbin/stop-yarn.sh