首页 博客列表 精选博客 每日焦点 关于我
[Hadoop] Windows 下的 Hadoop 2.7.5 环境搭建
发布日期：2021-05-09 05:15:31 浏览次数：23 分类：博客文章

本文共 20866 字，大约阅读时间需要 69 分钟。

原文地址：

准备说明：

jdk：jdk-8u161-windows-x64.exe

hadoop：hadoop-2.7.5.tar.gz

OS：Window 10

一、JDK 安装配置

详见：

二、Hadoop 安装配置

1、在   处下载 hadoop-2.7.5.tar.gz ；

2、将 hadoop-2.7.5.tar.gz 文件解压缩（以放在 D 盘根目录下为例）；

3、配置 HADOOP_HOME 环境路径；

并追加目录下的 bin 和 sbin 文件夹路径到 PATH 变量中；

4、在命令行窗口中输入 hadoop 命令进行验证；

如果提示 JAVA_HOME 路径不对，需要去修改 %HADOOP_HOME%\etc\hadoop\hadoop-env.cmd 里的配置：

set JAVA_HOME=%JAVA_HOME%@rem 修改为set JAVA_HOME=C:\Progra~1\Java\jdk1.8.0_161

 

三、Hadoop 配置文件

core-site.xml

       
        
            
     
      hadoop.tmp.dir
             
     
      /D:/hadoop/workplace/tmp
             
     
      namenode 上本地的 hadoop 临时文件夹
         
    	
            
     
      hadoop.name.dir
             
     
      /D:/hadoop/workplace/name
         
        
        
            
     
      fs.defaultFS
             
     
      hdfs://localhost:9000
             
     
      HDFS 的 URI，文件系统://namenode标识:端口号
         
        
            
     
      io.file.buffer.size
             
     
      131072

hdfs-site.xml

       
        
            
     
      dfs.replication
             
     
      1
             
     
      副本个数，配置默认是 3，应小于 datanode 服务器数量
         
        
            
     
      dfs.namenode.name.dir
             
     
      /D:/hadoop/workplace/name
             
     
      namenode 上存储 HDFS 命名空间元数据
         
        
            
     
      dfs.datanode.data.dir
             
     
      /D:/hadoop/workplace/data
             
     
      datanode 上数据块的物理存储位置
         
        
            
     
      dfs.webhdfs.enabled
             
     
      true
         
        
            
     
      dfs.permissions
             
     
      true
             
     
                  If "true", enable permission checking in HDFS.            If "false", permission checking is turned off,            but all other behavior is unchanged.            Switching from one parameter value to the other does not change the mode,            owner or group of files or directories.

mapred-site.xml

       
        
            
     
      mapreduce.framework.name
             
     
      yarn
         
        
            
     
      mapreduce.jobhistory.address
             
     
      localhost:10020
         
        
            
     
      mapreduce.jobhistory.webapp.address
             
     
      localhost:19888

yarn-site.xml

       
        
            
     
      yarn.nodemanager.aux-services
             
     
      mapreduce_shuffle
         
    	
           
     
      yarn.nodemanager.aux-services.mapreduce.shuffle.class
            
     
      org.apache.hadoop.mapred.ShuffleHandler

 

四、格式化 namenode

hadoop namenode –format 出现异常：

DEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.18/02/09 12:18:11 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable D:\hadoop\bin\winutils.exe in the Hadoop binaries.        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)        at org.apache.hadoop.util.Shell.
   
    (Shell.java:390)        at org.apache.hadoop.util.StringUtils.
    
     (StringUtils.java:80)        at org.apache.hadoop.hdfs.server.common.HdfsServerConstants$RollingUpgradeStartupOption.getAllOptionString(HdfsServerConstants.java:80)        at org.apache.hadoop.hdfs.server.namenode.NameNode.
     
      (NameNode.java:265)

下载 压缩包，解压并替换掉 hadoop\bin 目录下的文件，然后再重新格式化：

C:\Users\Memento>hadoop namenode -formatDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.18/06/07 06:25:02 INFO namenode.NameNode: STARTUP_MSG:/************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG:   host = PC-Name/IPSTARTUP_MSG:   args = [-format]STARTUP_MSG:   version = 2.7.5STARTUP_MSG:   classpath = D:\hadoop\etc\hadoop;D:\hadoop\share\hadoop\common\lib\activation-1.1.jar;D:\hadoop\share\hadoop\common\lib\apacheds-i18n-2.0.0-M15.jar;D:\hadoop\share\hadoop\common\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\hadoop\share\hadoop\common\lib\api-asn1-api-1.0.0-M20.jar;D:\hadoop\share\hadoop\common\lib\api-util-1.0.0-M20.jar;D:\hadoop\share\hadoop\common\lib\asm-3.2.jar;D:\hadoop\share\hadoop\common\lib\avro-1.7.4.jar;D:\hadoop\share\hadoop\common\lib\commons-beanutils-1.7.0.jar;D:\hadoop\share\hadoop\common\lib\commons-beanutils-core-1.8.0.jar;D:\hadoop\share\hadoop\common\lib\commons-cli-1.2.jar;D:\hadoop\share\hadoop\common\lib\commons-codec-1.4.jar;D:\hadoop\share\hadoop\common\lib\commons-collections-3.2.2.jar;D:\hadoop\share\hadoop\common\lib\commons-compress-1.4.1.jar;D:\hadoop\share\hadoop\common\lib\commons-configuration-1.6.jar;D:\hadoop\share\hadoop\common\lib\commons-digester-1.8.jar;D:\hadoop\share\hadoop\common\lib\commons-httpclient-3.1.jar;D:\hadoop\share\hadoop\common\lib\commons-io-2.4.jar;D:\hadoop\share\hadoop\common\lib\commons-lang-2.6.jar;D:\hadoop\share\hadoop\common\lib\commons-logging-1.1.3.jar;D:\hadoop\share\hadoop\common\lib\commons-math3-3.1.1.jar;D:\hadoop\share\hadoop\common\lib\commons-net-3.1.jar;D:\hadoop\share\hadoop\common\lib\curator-client-2.7.1.jar;D:\hadoop\share\hadoop\common\lib\curator-framework-2.7.1.jar;D:\hadoop\share\hadoop\common\lib\curator-recipes-2.7.1.jar;D:\hadoop\share\hadoop\common\lib\gson-2.2.4.jar;D:\hadoop\share\hadoop\common\lib\guava-11.0.2.jar;D:\hadoop\share\hadoop\common\lib\hadoop-annotations-2.7.5.jar;D:\hadoop\share\hadoop\common\lib\hadoop-auth-2.7.5.jar;D:\hadoop\share\hadoop\common\lib\hamcrest-core-1.3.jar;D:\hadoop\share\hadoop\common\lib\htrace-core-3.1.0-incubating.jar;D:\hadoop\share\hadoop\common\lib\httpclient-4.2.5.jar;D:\hadoop\share\hadoop\common\lib\httpcore-4.2.5.jar;D:\hadoop\share\hadoop\common\lib\jackson-core-asl-1.9.13.jar;D:\hadoop\share\hadoop\common\lib\jackson-jaxrs-1.9.13.jar;D:\hadoop\share\hadoop\common\lib\jackson-mapper-asl-1.9.13.jar;D:\hadoop\share\hadoop\common\lib\jackson-xc-1.9.13.jar;D:\hadoop\share\hadoop\common\lib\java-xmlbuilder-0.4.jar;D:\hadoop\share\hadoop\common\lib\jaxb-api-2.2.2.jar;D:\hadoop\share\hadoop\common\lib\jaxb-impl-2.2.3-1.jar;D:\hadoop\share\hadoop\common\lib\jersey-core-1.9.jar;D:\hadoop\share\hadoop\common\lib\jersey-json-1.9.jar;D:\hadoop\share\hadoop\common\lib\jersey-server-1.9.jar;D:\hadoop\share\hadoop\common\lib\jets3t-0.9.0.jar;D:\hadoop\share\hadoop\common\lib\jettison-1.1.jar;D:\hadoop\share\hadoop\common\lib\jetty-6.1.26.jar;D:\hadoop\share\hadoop\common\lib\jetty-sslengine-6.1.26.jar;D:\hadoop\share\hadoop\common\lib\jetty-util-6.1.26.jar;D:\hadoop\share\hadoop\common\lib\jsch-0.1.54.jar;D:\hadoop\share\hadoop\common\lib\jsp-api-2.1.jar;D:\hadoop\share\hadoop\common\lib\jsr305-3.0.0.jar;D:\hadoop\share\hadoop\common\lib\junit-4.11.jar;D:\hadoop\share\hadoop\common\lib\log4j-1.2.17.jar;D:\hadoop\share\hadoop\common\lib\mockito-all-1.8.5.jar;D:\hadoop\share\hadoop\common\lib\netty-3.6.2.Final.jar;D:\hadoop\share\hadoop\common\lib\paranamer-2.3.jar;D:\hadoop\share\hadoop\common\lib\protobuf-java-2.5.0.jar;D:\hadoop\share\hadoop\common\lib\servlet-api-2.5.jar;D:\hadoop\share\hadoop\common\lib\slf4j-api-1.7.10.jar;D:\hadoop\share\hadoop\common\lib\slf4j-log4j12-1.7.10.jar;D:\hadoop\share\hadoop\common\lib\snappy-java-1.0.4.1.jar;D:\hadoop\share\hadoop\common\lib\stax-api-1.0-2.jar;D:\hadoop\share\hadoop\common\lib\xmlenc-0.52.jar;D:\hadoop\share\hadoop\common\lib\xz-1.0.jar;D:\hadoop\share\hadoop\common\lib\zookeeper-3.4.6.jar;D:\hadoop\share\hadoop\common\hadoop-common-2.7.5-tests.jar;D:\hadoop\share\hadoop\common\hadoop-common-2.7.5.jar;D:\hadoop\share\hadoop\common\hadoop-nfs-2.7.5.jar;D:\hadoop\share\hadoop\hdfs;D:\hadoop\share\hadoop\hdfs\lib\asm-3.2.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-cli-1.2.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-codec-1.4.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-daemon-1.0.13.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-io-2.4.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-lang-2.6.jar;D:\hadoop\share\hadoop\hdfs\lib\commons-logging-1.1.3.jar;D:\hadoop\share\hadoop\hdfs\lib\guava-11.0.2.jar;D:\hadoop\share\hadoop\hdfs\lib\htrace-core-3.1.0-incubating.jar;D:\hadoop\share\hadoop\hdfs\lib\jackson-core-asl-1.9.13.jar;D:\hadoop\share\hadoop\hdfs\lib\jackson-mapper-asl-1.9.13.jar;D:\hadoop\share\hadoop\hdfs\lib\jersey-core-1.9.jar;D:\hadoop\share\hadoop\hdfs\lib\jersey-server-1.9.jar;D:\hadoop\share\hadoop\hdfs\lib\jetty-6.1.26.jar;D:\hadoop\share\hadoop\hdfs\lib\jetty-util-6.1.26.jar;D:\hadoop\share\hadoop\hdfs\lib\jsr305-3.0.0.jar;D:\hadoop\share\hadoop\hdfs\lib\leveldbjni-all-1.8.jar;D:\hadoop\share\hadoop\hdfs\lib\log4j-1.2.17.jar;D:\hadoop\share\hadoop\hdfs\lib\netty-3.6.2.Final.jar;D:\hadoop\share\hadoop\hdfs\lib\netty-all-4.0.23.Final.jar;D:\hadoop\share\hadoop\hdfs\lib\protobuf-java-2.5.0.jar;D:\hadoop\share\hadoop\hdfs\lib\servlet-api-2.5.jar;D:\hadoop\share\hadoop\hdfs\lib\xercesImpl-2.9.1.jar;D:\hadoop\share\hadoop\hdfs\lib\xml-apis-1.3.04.jar;D:\hadoop\share\hadoop\hdfs\lib\xmlenc-0.52.jar;D:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.7.5-tests.jar;D:\hadoop\share\hadoop\hdfs\hadoop-hdfs-2.7.5.jar;D:\hadoop\share\hadoop\hdfs\hadoop-hdfs-nfs-2.7.5.jar;D:\hadoop\share\hadoop\yarn\lib\activation-1.1.jar;D:\hadoop\share\hadoop\yarn\lib\aopalliance-1.0.jar;D:\hadoop\share\hadoop\yarn\lib\asm-3.2.jar;D:\hadoop\share\hadoop\yarn\lib\commons-cli-1.2.jar;D:\hadoop\share\hadoop\yarn\lib\commons-codec-1.4.jar;D:\hadoop\share\hadoop\yarn\lib\commons-collections-3.2.2.jar;D:\hadoop\share\hadoop\yarn\lib\commons-compress-1.4.1.jar;D:\hadoop\share\hadoop\yarn\lib\commons-io-2.4.jar;D:\hadoop\share\hadoop\yarn\lib\commons-lang-2.6.jar;D:\hadoop\share\hadoop\yarn\lib\commons-logging-1.1.3.jar;D:\hadoop\share\hadoop\yarn\lib\guava-11.0.2.jar;D:\hadoop\share\hadoop\yarn\lib\guice-3.0.jar;D:\hadoop\share\hadoop\yarn\lib\guice-servlet-3.0.jar;D:\hadoop\share\hadoop\yarn\lib\jackson-core-asl-1.9.13.jar;D:\hadoop\share\hadoop\yarn\lib\jackson-jaxrs-1.9.13.jar;D:\hadoop\share\hadoop\yarn\lib\jackson-mapper-asl-1.9.13.jar;D:\hadoop\share\hadoop\yarn\lib\jackson-xc-1.9.13.jar;D:\hadoop\share\hadoop\yarn\lib\javax.inject-1.jar;D:\hadoop\share\hadoop\yarn\lib\jaxb-api-2.2.2.jar;D:\hadoop\share\hadoop\yarn\lib\jaxb-impl-2.2.3-1.jar;D:\hadoop\share\hadoop\yarn\lib\jersey-client-1.9.jar;D:\hadoop\share\hadoop\yarn\lib\jersey-core-1.9.jar;D:\hadoop\share\hadoop\yarn\lib\jersey-guice-1.9.jar;D:\hadoop\share\hadoop\yarn\lib\jersey-json-1.9.jar;D:\hadoop\share\hadoop\yarn\lib\jersey-server-1.9.jar;D:\hadoop\share\hadoop\yarn\lib\jettison-1.1.jar;D:\hadoop\share\hadoop\yarn\lib\jetty-6.1.26.jar;D:\hadoop\share\hadoop\yarn\lib\jetty-util-6.1.26.jar;D:\hadoop\share\hadoop\yarn\lib\jsr305-3.0.0.jar;D:\hadoop\share\hadoop\yarn\lib\leveldbjni-all-1.8.jar;D:\hadoop\share\hadoop\yarn\lib\log4j-1.2.17.jar;D:\hadoop\share\hadoop\yarn\lib\netty-3.6.2.Final.jar;D:\hadoop\share\hadoop\yarn\lib\protobuf-java-2.5.0.jar;D:\hadoop\share\hadoop\yarn\lib\servlet-api-2.5.jar;D:\hadoop\share\hadoop\yarn\lib\stax-api-1.0-2.jar;D:\hadoop\share\hadoop\yarn\lib\xz-1.0.jar;D:\hadoop\share\hadoop\yarn\lib\zookeeper-3.4.6-tests.jar;D:\hadoop\share\hadoop\yarn\lib\zookeeper-3.4.6.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-api-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-distributedshell-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-applications-unmanaged-am-launcher-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-client-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-common-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-registry-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-applicationhistoryservice-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-common-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-nodemanager-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-resourcemanager-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-sharedcachemanager-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-tests-2.7.5.jar;D:\hadoop\share\hadoop\yarn\hadoop-yarn-server-web-proxy-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\lib\aopalliance-1.0.jar;D:\hadoop\share\hadoop\mapreduce\lib\asm-3.2.jar;D:\hadoop\share\hadoop\mapreduce\lib\avro-1.7.4.jar;D:\hadoop\share\hadoop\mapreduce\lib\commons-compress-1.4.1.jar;D:\hadoop\share\hadoop\mapreduce\lib\commons-io-2.4.jar;D:\hadoop\share\hadoop\mapreduce\lib\guice-3.0.jar;D:\hadoop\share\hadoop\mapreduce\lib\guice-servlet-3.0.jar;D:\hadoop\share\hadoop\mapreduce\lib\hadoop-annotations-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\lib\hamcrest-core-1.3.jar;D:\hadoop\share\hadoop\mapreduce\lib\jackson-core-asl-1.9.13.jar;D:\hadoop\share\hadoop\mapreduce\lib\jackson-mapper-asl-1.9.13.jar;D:\hadoop\share\hadoop\mapreduce\lib\javax.inject-1.jar;D:\hadoop\share\hadoop\mapreduce\lib\jersey-core-1.9.jar;D:\hadoop\share\hadoop\mapreduce\lib\jersey-guice-1.9.jar;D:\hadoop\share\hadoop\mapreduce\lib\jersey-server-1.9.jar;D:\hadoop\share\hadoop\mapreduce\lib\junit-4.11.jar;D:\hadoop\share\hadoop\mapreduce\lib\leveldbjni-all-1.8.jar;D:\hadoop\share\hadoop\mapreduce\lib\log4j-1.2.17.jar;D:\hadoop\share\hadoop\mapreduce\lib\netty-3.6.2.Final.jar;D:\hadoop\share\hadoop\mapreduce\lib\paranamer-2.3.jar;D:\hadoop\share\hadoop\mapreduce\lib\protobuf-java-2.5.0.jar;D:\hadoop\share\hadoop\mapreduce\lib\snappy-java-1.0.4.1.jar;D:\hadoop\share\hadoop\mapreduce\lib\xz-1.0.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-app-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-common-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-core-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-hs-plugins-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.7.5-tests.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-jobclient-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-client-shuffle-2.7.5.jar;D:\hadoop\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.7.5.jarSTARTUP_MSG:   build = https://shv@git-wip-us.apache.org/repos/asf/hadoop.git -r 18065c2b6806ed4aa6a3187d77cbe21bb3dba075; compiled by 'kshvachk' on 2017-12-16T01:06ZSTARTUP_MSG:   java = 1.8.0_151************************************************************/18/06/07 06:25:02 INFO namenode.NameNode: createNameNode [-format]18/06/07 06:25:03 WARN common.Util: Path /usr/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.18/06/07 06:25:03 WARN common.Util: Path /usr/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.Formatting using clusterid: CID-923c0653-5a78-46ca-a788-6502dc43047d18/06/07 06:25:04 INFO namenode.FSNamesystem: No KeyProvider found.18/06/07 06:25:04 INFO namenode.FSNamesystem: fsLock is fair: true18/06/07 06:25:04 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false18/06/07 06:25:04 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=100018/06/07 06:25:04 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true18/06/07 06:25:04 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.00018/06/07 06:25:04 INFO blockmanagement.BlockManager: The block deletion will start around 2018 六月 07 06:25:0418/06/07 06:25:04 INFO util.GSet: Computing capacity for map BlocksMap18/06/07 06:25:04 INFO util.GSet: VM type       = 64-bit18/06/07 06:25:04 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB18/06/07 06:25:04 INFO util.GSet: capacity      = 2^21 = 2097152 entries18/06/07 06:25:04 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false18/06/07 06:25:04 INFO blockmanagement.BlockManager: defaultReplication         = 318/06/07 06:25:04 INFO blockmanagement.BlockManager: maxReplication             = 51218/06/07 06:25:04 INFO blockmanagement.BlockManager: minReplication             = 118/06/07 06:25:04 INFO blockmanagement.BlockManager: maxReplicationStreams      = 218/06/07 06:25:04 INFO blockmanagement.BlockManager: replicationRecheckInterval = 300018/06/07 06:25:04 INFO blockmanagement.BlockManager: encryptDataTransfer        = false18/06/07 06:25:04 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 100018/06/07 06:25:04 INFO namenode.FSNamesystem: fsOwner             = Memento (auth:SIMPLE)18/06/07 06:25:04 INFO namenode.FSNamesystem: supergroup          = supergroup18/06/07 06:25:04 INFO namenode.FSNamesystem: isPermissionEnabled = true18/06/07 06:25:04 INFO namenode.FSNamesystem: HA Enabled: false18/06/07 06:25:04 INFO namenode.FSNamesystem: Append Enabled: true18/06/07 06:25:04 INFO util.GSet: Computing capacity for map INodeMap18/06/07 06:25:04 INFO util.GSet: VM type       = 64-bit18/06/07 06:25:04 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB18/06/07 06:25:04 INFO util.GSet: capacity      = 2^20 = 1048576 entries18/06/07 06:25:04 INFO namenode.FSDirectory: ACLs enabled? false18/06/07 06:25:04 INFO namenode.FSDirectory: XAttrs enabled? true18/06/07 06:25:04 INFO namenode.FSDirectory: Maximum size of an xattr: 1638418/06/07 06:25:04 INFO namenode.NameNode: Caching file names occuring more than 10 times18/06/07 06:25:04 INFO util.GSet: Computing capacity for map cachedBlocks18/06/07 06:25:04 INFO util.GSet: VM type       = 64-bit18/06/07 06:25:04 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB18/06/07 06:25:04 INFO util.GSet: capacity      = 2^18 = 262144 entries18/06/07 06:25:04 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.999000012874603318/06/07 06:25:04 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 018/06/07 06:25:04 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 3000018/06/07 06:25:04 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 1018/06/07 06:25:04 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 1018/06/07 06:25:04 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,2518/06/07 06:25:04 INFO namenode.FSNamesystem: Retry cache on namenode is enabled18/06/07 06:25:04 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis18/06/07 06:25:04 INFO util.GSet: Computing capacity for map NameNodeRetryCache18/06/07 06:25:04 INFO util.GSet: VM type       = 64-bit18/06/07 06:25:04 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB18/06/07 06:25:04 INFO util.GSet: capacity      = 2^15 = 32768 entries18/06/07 06:25:04 INFO namenode.FSImage: Allocated new BlockPoolId: BP-869377568-192.168.1.104-152832390486218/06/07 06:25:04 INFO common.Storage: Storage directory C:\usr\hadoop\hdfs\name has been successfully formatted.18/06/07 06:25:04 INFO namenode.FSImageFormatProtobuf: Saving image file C:\usr\hadoop\hdfs\name\current\fsimage.ckpt_0000000000000000000 using no compression18/06/07 06:25:05 INFO namenode.FSImageFormatProtobuf: Image file C:\usr\hadoop\hdfs\name\current\fsimage.ckpt_0000000000000000000 of size 324 bytes saved in 0 seconds.18/06/07 06:25:05 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 018/06/07 06:25:05 INFO util.ExitUtil: Exiting with status 018/06/07 06:25:05 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NameNode at Memento-PC/192.168.1.104************************************************************/

 

五、启动 Hadoop

C:\Users\Memento>start-all.cmdThis script is Deprecated. Instead use start-dfs.cmd and start-yarn.cmdstarting yarn daemons

如果出现如下异常，提示说无法解析 master 地址：

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: Failed on local exception: java.net.SocketException: Unresolved address; Host Details : local host is: "master"; destination host is: (unknown):0

此时需要在 C:\Windows\System32\drivers\etc\hosts 文件中追加 master 的映射：192.168.1.104    master

然后再重新执行启动命令 start-all.cmd；

随后会出现四个命令窗口，依次如下：

1、Apache Hadoop Distribution - hadoop namenode

2、Apache Hadoop Distribution - yarn resourcemanager

3、Apache Hadoop Distribution - yarn nodemanager

4、Apache Hadoop Distribution - hadoop datanode

六、JPS 查看启动进程

C:\Users\XXXXX>jps13460 Jps14676 NodeManager12444 NameNode14204 DataNode14348 ResourceManager

 

七、MapReduce 任务和 hdfs 文件

通过浏览器浏览 localhost:8080 和 localhost:50070 访问浏览：

至此，Hadoop 在 Windows 下的环境搭建完成！

关闭 hadoop

C:\Users\XXXXX>stop-all.cmdThis script is Deprecated. Instead use stop-dfs.cmd and stop-yarn.cmd成功: 给进程发送了终止信号，进程的 PID 为 27204。成功: 给进程发送了终止信号，进程的 PID 为 7884。stopping yarn daemons成功: 给进程发送了终止信号，进程的 PID 为 20464。成功: 给进程发送了终止信号，进程的 PID 为 12516。信息: 没有运行的带有指定标准的任务。

 

相关参考：

winutils：

不想下火车的人：

bin 附件下载：

 

By. Memento

上一篇：[Spark] Spark 安装配置
下一篇：[IDEA] IntelliJ IDEA 安装教程

发表评论

最新留言

路过，博主的博客真漂亮。。
[***.116.15.85]2025年04月17日 07时58分42秒

关于作者

    喝酒易醉，品茶养心，人生如梦，品茶悟道，何以解忧？唯有杜康！
-- 愿君每日到此一游！

推荐文章

大视角、大方向、大问题、大架构：（二）应用的相关问题 2019-03-06
git 2019-03-06
文件上传C:\fakepath\解决方案 2019-03-06
JAVA开发工作流程 2019-03-06
按照list中实体类的某一属性排序 2019-03-06
java计算某日期之后的日期 2019-03-06
struts2自定义类型转换器 2019-03-06
Java调用WebService之Axis实现 2019-03-06
SpringBoot Web（SpringMVC） 2019-03-06
安装rabbitMQ 2019-03-06
javascript 之对象-13 2019-03-06
解决：angularjs radio默认选中失效问题 2019-03-06
java按照关键字指定的key删除redis(支持模糊删除) 2019-03-06
Appium+java ---- Intellij IDEA +genymotion安装配置 2019-03-06
tl-wr742n 怎么设置dns 2019-03-06
Vue基础入门学习 2019-03-06
Spring Validation 校验 2019-03-06
如何使用Postman生成不同格式测试的报告 2019-03-06
Jmeter-ForEach控制器 2019-03-06
Jmeter发送jdbc请求（操作mysql） 2019-03-06
白红宇的个人博客 - 记录点点滴滴的事 - 您是第 459930500 位访客
访问时间: 2025-04-21 14:13:06 访问IP: 3.145.87.56     Copyright © 2020 - 2025 css8.cn 京ICP备2021015314号-1 手机版