
[Spark] Spark 安装配置
发布日期:2021-05-09 05:15:31
浏览次数:21
分类:博客文章
本文共 3427 字,大约阅读时间需要 11 分钟。
原文地址:
Windows 上的单机版安装
下载地址:
本文以 Spark 2.3.0 为例
>>> 下载下来的文件是 tgz 格式的压缩文件,直接利用压缩软件将其打开,可以看见里面有一个 tar 格式的压缩文件,继续用压缩软件打开,最终如下图所示:
>>> 将其中的 spark-2.3.0-bin-hadoop2.7 文件夹解压,新增 SPARK_HOME 环境变量,设置为解压路径,并将其追加到 PATH 环境变量;
>>> 此时在 cmd 窗口中执行 "spark-shell" 命令可得到如下提示:
C:\Users\Memento>spark-shell2018-06-06 23:39:36 ERROR Shell:397 - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394) at org.apache.hadoop.util.Shell.(Shell.java:387) at org.apache.hadoop.util.StringUtils. (StringUtils.java:80) at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611) at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261) at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464) at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2464) at org.apache.spark.SecurityManager. (SecurityManager.scala:222) at org.apache.spark.deploy.SparkSubmit$.secMgr$lzycompute$1(SparkSubmit.scala:393) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:393) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at scala.Option.map(Option.scala:146) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:400) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)2018-06-06 23:39:36 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicableSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).Spark context Web UI available at http://Memento-PC:4040Spark context available as 'sc' (master = local[*], app id = local-1528299586814).Spark session available as 'spark'.Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.0 /_/Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151)Type in expressions to have them evaluated.Type :help for more information.
提示说在 hadoop 路径下无法定位到 winutils,所以接下来需要配置 Hadoop;
>>> 详见:
>>> 随后再重新执行 "spark-shell" 命令即可;
By. Memento
发表评论
最新留言
初次前来,多多关照!
[***.217.46.12]2025年04月05日 11时10分44秒
关于作者

喝酒易醉,品茶养心,人生如梦,品茶悟道,何以解忧?唯有杜康!
-- 愿君每日到此一游!
推荐文章
wxWidgets源码分析(6) - 窗口关闭过程
2021-05-09
wxWidgets源码分析(7) - 窗口尺寸
2021-05-09
wxWidgets源码分析(8) - MVC架构
2021-05-09
wxWidgets源码分析(9) - wxString
2021-05-09
TCP黏包
2021-05-09
python数据类型(总结篇)
2021-05-09
机器学习之数据探索——数据质量分析
2021-05-09
机器学习之数据探索——数据特征分析(分布分析)
2021-05-09
机器学习之数据探索——数据特征分析(帕累托分析)
2021-05-09
深度解密Go语言之 scheduler
2021-05-09
分库分表(4) ---SpringBoot + ShardingSphere 实现分表
2021-05-09
SpringBoot(20)---断言(Assert)
2021-05-09
eclipse快捷键
2021-05-09
java提高(6)---Serializable
2021-05-09
VueJs(3)---V-指令(1)
2021-05-09
【Docker】(1)---Docker入门篇
2021-05-09
【Docker】(5)---springCloud注册中心打包Docker镜像
2021-05-09
Spring 学习笔记(三)——IoC容器(ApplicationContext)
2021-05-09