Spark_3.2.3编译

# 编译前提说明：R 环境必须提前准备

Spark 构建脚本中启用了 --r 参数来支持 sparkr 模块，因此 R 环境必须提前配置，否则编译将会中断。

注意

缺少 R 会导致 make-distribution.sh 脚本失败，无法完成构建过程。

# 1. 构建环境准备

整个构建流程基于 Bigtop 提供的打包能力，依赖如下基础工具环境：

环境组件	推荐版本	安装参考
JDK	1.8	ONEKEY——安装JDK1.8
Maven	3.8.4	ONEKEY——安装Maven3.8.4
Gradle	内嵌于 Bigtop	ONEKEY——安装Gradle5.6.4

# 2. 执行打包构建命令

我们使用 Bigtop 中的构建任务 spark-rpm 来一键构建 RPM：

gradle spark-rpm \
  -PparentDir=/usr/bigtop \
  -Dbuildwithdeps=true \
  -PpkgSuffix -d

1
2
3
4

构建过程中，会自动下载 Spark 源码，并进行编译：

下载源码

接着进入 RPM 构建流程，日志中会显示关键命令：

01:29:19.936 [ERROR] [system.err] + STATUS=0
01:29:19.936 [ERROR] [system.err] + '[' 0 -ne 0 ']'
01:29:19.936 [ERROR] [system.err] + cd spark-3.2.3
01:29:19.936 [ERROR] [system.err] + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
01:29:20.025 [ERROR] [system.err] + exit 0
01:29:20.026 [QUIET] [system.out] Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.7uHZhq
01:29:20.027 [ERROR] [system.err] + umask 022
01:29:20.027 [ERROR] [system.err] + cd /opt/modules/bigtop/build/spark/rpm//BUILD
01:29:20.027 [ERROR] [system.err] + cd spark-3.2.3
01:29:20.027 [ERROR] [system.err] + bash /opt/modules/bigtop/build/spark/rpm//SOURCES/do-component-build
01:29:20.028 [ERROR] [system.err] ++ dirname /opt/modules/bigtop/build/spark/rpm//SOURCES/do-component-build
01:29:20.029 [ERROR] [system.err] + . /opt/modules/bigtop/build/spark/rpm//SOURCES/bigtop.bom
01:29:20.029 [ERROR] [system.err] ++ ZOOKEEPER_VERSION=3.5.9
01:29:20.029 [ERROR] [system.err] ++ HADOOP_VERSION=3.3.4
01:29:20.029 [ERROR] [system.err] ++ HBASE_VERSION=2.4.13
01:29:20.029 [ERROR] [system.err] ++ HIVE_VERSION=3.1.3
01:29:20.029 [ERROR] [system.err] ++ TEZ_VERSION=0.10.1
01:29:20.029 [ERROR] [system.err] ++ OOZIE_VERSION=5.2.1
01:29:20.029 [ERROR] [system.err] ++ SOLR_VERSION=8.11.2
01:29:20.029 [ERROR] [system.err] ++ SPARK_VERSION=3.2.3
01:29:20.029 [ERROR] [system.err] ++ FLINK_VERSION=1.17.2
01:29:20.029 [ERROR] [system.err] ++ PHOENIX_VERSION=5.1.2
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_GROOVY_VERSION=2.5.4
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_UTILS_VERSION=3.2.0-SNAPSHOT
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_SELECT_VERSION=3.2.0-SNAPSHOT
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_JSVC_VERSION=1.2.4
01:29:20.029 [ERROR] [system.err] ++ ALLUXIO_VERSION=2.8.0
01:29:20.029 [ERROR] [system.err] ++ KAFKA_VERSION=2.8.1
01:29:20.029 [ERROR] [system.err] ++ YCSB_VERSION=0.17.0
01:29:20.029 [ERROR] [system.err] ++ ZEPPELIN_VERSION=0.10.1
01:29:20.029 [ERROR] [system.err] ++ GPDB_VERSION=5.28.5
01:29:20.029 [ERROR] [system.err] ++ AMBARI_VERSION=2.7.5
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_AMBARI_MPACK_VERSION=2.7.5
01:29:20.029 [ERROR] [system.err] ++ LIVY_VERSION=0.7.1
01:29:20.029 [ERROR] [system.err] ++ RANGER_VERSION=2.4.0
01:29:20.029 [ERROR] [system.err] ++ SQOOP_VERSION=1.4.7
01:29:20.029 [ERROR] [system.err] ++ REDIS_VERSION=7.4.0
01:29:20.029 [ERROR] [system.err] ++ DOLPHINSCHEDULER_VERSION=3.2.2
01:29:20.029 [ERROR] [system.err] ++ DORIS_VERSION=2.1.7
01:29:20.029 [ERROR] [system.err] ++ NIGHTINGALE_VERSION=7.7.2
01:29:20.029 [ERROR] [system.err] ++ CATEGRAF_VERSION=0.4.1
01:29:20.029 [ERROR] [system.err] ++ VICTORIAMETRICS_VERSION=1.109.1
01:29:20.029 [ERROR] [system.err] ++ CLOUDBEAVER_VERSION=24.3.3
01:29:20.029 [ERROR] [system.err] ++ CELEBORN_VERSION=0.5.3
01:29:20.029 [ERROR] [system.err] ++ OZONE_VERSION=1.4.1
01:29:20.029 [ERROR] [system.err] ++ IMPALA_VERSION=4.4.1
01:29:20.029 [ERROR] [system.err] ++ TRINO_VERSION=474
01:29:20.029 [ERROR] [system.err] ++ HUDI_VERSION=1.0.1
01:29:20.029 [ERROR] [system.err] ++ PAIMON_VERSION=1.1.0
01:29:20.029 [ERROR] [system.err] ++ JDK_VERSION=1.8
01:29:20.029 [ERROR] [system.err] ++ SCALA_VERSION=2.12.13
01:29:20.029 [ERROR] [system.err] + BUILD_OPTS='-Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root             -Drepo.mav
en.org=             -Dreactor.repo=file:///root/.m2/repository             -Dhadoop.version=3.3.4             -Dyarn.version=3.3.4
     -Pyarn -Phadoop-3.2             -Phive -Phive-thriftserver             -Psparkr -Pkubernetes             -Pscala-2.12             -Dg
uava.version=27.0-jre             '
01:29:20.030 [ERROR] [system.err] + export 'MAVEN_OPTS=-Xss64m -Xmx4g -XX:ReservedCodeCacheSize=1g'
01:29:20.030 [ERROR] [system.err] + MAVEN_OPTS='-Xss64m -Xmx4g -XX:ReservedCodeCacheSize=1g'
01:29:20.030 [ERROR] [system.err] + ./dev/make-distribution.sh --mvn mvn --r -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home
=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.4 -Dyarn.version=3.3.4 -Pyarn -Phadoop-3.2 -Phive
 -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
01:29:20.031 [ERROR] [system.err] +++ dirname ./dev/make-distribution.sh
01:29:20.031 [ERROR] [system.err] ++ cd ./dev/..
01:29:20.032 [ERROR] [system.err] ++ pwd
01:29:20.032 [ERROR] [system.err] + SPARK_HOME=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3
01:29:20.032 [ERROR] [system.err] + DISTDIR=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3/dist
01:29:20.032 [ERROR] [system.err] + MAKE_TGZ=false
01:29:20.032 [ERROR] [system.err] + MAKE_PIP=false
01:29:20.032 [ERROR] [system.err] + MAKE_R=false
01:29:20.032 [ERROR] [system.err] + NAME=none
01:29:20.032 [ERROR] [system.err] + MVN=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3/build/mvn
01:29:20.032 [ERROR] [system.err] + ((  19  ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + MVN=mvn
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + ((  17  ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + MAKE_R=true
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + ((  16  ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + break
01:29:20.032 [ERROR] [system.err] + '[' -z /opt/modules/jdk1.8.0_202 ']'
01:29:20.032 [ERROR] [system.err] + '[' -z /opt/modules/jdk1.8.0_202 ']'
01:29:20.032 [ERROR] [system.err] ++ command -v git
01:29:20.032 [ERROR] [system.err] + '[' /usr/bin/git ']'
01:29:20.033 [ERROR] [system.err] ++ git rev-parse --short HEAD
01:29:20.223 [ERROR] [system.err] + GITREV=13021fc
01:29:20.223 [ERROR] [system.err] + '[' '!' -z 13021fc ']'
01:29:20.223 [ERROR] [system.err] + GITREVSTRING=' (git revision 13021fc)'
01:29:20.223 [ERROR] [system.err] + unset GITREV
01:29:20.224 [ERROR] [system.err] ++ command -v mvn
01:29:20.224 [ERROR] [system.err] + '[' '!' /opt/modules/apache-maven-3.8.4/bin/mvn ']'
01:29:20.225 [ERROR] [system.err] ++ mvn help:evaluate -Dexpression=project.version -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Dus
er.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.4 -Dyarn.version=3.3.4 -Pyarn -Phadoop-3.2
 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
01:29:20.225 [ERROR] [system.err] ++ grep -v INFO
01:29:20.225 [ERROR] [system.err] ++ grep -v WARNING
01:29:20.225 [ERROR] [system.err] ++ tail -n 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

我们从日志中提取出最终的实际 Maven 编译命令如下：

mvn clean package -DskipTests -Dmaven.javadoc.skip=true \
  -Dmaven.scaladoc.skip=true \
  -Dmaven.source.skip \
  -Dcyclonedx.skip=true \
  -Divy.home=/root/.ivy2 \
  -Dsbt.ivy.home=/root/.ivy2 \
  -Duser.home=/root \
  -Drepo.maven.org= \
  -Dreactor.repo=file:///root/.m2/repository \
  -Dhadoop.version=3.3.4 \
  -Dyarn.version=3.3.4 \
  -Pyarn \
  -Phadoop-3.2 \
  -Phive \
  -Phive-thriftserver \
  -Psparkr \
  -Pkubernetes \
  -Pscala-2.12 \
  -Dguava.version=27.0-jre \
  -DskipTests

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

建议你熟悉这些参数的含义，便于后续构建调试或裁剪模块。构建核心参数

# 3. 编译结果

打包完成后，你将看到如下结果：

编译结果

spark_3_2_0-3.2.3-1.el7.noarch.rpm
spark_3_2_0-core-3.2.3-1.el7.noarch.rpm
spark_3_2_0-datanucleus-3.2.3-1.el7.noarch.rpm
spark_3_2_0-external-3.2.3-1.el7.noarch.rpm
spark_3_2_0-history-server-3.2.3-1.el7.noarch.rpm
spark_3_2_0-master-3.2.3-1.el7.noarch.rpm
spark_3_2_0-python-3.2.3-1.el7.noarch.rpm
spark_3_2_0-sparkr-3.2.3-1.el7.noarch.rpm
spark_3_2_0-thriftserver-3.2.3-1.el7.noarch.rpm
spark_3_2_0-worker-3.2.3-1.el7.noarch.rpm
spark_3_2_0-yarn-shuffle-3.2.3-1.el7.noarch.rpm

1
2
3
4
5
6
7
8
9
10
11

# 4.涉及修改点

无

Bigtop提供的代码可以直接编译，只需要解决外部环境即可成功编译（如R环境）。故该版本后续无其他需要修改处。

#Spark #Bigtop #源码编译 #RPM

← [B]Flink版本适配改造（三） Spark_3.5.5编译→