Spark_3.2.3编译1.0.0+
# 编译前提说明:R 环境必须提前准备
Spark 构建脚本中启用了 --r
参数来支持 sparkr
模块,因此 R 环境必须提前配置,否则编译将会中断。
注意
缺少 R 会导致 make-distribution.sh
脚本失败,无法完成构建过程。
👉 R 环境一键安装脚本 必须准备
# 1. 构建环境准备
整个构建流程基于 Bigtop 提供的打包能力,依赖如下基础工具环境:
环境组件 | 推荐版本 | 安装参考 |
---|---|---|
JDK | 1.8 | ONEKEY——安装JDK1.8 |
Maven | 3.8.4 | ONEKEY——安装Maven3.8.4 |
Gradle | 内嵌于 Bigtop | ONEKEY——安装Gradle5.6.4 |
# 2. 执行打包构建命令
我们使用 Bigtop 中的构建任务 spark-rpm
来一键构建 RPM:
gradle spark-rpm \
-PparentDir=/usr/bigtop \
-Dbuildwithdeps=true \
-PpkgSuffix -d
1
2
3
4
2
3
4
构建过程中,会自动下载 Spark 源码,并进行编译:
接着进入 RPM 构建流程,日志中会显示关键命令:
01:29:19.936 [ERROR] [system.err] + STATUS=0
01:29:19.936 [ERROR] [system.err] + '[' 0 -ne 0 ']'
01:29:19.936 [ERROR] [system.err] + cd spark-3.2.3
01:29:19.936 [ERROR] [system.err] + /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
01:29:20.025 [ERROR] [system.err] + exit 0
01:29:20.026 [QUIET] [system.out] Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.7uHZhq
01:29:20.027 [ERROR] [system.err] + umask 022
01:29:20.027 [ERROR] [system.err] + cd /opt/modules/bigtop/build/spark/rpm//BUILD
01:29:20.027 [ERROR] [system.err] + cd spark-3.2.3
01:29:20.027 [ERROR] [system.err] + bash /opt/modules/bigtop/build/spark/rpm//SOURCES/do-component-build
01:29:20.028 [ERROR] [system.err] ++ dirname /opt/modules/bigtop/build/spark/rpm//SOURCES/do-component-build
01:29:20.029 [ERROR] [system.err] + . /opt/modules/bigtop/build/spark/rpm//SOURCES/bigtop.bom
01:29:20.029 [ERROR] [system.err] ++ ZOOKEEPER_VERSION=3.5.9
01:29:20.029 [ERROR] [system.err] ++ HADOOP_VERSION=3.3.4
01:29:20.029 [ERROR] [system.err] ++ HBASE_VERSION=2.4.13
01:29:20.029 [ERROR] [system.err] ++ HIVE_VERSION=3.1.3
01:29:20.029 [ERROR] [system.err] ++ TEZ_VERSION=0.10.1
01:29:20.029 [ERROR] [system.err] ++ OOZIE_VERSION=5.2.1
01:29:20.029 [ERROR] [system.err] ++ SOLR_VERSION=8.11.2
01:29:20.029 [ERROR] [system.err] ++ SPARK_VERSION=3.2.3
01:29:20.029 [ERROR] [system.err] ++ FLINK_VERSION=1.17.2
01:29:20.029 [ERROR] [system.err] ++ PHOENIX_VERSION=5.1.2
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_GROOVY_VERSION=2.5.4
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_UTILS_VERSION=3.2.0-SNAPSHOT
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_SELECT_VERSION=3.2.0-SNAPSHOT
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_JSVC_VERSION=1.2.4
01:29:20.029 [ERROR] [system.err] ++ ALLUXIO_VERSION=2.8.0
01:29:20.029 [ERROR] [system.err] ++ KAFKA_VERSION=2.8.1
01:29:20.029 [ERROR] [system.err] ++ YCSB_VERSION=0.17.0
01:29:20.029 [ERROR] [system.err] ++ ZEPPELIN_VERSION=0.10.1
01:29:20.029 [ERROR] [system.err] ++ GPDB_VERSION=5.28.5
01:29:20.029 [ERROR] [system.err] ++ AMBARI_VERSION=2.7.5
01:29:20.029 [ERROR] [system.err] ++ BIGTOP_AMBARI_MPACK_VERSION=2.7.5
01:29:20.029 [ERROR] [system.err] ++ LIVY_VERSION=0.7.1
01:29:20.029 [ERROR] [system.err] ++ RANGER_VERSION=2.4.0
01:29:20.029 [ERROR] [system.err] ++ SQOOP_VERSION=1.4.7
01:29:20.029 [ERROR] [system.err] ++ REDIS_VERSION=7.4.0
01:29:20.029 [ERROR] [system.err] ++ DOLPHINSCHEDULER_VERSION=3.2.2
01:29:20.029 [ERROR] [system.err] ++ DORIS_VERSION=2.1.7
01:29:20.029 [ERROR] [system.err] ++ NIGHTINGALE_VERSION=7.7.2
01:29:20.029 [ERROR] [system.err] ++ CATEGRAF_VERSION=0.4.1
01:29:20.029 [ERROR] [system.err] ++ VICTORIAMETRICS_VERSION=1.109.1
01:29:20.029 [ERROR] [system.err] ++ CLOUDBEAVER_VERSION=24.3.3
01:29:20.029 [ERROR] [system.err] ++ CELEBORN_VERSION=0.5.3
01:29:20.029 [ERROR] [system.err] ++ OZONE_VERSION=1.4.1
01:29:20.029 [ERROR] [system.err] ++ IMPALA_VERSION=4.4.1
01:29:20.029 [ERROR] [system.err] ++ TRINO_VERSION=474
01:29:20.029 [ERROR] [system.err] ++ HUDI_VERSION=1.0.1
01:29:20.029 [ERROR] [system.err] ++ PAIMON_VERSION=1.1.0
01:29:20.029 [ERROR] [system.err] ++ JDK_VERSION=1.8
01:29:20.029 [ERROR] [system.err] ++ SCALA_VERSION=2.12.13
01:29:20.029 [ERROR] [system.err] + BUILD_OPTS='-Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home=/root -Drepo.mav
en.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.4 -Dyarn.version=3.3.4
-Pyarn -Phadoop-3.2 -Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dg
uava.version=27.0-jre '
01:29:20.030 [ERROR] [system.err] + export 'MAVEN_OPTS=-Xss64m -Xmx4g -XX:ReservedCodeCacheSize=1g'
01:29:20.030 [ERROR] [system.err] + MAVEN_OPTS='-Xss64m -Xmx4g -XX:ReservedCodeCacheSize=1g'
01:29:20.030 [ERROR] [system.err] + ./dev/make-distribution.sh --mvn mvn --r -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Duser.home
=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.4 -Dyarn.version=3.3.4 -Pyarn -Phadoop-3.2 -Phive
-Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
01:29:20.031 [ERROR] [system.err] +++ dirname ./dev/make-distribution.sh
01:29:20.031 [ERROR] [system.err] ++ cd ./dev/..
01:29:20.032 [ERROR] [system.err] ++ pwd
01:29:20.032 [ERROR] [system.err] + SPARK_HOME=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3
01:29:20.032 [ERROR] [system.err] + DISTDIR=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3/dist
01:29:20.032 [ERROR] [system.err] + MAKE_TGZ=false
01:29:20.032 [ERROR] [system.err] + MAKE_PIP=false
01:29:20.032 [ERROR] [system.err] + MAKE_R=false
01:29:20.032 [ERROR] [system.err] + NAME=none
01:29:20.032 [ERROR] [system.err] + MVN=/opt/modules/bigtop/build/spark/rpm/BUILD/spark-3.2.3/build/mvn
01:29:20.032 [ERROR] [system.err] + (( 19 ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + MVN=mvn
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + (( 17 ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + MAKE_R=true
01:29:20.032 [ERROR] [system.err] + shift
01:29:20.032 [ERROR] [system.err] + (( 16 ))
01:29:20.032 [ERROR] [system.err] + case $1 in
01:29:20.032 [ERROR] [system.err] + break
01:29:20.032 [ERROR] [system.err] + '[' -z /opt/modules/jdk1.8.0_202 ']'
01:29:20.032 [ERROR] [system.err] + '[' -z /opt/modules/jdk1.8.0_202 ']'
01:29:20.032 [ERROR] [system.err] ++ command -v git
01:29:20.032 [ERROR] [system.err] + '[' /usr/bin/git ']'
01:29:20.033 [ERROR] [system.err] ++ git rev-parse --short HEAD
01:29:20.223 [ERROR] [system.err] + GITREV=13021fc
01:29:20.223 [ERROR] [system.err] + '[' '!' -z 13021fc ']'
01:29:20.223 [ERROR] [system.err] + GITREVSTRING=' (git revision 13021fc)'
01:29:20.223 [ERROR] [system.err] + unset GITREV
01:29:20.224 [ERROR] [system.err] ++ command -v mvn
01:29:20.224 [ERROR] [system.err] + '[' '!' /opt/modules/apache-maven-3.8.4/bin/mvn ']'
01:29:20.225 [ERROR] [system.err] ++ mvn help:evaluate -Dexpression=project.version -Divy.home=/root/.ivy2 -Dsbt.ivy.home=/root/.ivy2 -Dus
er.home=/root -Drepo.maven.org= -Dreactor.repo=file:///root/.m2/repository -Dhadoop.version=3.3.4 -Dyarn.version=3.3.4 -Pyarn -Phadoop-3.2
-Phive -Phive-thriftserver -Psparkr -Pkubernetes -Pscala-2.12 -Dguava.version=27.0-jre -DskipTests
01:29:20.225 [ERROR] [system.err] ++ grep -v INFO
01:29:20.225 [ERROR] [system.err] ++ grep -v WARNING
01:29:20.225 [ERROR] [system.err] ++ tail -n 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
我们从日志中提取出最终的实际 Maven 编译命令如下:
mvn clean package -DskipTests -Dmaven.javadoc.skip=true \
-Dmaven.scaladoc.skip=true \
-Dmaven.source.skip \
-Dcyclonedx.skip=true \
-Divy.home=/root/.ivy2 \
-Dsbt.ivy.home=/root/.ivy2 \
-Duser.home=/root \
-Drepo.maven.org= \
-Dreactor.repo=file:///root/.m2/repository \
-Dhadoop.version=3.3.4 \
-Dyarn.version=3.3.4 \
-Pyarn \
-Phadoop-3.2 \
-Phive \
-Phive-thriftserver \
-Psparkr \
-Pkubernetes \
-Pscala-2.12 \
-Dguava.version=27.0-jre \
-DskipTests
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
建议你熟悉这些参数的含义,便于后续构建调试或裁剪模块。构建核心参数
# 3. 编译结果
打包完成后,你将看到如下结果:
spark_3_2_0-3.2.3-1.el7.noarch.rpm
spark_3_2_0-core-3.2.3-1.el7.noarch.rpm
spark_3_2_0-datanucleus-3.2.3-1.el7.noarch.rpm
spark_3_2_0-external-3.2.3-1.el7.noarch.rpm
spark_3_2_0-history-server-3.2.3-1.el7.noarch.rpm
spark_3_2_0-master-3.2.3-1.el7.noarch.rpm
spark_3_2_0-python-3.2.3-1.el7.noarch.rpm
spark_3_2_0-sparkr-3.2.3-1.el7.noarch.rpm
spark_3_2_0-thriftserver-3.2.3-1.el7.noarch.rpm
spark_3_2_0-worker-3.2.3-1.el7.noarch.rpm
spark_3_2_0-yarn-shuffle-3.2.3-1.el7.noarch.rpm
1
2
3
4
5
6
7
8
9
10
11
2
3
4
5
6
7
8
9
10
11
# 4.涉及修改点
无
Bigtop提供的代码可以直接编译,只需要解决外部环境即可成功编译(如R环境)。故该版本后续无其他需要修改处。