Tez_0.10.1 编译
# 1. 构建环境准备与依赖
Tez 0.10.1 编译同样强依赖 Bigtop 体系,强烈建议直接在官方 Bigtop 容器或预设环境下完成,极大减少环境不一致导致的构建失败。
环境组件 | 推荐版本 | 安装参考 |
---|---|---|
JDK | 1.8 | ONEKEY——安装JDK1.8 |
Maven | 3.8.4 | ONEKEY——安装Maven3.8.4 |
Gradle | 内嵌于 Bigtop | ONEKEY——安装Gradle5.6.4 |
提示
如未配置上述环境,可参考本站 JDK、Maven、Gradle 文档。不仅有详细安装说明,还附带一键安装脚本与离线包下载,覆盖主流运维场景,极大提升稳定性与复现度。
# 2. 源码包准备与离线编译建议
Tez 的源码包建议提前下载,默认存放路径为 /opt/modules/bigtop/dl/apache-tez-0.10.1-src.tar.gz
,首次构建会自动检测,缺失时自动尝试联网下载。
警告
企业云环境/内网环境建议务必提前下载所有源码和依赖包,否则编译过程中网络波动或墙代理问题易致失败。
# 3. 一键构建命令与参数说明 推荐
进入 Bigtop 根目录后,直接执行如下命令:
gradle tez-rpm -PparentDir=/usr/bigtop -Dbuildwithdeps=true -PpkgSuffix -d
1
参数 | 作用说明 |
---|---|
-PparentDir | 指定 Bigtop 根路径,推荐必填 |
-Dbuildwithdeps=true | 强制自动拉取全部依赖组件,保证依赖完整 |
-PpkgSuffix | 自定义包名后缀,选填 |
-d | 输出详细调试日志,有助问题定位 |
注意
务必确认当前 JAVA_HOME
、M2_HOME
等变量已正确指向实际路径,避免依赖找不到。
# 4. 编译流程与关键日志解析
执行 gradle 任务后,Bigtop 会自动串联底层脚本与 Maven 任务,核心日志示例:
> Task :tez-rpm
Installing /opt/modules/bigtop/output/tez/tez_3_2_0-0.10.1-1.el8.src.rpm
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.SMlF9E
+ umask 022
+ cd /opt/modules/bigtop/build/tez/rpm//BUILD
+ cd /opt/modules/bigtop/build/tez/rpm/BUILD
+ rm -rf apache-tez-0.10.1-src
+ /usr/bin/gzip -dc /opt/modules/bigtop/build/tez/rpm/SOURCES/apache-tez-0.10.1-src.tar.gz
+ /usr/bin/tar -xof -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd apache-tez-0.10.1-src
+ /usr/bin/chmod -Rf a+rX,u+w,g-w,o-w .
Patch #0 (patch0-remove-phantomjs.diff):
+ echo 'Patch #0 (patch0-remove-phantomjs.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
patching file tez-ui/src/main/webapp/package.json
Patch #1 (patch1-TEZ-4214.diff):
+ echo 'Patch #1 (patch1-TEZ-4214.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
patching file pom.xml
patching file tez-plugins/tez-aux-services/pom.xml
Patch #2 (patch2-TEZ-4319.diff):
+ echo 'Patch #2 (patch2-TEZ-4319.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
patching file pom.xml
patching file tez-api/pom.xml
+ echo 'Patch #3 (patch3-TEZ-4411.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
Patch #3 (patch3-TEZ-4411.diff):
patching file tez-ui/src/main/resources/META-INF/LICENSE.txt
patching file tez-ui/src/main/webapp/bower-shrinkwrap.json
patching file tez-ui/src/main/webapp/bower.json
Hunk #1 succeeded at 21 (offset -1 lines).
patching file tez-ui/src/main/webapp/ember-cli-build.js
patching file tez-ui/src/main/webapp/bower-shrinkwrap.json
patching file tez-ui/src/main/webapp/bower.json
Hunk #1 succeeded at 22 (offset -1 lines).
Patch #4 (patch4-TEZ-4300-and-TEZ-4428.diff):
+ echo 'Patch #4 (patch4-TEZ-4300-and-TEZ-4428.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
patching file .travis.yml
patching file BUILDING.txt
patching file build-tools/.gitignore
patching file build-tools/install-protobuf.sh
patching file tez-api/pom.xml
patching file tez-dag/pom.xml
patching file tez-ext-service-tests/pom.xml
patching file tez-mapreduce/pom.xml
patching file tez-plugins/tez-protobuf-history-plugin/pom.xml
patching file tez-runtime-internals/pom.xml
patching file tez-runtime-library/pom.xml
+ echo 'Patch #5 (patch5-TEZ-4298.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
Patch #5 (patch5-TEZ-4298.diff):
patching file tez-plugins/tez-aux-services/findbugs-exclude.xml
patching file tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java
Patch #6 (patch6-TEZ-COMPILE-FAST.diff):
+ echo 'Patch #6 (patch6-TEZ-COMPILE-FAST.diff):'
+ /usr/bin/patch --no-backup-if-mismatch -p1 --fuzz=0
patching file tez-ui/pom.xml
patching file tez-ui/src/main/webapp/.bowerrc
patching file tez-ui/src/main/webapp/bower.json
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.0uGeOJ
+ umask 022
+ cd /opt/modules/bigtop/build/tez/rpm//BUILD
+ cd apache-tez-0.10.1-src
+ env TEZ_VERSION=0.10.1 bash /opt/modules/bigtop/build/tez/rpm/SOURCES/do-component-build
++ dirname /opt/modules/bigtop/build/tez/rpm/SOURCES/do-component-build
+ . /opt/modules/bigtop/build/tez/rpm/SOURCES/bigtop.bom
++ ZOOKEEPER_VERSION=3.5.9
++ HADOOP_VERSION=3.3.4
++ HBASE_VERSION=2.4.13
++ HIVE_VERSION=3.1.3
++ TEZ_VERSION=0.10.1
++ OOZIE_VERSION=5.2.1
++ SOLR_VERSION=8.11.2
++ SPARK_VERSION=3.5.5
++ FLINK_VERSION=1.17.2
++ PHOENIX_VERSION=5.1.2
++ BIGTOP_GROOVY_VERSION=2.5.4
++ BIGTOP_UTILS_VERSION=3.2.0-SNAPSHOT
++ BIGTOP_SELECT_VERSION=3.2.0-SNAPSHOT
++ BIGTOP_JSVC_VERSION=1.2.4
++ ALLUXIO_VERSION=2.8.0
++ KAFKA_VERSION=2.8.1
++ YCSB_VERSION=0.17.0
++ ZEPPELIN_VERSION=0.10.1
++ GPDB_VERSION=5.28.5
++ AMBARI_VERSION=2.7.5
++ BIGTOP_AMBARI_MPACK_VERSION=2.7.5
++ LIVY_VERSION=0.7.1
++ RANGER_VERSION=2.4.0
++ SQOOP_VERSION=1.4.7
++ REDIS_VERSION=7.4.0
++ DOLPHINSCHEDULER_VERSION=3.2.2
++ DORIS_VERSION=2.1.7
++ NIGHTINGALE_VERSION=7.7.2
++ CATEGRAF_VERSION=0.4.1
++ VICTORIAMETRICS_VERSION=1.109.1
++ CLOUDBEAVER_VERSION=24.3.3
++ CELEBORN_VERSION=0.5.3
++ OZONE_VERSION=1.4.1
++ IMPALA_VERSION=4.4.1
++ TRINO_VERSION=474
++ HUDI_VERSION=1.0.1
++ PAIMON_VERSION=1.1.0
++ ATLAS_VERSION=2.4.0
++ SUPERSET_VERSION=4.1.2
++ JDK_VERSION=1.8
++ SCALA_VERSION=2.12.13
+ '[' x86_64 = powerpc64le ']'
+ '[' x86_64 = aarch64 ']'
+ BUILD_TEZ_OPTS='clean package -Dtar -Dhadoop.version=3.3.4 -Phadoop28 -DskipTests'
+ mvn -X clean package -Dtar -Dhadoop.version=3.3.4 -Phadoop28 -DskipTests
Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937)
Maven home: /opt/modules/apache-maven-3.9.9
Java version: 1.8.0_202, vendor: Oracle Corporation, runtime: /opt/modules/jdk1.8.0_202/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "6.6.87.1-microsoft-stand
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
成功时,产物将归档到 output 目录,详见下节。编译日志片段
# 5. 编译产物与输出目录
最终产物在如下路径:
/opt/modules/bigtop/output/tez/noarch
1
例如:
drwxr-xr-x 2 root root 4096 Jun 8 06:10 .
drwxr-xr-x 3 root root 4096 Jun 8 06:10 ..
-rw-r--r-- 1 root root 90301388 Jun 8 06:10 tez_3_2_0-0.10.1-1.el8.noarch.rpm
[root@rocky8 noarch]# pwd
/opt/modules/bigtop/output/tez/noarch
[root@rocky8 noarch]#
1
2
3
4
5
6
7
2
3
4
5
6
7
# 6. 常见问题与后续集成
Tez 包二次开发与适配过程常需涉及如下修改(参考实际业务需求可按需调整):
按顺序修改即可
- 01
- bigtop-select 打包缺 compat 报错修复 deb07-16
- 02
- bigtop-select 打包缺 control 文件报错修复 deb07-16
- 03
- 首次编译-环境初始化 必装07-16