[B]Hive版本适配改造(一)1.0.0+
# 场景背景
在大数据平台集成过程中,Hive 版本适配始终是高频刚需,尤其是当 Spark 升级到 3.5.5 后,涉及组件依赖、日志适配和前端环境都要做细致处理。基于 Bigtop 框架编译 RPM 包时,除了常规源码 patch,还需关注本地 npm 依赖和国内离线包下载。
注意
随着 Spark 版本的演进,Hive 相关依赖包和日志库经常面临兼容性断层,官方文档难以覆盖国内离线或半离线环境,需通过补丁和本地环境脚本进行特殊适配。
# 关键 Patch 补丁说明
本次适配的核心内容,是为 Hive 增加对 Spark 3.5.5 的支持,补丁文件如下:
bigtop-packages/src/common/hive/patch13-HIVE-SPARK355-SUPPORT.diff
Subject: [PATCH] feature: 支持spark 3.5.5
---
Index: pom.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/pom.xml b/pom.xml
--- a/pom.xml (revision 8a522b69ea79ee354ce26ed0c3e9fc061bf5ffc6)
+++ b/pom.xml (date 1749293089939)
@@ -365,6 +365,21 @@
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-core</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-web</artifactId>
+ <version>${log4j2.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
<version>${log4j2.version}</version>
</dependency>
Index: spark-client/pom.xml
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/spark-client/pom.xml b/spark-client/pom.xml
--- a/spark-client/pom.xml (revision 8a522b69ea79ee354ce26ed0c3e9fc061bf5ffc6)
+++ b/spark-client/pom.xml (date 1749290739548)
@@ -86,6 +86,10 @@
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-slf4j2-impl</artifactId>
+ </exclusion>
+ <exclusion>
<groupId>commmons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
其中,pom.xml
中增补了 log4j-api
、log4j-core
、log4j-web
等依赖,并为 spark-client 模块排除了 log4j-slf4j2-impl
,以防止运行期日志冲突。