TT Bigdata
JaneTTR
2025-05-06
目录

[B]Kafka版本适配改造（一）1.0.0+

# 1. 场景背景与适配目标

实际大数据平台自建私服（如 Nexus/Artifactory）已是主流，用于缓存和加速 Maven/Gradle/Node 各类依赖资源。但 Bigtop 源码编译流程默认拉取外网源，常遇如下痛点：

  • 依赖包下载慢甚至失败
  • 版本不可控，易受网络波动影响
  • 线上/离线环境难以复现

提示

本次针对 kafka/do-component-build 和 gradle 相关脚本做补丁，核心目标就是让所有依赖优先从本地私服拉取，彻底屏蔽外部环境变量的影响，提升编译稳定性。

# 2. 补丁一：do-component-build 传递 NEXUS_URL

为让构建脚本能够感知私服地址，需在 BUILD_OPTS 中显式注入 -DNEXUS_URL 参数，patch 如下：

Subject: [PATCH] feature: 使用私服来下载
---
Index: bigtop-packages/src/common/kafka/do-component-build
@@ -21,6 +21,7 @@
 BUILD_OPTS="-Divy.home=${HOME}/.ivy2 -Dsbt.ivy.home=${HOME}/.ivy2 -Duser.home=${HOME} \
             -Drepo.maven.org=$IVY_MIRROR_PROP \
             -Dreactor.repo=file://${HOME}/.m2/repository \
+            -DNEXUS_URL=${NEXUS_URL} \
             -DskipTests -DrecompileMode=all"
1
2
3
4
5
6
7
8
9

后续无论是 Gradle 还是 Node/yarn 依赖，都可通过 ${NEXUS_URL} 变量实现私服切换，兼容所有主流仓库。

笔记

建议配合 CI 环境变量预先配置好 NEXUS_URL，避免脚本硬编码。

# 3. 补丁二：加速构建流程与归档逻辑

进一步对构建命令和产物归档部分优化，屏蔽签名环节并修正包输出逻辑：

@@ -36,9 +37,10 @@
 # Rewrite the zookeeper version defined in gradle/dependencies.gradle directly
 # with $ZOOKEEPER_VERSION, because we can't override it via command line option.
 sed -i -e 's/zookeeper: *"\([0-9]\{1,\}\.\)*[0-9]\{1,\}"/zookeeper: "'${ZOOKEEPER_VERSION}'"/' gradle/dependencies.gradle
-./gradlew -PscalaVersion=${SCALA_VERSION} clean releaseTarGz
+./gradlew -PscalaVersion=${SCALA_VERSION} clean releaseTarGz -x signArchives -d
 
 rm -rf build
 mkdir build
+  
+cp core/build/distributions/kafka_*.tgz build/
1
2
3
4
5
6
7
8
9
10
11
实测加速 跳过签名、提前归档，能缩短本地构建用时，也规避 CI 无 key 环境下的潜在报错。

kafka patch 效果图

# 4. 补丁三：Gradle/Node 源切换全覆盖

为彻底解决 gradle、node、wrapper 等一切第三方下载均能走私服，还需叠加如下 patch：

diff --git a/build.gradle b/build.gradle
@@ -19,7 +19,13 @@
 buildscript {
   repositories {
-    mavenCentral()
+    maven {
+      url "http://${System.getenv('NEXUS_URL')}:8081/repository/maven-public/"
+      credentials {
+        username = 'admin'
+        password = 'admin123'
+      }
+    }
    maven { url "https://plugins.gradle.org/m2/" }
  }
}
@@ -56,7 +62,13 @@
allprojects {
  repositories {
-    mavenCentral()
+    maven {
+      url "http://${System.getenv('NEXUS_URL')}:8081/repository/maven-public/"
+      credentials {
+        username = 'admin'
+        password = 'admin123'
+      }
+    }
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

以及对 gradle-wrapper/wget/curl 下载源的统一加速替换：

  • gradle-wrapper.properties、gradlew、wrapper.gradle、vagrant/aws/aws-init.sh，全部指向阿里云或内网镜像。
  • 解决外网流控、TLS 降级和墙的问题，提升稳定性。

注意

如团队私服需要认证，注意 patch 里显式配置账号密码。涉及公共仓库时可适当降权用户权限，避免信息泄露风险。

# 5. 完整 Patch 附录

以下为本次 Kafka 私服适配的所有实际 patch 内容，供后续直接应用和对比。

提示

建议保存到本地，结合实际 bigtop 代码目录和分支环境，逐步应用。

Subject: [PATCH] feature: 使用私服来下载
---
Index: bigtop-packages/src/common/kafka/do-component-build
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/bigtop-packages/src/common/kafka/do-component-build b/bigtop-packages/src/common/kafka/do-component-build
--- a/bigtop-packages/src/common/kafka/do-component-build	(date 1749371358018)
+++ b/bigtop-packages/src/common/kafka/do-component-build	(revision e9671be7d052e2808b5b8a5fa5d932af95693212)
@@ -21,6 +21,7 @@
 BUILD_OPTS="-Divy.home=${HOME}/.ivy2 -Dsbt.ivy.home=${HOME}/.ivy2 -Duser.home=${HOME} \
             -Drepo.maven.org=$IVY_MIRROR_PROP \
             -Dreactor.repo=file://${HOME}/.m2/repository \
+            -DNEXUS_URL=${NEXUS_URL} \
             -DskipTests -DrecompileMode=all"
 
 # if BIGTOP_PREFERRED_JAVA_VERSION is set, switch java version for the build
@@ -36,9 +37,10 @@
 # Rewrite the zookeeper version defined in gradle/dependencies.gradle directly
 # with $ZOOKEEPER_VERSION, because we can't override it via command line option.
 sed -i -e 's/zookeeper: *"\([0-9]\{1,\}\.\)*[0-9]\{1,\}"/zookeeper: "'${ZOOKEEPER_VERSION}'"/' gradle/dependencies.gradle
-./gradlew -PscalaVersion=${SCALA_VERSION} clean releaseTarGz
+./gradlew -PscalaVersion=${SCALA_VERSION} clean releaseTarGz -x signArchives -d
 
 rm -rf build
 mkdir build
+  
+cp core/build/distributions/kafka_*.tgz build/
 
-cp core/build/distributions/kafka_*.tgz build/
\ No newline at end of file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
diff --git a/build.gradle b/build.gradle
index d36170c..e53b2d0 100644
--- a/build.gradle
+++ b/build.gradle
@@ -19,7 +19,13 @@ import java.nio.charset.StandardCharsets

 buildscript {
   repositories {
-    mavenCentral()
+    maven {
+      url "http://${System.getenv('NEXUS_URL')}:8081/repository/maven-public/"
+      credentials {
+        username = 'admin'
+        password = 'admin123'
+      }
+    }
     maven {
       url "https://plugins.gradle.org/m2/"
     }
@@ -56,7 +62,13 @@ spotless {
 allprojects {

   repositories {
-    mavenCentral()
+    maven {
+      url "http://${System.getenv('NEXUS_URL')}:8081/repository/maven-public/"
+      credentials {
+        username = 'admin'
+        password = 'admin123'
+      }
+    }
   }

   apply plugin: 'idea'
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
index 1c4bcc2..4b49dc5 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.1-all.zip
+distributionUrl=https\://mirrors.aliyun.com/macports/distfiles/gradle/gradle-6.8.1-all.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
diff --git a/gradlew b/gradlew
index fcb97d2..bf66147 100755
--- a/gradlew
+++ b/gradlew
@@ -81,7 +81,7 @@ esac
 # Loop in case we encounter an error.
 for attempt in 1 2 3; do
   if [ ! -e "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" ]; then
-    if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://raw.githubusercontent.com/gradle/gradle/v6.8.1/gradle/wrapper/gradle-wrapper.jar"; then
+    if ! curl -s -S --retry 3 -L -o "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" "https://mirror.ghproxy.com/https://raw.githubusercontent.com/gradle/gradle/v6.8.1/gradle/wrapper/gradle-wrapper.jar"; then
       rm -f "$APP_HOME/gradle/wrapper/gradle-wrapper.jar"
       # Pause for a bit before looping in case the server throttled us.
       sleep 5
diff --git a/vagrant/aws/aws-init.sh b/vagrant/aws/aws-init.sh
index f36d4e3..21bb641 100755
--- a/vagrant/aws/aws-init.sh
+++ b/vagrant/aws/aws-init.sh
@@ -61,7 +61,7 @@ fi
 gradle="gradle-2.2.1"
 if [ -z `which gradle` ] && [ ! -d $base_dir/$gradle ]; then
     if [ ! -e $gradle-bin.zip ]; then
-        wget https://services.gradle.org/distributions/$gradle-bin.zip
+        wget https://mirrors.aliyun.com/macports/distfiles/gradle/$gradle-bin.zip
     fi
     unzip $gradle-bin.zip
     rm -rf $gradle-bin.zip
diff --git a/wrapper.gradle b/wrapper.gradle
index 2dfca19..6619b12 100644
--- a/wrapper.gradle
+++ b/wrapper.gradle
@@ -43,7 +43,7 @@ task bootstrapWrapper() {
         // github.com servers deprecated TLSv1/TLSv1.1 support some time ago, so older versions
         // of curl (built against OpenSSL library that doesn't support TLSv1.2) would fail to
         // fetch the jar.
-        def wrapperBaseUrl = "https://raw.githubusercontent.com/gradle/gradle/v$fullVersion/gradle/wrapper"
+        def wrapperBaseUrl = "https://mirror.ghproxy.com/https://raw.githubusercontent.com/gradle/gradle/v$fullVersion/gradle/wrapper"
         def wrapperJarUrl = wrapperBaseUrl + "/gradle-wrapper.jar"

         def bootstrapString = """
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
#Kafka#Bigtop#RPM#私服适配#构建优化
[O]Kafka版本适配改造（二）
Solr_8.11.2 编译

← Solr_8.11.2 编译→

最近更新
01
Pandoc 缺失导致 SparkR 构建失败
06-08
02
Cyrus SASL/GSASL 缺失解决
06-07
03
Hadoop_3.3.4 编译 1.0.0+
06-06
更多文章>
Theme by Vdoing | Copyright © 2017-2025 JaneTTR | MIT License