Ambari Metrics 进阶安装技巧

# 一、问题现象与触发条件

部署 Ambari Metrics（AMS）Collector 后，服务启动失败，关键日志集中在 AMSApplicationServer WebApp 启动阶段：

2026-01-27 13:34:46,068 INFO org.apache.hadoop.http.HttpServer2: HttpServer.start() threw a non Bind IOException
java.net.BindException: Port in use: dev3:6188
        at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1344)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1366)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1429)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1275)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:472)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:461)
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:135)
        at org.apache.ambari.metrics.AMSApplicationServer.serviceStart(AMSApplicationServer.java:73)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.ambari.metrics.AMSApplicationServer.launchAMSApplicationServer(AMSApplicationServer.java:98)
        at org.apache.ambari.metrics.AMSApplicationServer.main(AMSApplicationServer.java:107)
Caused by: java.io.IOException: Failed to bind to dev3/192.168.3.157:6188
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1331)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1362)
        ... 9 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 12 more
2026-01-27 13:34:46,070 ERROR org.apache.ambari.metrics.AMSApplicationServer: AHSWebApp failed to start.
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:476)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:461)
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:135)
        at org.apache.ambari.metrics.AMSApplicationServer.serviceStart(AMSApplicationServer.java:73)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.ambari.metrics.AMSApplicationServer.launchAMSApplicationServer(AMSApplicationServer.java:98)
        at org.apache.ambari.metrics.AMSApplicationServer.main(AMSApplicationServer.java:107)
Caused by: java.net.BindException: Port in use: dev3:6188
        at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1344)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1366)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1429)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1275)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:472)
        ... 6 more
Caused by: java.io.IOException: Failed to bind to dev3/192.168.3.157:6188
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1331)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1362)
        ... 9 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 12 more
2026-01-27 13:34:46,071 INFO org.apache.hadoop.service.AbstractService: Service org.apache.ambari.metrics.AMSApplicationServer failed in state STARTED
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: AHSWebApp failed to start.
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:139)
        at org.apache.ambari.metrics.AMSApplicationServer.serviceStart(AMSApplicationServer.java:73)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.ambari.metrics.AMSApplicationServer.launchAMSApplicationServer(AMSApplicationServer.java:98)
        at org.apache.ambari.metrics.AMSApplicationServer.main(AMSApplicationServer.java:107)
Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:476)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:461)
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:135)
        ... 4 more
Caused by: java.net.BindException: Port in use: dev3:6188
        at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1344)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1366)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1429)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1275)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:472)
        ... 6 more
Caused by: java.io.IOException: Failed to bind to dev3/192.168.3.157:6188
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1331)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1362)
        ... 9 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 12 more
2026-01-27 13:34:46,072 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping Phoenix metrics system...
2026-01-27 13:34:46,072 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system stopped.
2026-01-27 13:34:46,072 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Phoenix metrics system shutdown complete.
2026-01-27 13:34:46,073 FATAL org.apache.ambari.metrics.AMSApplicationServer: Error starting AMSApplicationServer
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: AHSWebApp failed to start.
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:139)
        at org.apache.ambari.metrics.AMSApplicationServer.serviceStart(AMSApplicationServer.java:73)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.ambari.metrics.AMSApplicationServer.launchAMSApplicationServer(AMSApplicationServer.java:98)
        at org.apache.ambari.metrics.AMSApplicationServer.main(AMSApplicationServer.java:107)
Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:476)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:461)
        at org.apache.ambari.metrics.AMSApplicationServer.startWebApp(AMSApplicationServer.java:135)
        ... 4 more
Caused by: java.net.BindException: Port in use: dev3:6188
        at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1344)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1366)
        at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1429)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1275)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:472)
        ... 6 more
Caused by: java.io.IOException: Failed to bind to dev3/192.168.3.157:6188
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1331)
        at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1362)
        ... 9 more
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 12 more
2026-01-27 13:34:46,074 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1: Error starting AMSApplicationServer
2026-01-27 13:34:46,075 INFO org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController: Shooting down Metrics Collector's HAController.
2026-01-27 13:34:46,077 INFO org.apache.helix.manager.zk.ZKHelixManager: Removing listener: org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController$LiveInstanceTracker@42714a7 on path: /ambari-metrics-cluster/LIVEINSTANCES from cluster: ambari-metrics-cluster by instance: dev2
2026-01-27 13:34:46,077 INFO org.apache.ambari.metrics.AMSApplicationServer: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down AMSApplicationServer at dev2/192.168.3.156
************************************************************/
2026-01-27 13:34:46,078 INFO org.apache.helix.manager.zk.CallbackHandler: 106 START:INVOKE /ambari-metrics-cluster/LIVEINSTANCES listener:org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController.LiveInstanceTracker
2026-01-27 13:34:46,078 INFO org.apache.helix.manager.zk.CallbackHandler: dev2 unsubscribe child-change. path: /ambari-metrics-cluster/LIVEINSTANCES, listener: org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController$LiveInstanceTracker@42714a7
2026-01-27 13:34:46,079 INFO org.apache.helix.manager.zk.CallbackHandler: dev2 unsubscribe data-change. path: /ambari-metrics-cluster/LIVEINSTANCES/dev3_12001, listener: org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController$LiveInstanceTracker@42714a7
2026-01-27 13:34:46,080 INFO org.apache.helix.manager.zk.CallbackHandler: dev2 unsubscribe data-change. path: /ambari-metrics-cluster/LIVEINSTANCES/dev2_12001, listener: org.apache.ambari.metrics.core.timeline.availability.MetricCollectorHAController$LiveInstanceTracker@42714a7

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

触发条件（常见组合）

1）安装或扩容 多台 Collector，并希望启用 Collector HA；
2）某些环境下，Collector 会把 WebApp 绑定到 特定 hostname/IP（例如 dev3/192.168.3.157:6188）；
3）当该 IP 在本机并不存在、或本机网络栈不允许绑定该地址时，就会出现 Cannot assign requested address；
4）日志里同时出现 “Port in use” 也并不矛盾：它是 HttpServer2 的统一报错包装，根因仍可能是 绑定地址不合法。

# 二、错误分析与排查思路

# 1、先确认：到底在绑定哪个地址

从日志可直接定位到绑定目标：

Failed to bind to dev3/192.168.3.157:6188
说明 WebApp 不是仅监听 0.0.0.0，而是被配置为绑定到 dev3 / 192.168.3.157

快速判断

Cannot assign requested address：更像是“绑定了一个本机不存在的地址/网卡没起来/绑定策略不允许”。
Address already in use：更像是“本机已有进程占用端口”。两者一起出现时，优先相信日志里那个更具体的：Failed to bind to <host>/<ip>:6188。

# 2、排查清单（建议按顺序执行）

排查项	目的	参考命令
本机是否存在目标 IP	避免绑定到“别人的 IP”	`ip a \\| grep -n "192.168.3.157"`
端口是否被占用	排除真实端口冲突	`ss -lntp \\| grep 6188` / `lsof -i:6188`
hostname 解析是否异常	避免解析到错误 IP	`getent hosts dev3` / `nslookup dev3`
AMS 配置的绑定地址	找到最终落点	Ambari UI / `ams-site.xml`

结论导向

如果你的目标是“Collector WebApp 能稳定起来”，最稳的策略就是让 WebApp 监听 0.0.0.0:6188，把“绑定具体 hostname/IP”的不确定性直接移除。

# 三、解决办法

# 1、优先推荐：改 ams-site 配置（通用、最少改动）

我们主要修改这个配置项（ams-site）：

timeline.metrics.service.webapp.address : 0.0.0.0:6188

为什么改成 0.0.0.0 有效？

0.0.0.0 表示监听本机所有网卡地址；
不再依赖 hostname 解析结果，也不要求绑定到“某个固定 IP”；
对多 Collector、漂移 IP、容器化/多网卡环境更友好。

版本说明

ttr 2.2.3 以上版本：该问题通常已做默认修复，可以忽略手工处理；
ttr 2.2.3 以下版本：建议继续看本文后半段的“源码层修复”，避免被 StackAdvisor 再次推荐回旧值。

# 2、验证 HA：三节点 Collector 启动与页面状态

先启动 3 个 collector 后，页面表现正常：

两个节点变为 online：

# 3、故障演练：停掉一个 Collector 观察自动下线/恢复

我们尝试停止一个 collector：

停止完毕后：

可以看到其它节点的日志会自动对故障节点进行下线，然后重新上线，说明 HA 逻辑是通的：

一句话总结这轮验证看什么？

1）UI/实例列表是否出现 offline → online 的状态变化；
2）其余 Collector 是否能持续对外提供服务；
3）恢复后是否能重新加入集群（不需要手工干预）。

# 四、低版本（< 2.2.3）源码层修复：避免推荐值把你“改回去”

如果版本低于 2.2.3，很多场景下即便你在 UI 改成 0.0.0.0:6188，后续 重新保存配置 / StackAdvisor 重新推荐 时仍可能把值写回“绑定某个主机”的形式。

对应源码位置：

ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/stack_advisor.py

像下图一样注释掉该推荐逻辑：

原来的版本是：

putAmsSiteProperty("timeline.metrics.service.webapp.address", str(metric_collector_host) + ":6188")

把这一行注释掉：

放到服务器上即可（示意）：

注意点

改完 stack_advisor.py 后，务必确保 ambari-server 侧资源生效（常见做法是重启 ambari-server 或刷新相关缓存流程）；
如果你在多环境同步 stack，建议把该改动纳入版本管理，避免后续升级/覆盖丢失。

#Ambari #Ambari-Metrics #AMS #Metrics Collector #HA #ams-site #timeline.metrics.service.webapp.address

← Knox 安装 Atlas 进阶安装技巧→