[开启Kerberos]-Atlas启动-Solr权限异常
# 一、问题背景
ttr-2.2.1 以上版本已修复
在这些版本中,当开启 Kerberos 后,Atlas 启动可能因 Solr 认证失败 报 401。 高版本已修复该问题,无需额外操作。 如在部署、二开或测试中遇到类似问题,可咨询小饕 (opens new window)
Atlas 启动报错通常分两阶段:
- HBase 无权限(上文已有说明);
- Solr 401 Unauthorized(本篇重点)。
# 二、错误日志分析
Atlas 启动失败时控制台输出如下堆栈信息:
.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:336)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:334)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:209)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:399)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:278)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:111)
at org.apache.atlas.Atlas.main(Atlas.java:133)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT1M40S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:98)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:52)
... 60 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Unable to complete query on Solr.
at org.janusgraph.diskstorage.solr.SolrIndex.storageException(SolrIndex.java:1312)
at org.janusgraph.diskstorage.solr.SolrIndex.mutate(SolrIndex.java:487)
at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:151)
at org.janusgraph.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:148)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:66)
... 61 common frames omitted
Caused by: org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at http://dev1:8983/solr/vertex_index_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Unauthorized access</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized access</h2>
<table>
<tr><th>URI:</th><td>/solr/vertex_index_shard1_replica_n1/update</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized access</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:579)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1076)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:866)
at org.janusgraph.diskstorage.solr.SolrIndex.commitChanges(SolrIndex.java:609)
at org.janusgraph.diskstorage.solr.SolrIndex.mutate(SolrIndex.java:482)
... 64 common frames omitted
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://dev1:8983/solr/vertex_index_shard1_replica_n1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Unauthorized access</title>
</head>
<body><h2>HTTP ERROR 401 Unauthorized access</h2>
<table>
<tr><th>URI:</th><td>/solr/vertex_index_shard1_replica_n1/update</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Unauthorized access</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:635)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:369)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:297)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:555)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[root@dev1 atlas]#
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
可以看到:
Atlas 已成功启动 Jetty,但在初始化 Solr 索引时被拒绝访问。
这通常是因为 Solr 启用了 Kerberos 认证,而 Atlas 端未正确配置 Kerberos HTTP Client。

# 三、问题定位
Atlas 与 Solr 交互依赖 solrj 客户端。 不同版本 Solr 使用的 Kerberos 客户端工厂类不同。若配置不匹配,就会出现 401 错误。
为了确认问题,我们检查相关 JAR 包内是否包含 Kerberos 客户端类。
# 四、版本验证与差异确认
执行以下命令:
jar tf /usr/bigtop/current/atlas-server/server/webapp/atlas/WEB-INF/lib/janusgraph-solr-1.0.0.jar | \
grep -F 'org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.class' \
|| echo 'NOT_FOUND'
jar tf /usr/bigtop/current/atlas-server/server/webapp/atlas/WEB-INF/lib/solr-core-8.11.3.jar | \
grep -F 'org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.class' \
|| echo 'NOT_FOUND'
jar tf /usr/bigtop/current/atlas-server/server/webapp/atlas/WEB-INF/lib/solr-core-8.11.3.jar | \
grep -F 'org/apache/solr/client/solrj/impl/Krb5HttpClientConfigurer.class' \
|| echo 'NOT_FOUND'
2
3
4
5
6
7
8
9
10
11
12
13
结果均为:
NOT_FOUND

表示当前 Atlas 使用的 janusgraph-solr 与 solr-core 中,已无旧版 Krb5HttpClientConfigurer 类。
接着验证 solr-solrj-8.11.3.jar:
jar tf /usr/bigtop/current/atlas-server/server/webapp/atlas/WEB-INF/lib/solr-solrj-8.11.3.jar \
| egrep 'Krb5HttpClientBuilder|Krb5HttpClientConfigurer'
2
输出如下:
[root@dev1 ~]# jar tf /usr/bigtop/current/atlas-server/server/webapp/atlas/WEB-INF/lib/solr-solrj-8.11.3.jar \
> | egrep 'Krb5HttpClientBuilder|Krb5HttpClientConfigurer'
org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder$1.class
org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder$2.class
org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder$SolrJaasConfiguration.class
org/apache/solr/client/solrj/impl/Krb5HttpClientBuilder.class
[root@dev1 ~]#
2
3
4
5
6
7
8
9

结论
Solr 8.x 已弃用旧类 Krb5HttpClientConfigurer,改用 Krb5HttpClientBuilder。
Atlas 默认模板未更新,导致 HTTP 客户端无法加载 Kerberos 认证模块,从而返回 401。
# 五、解决方案
# 1、修改 atlas-env.sh 模板
找到 Ambari 模板:
/usr/bigtop/current/atlas-server/conf/atlas-env.sh
在启用 Kerberos 区间中,加入 Solr HTTPClient 参数(对应上图修改):

{% if security_enabled %}
export ATLAS_OPTS="{{metadata_opts}} \
-Dzookeeper.sasl.client.username={{zk_principal_user}} \
-Djava.security.auth.login.config={{atlas_jaas_file}} \
-Dsolr.kerberos.jaas.appname=Client \
-Djavax.security.auth.useSubjectCredsOnly=false \
-Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder \
-Dsun.security.krb5.debug=true"
{% else %}
export ATLAS_OPTS="{{metadata_opts}}"
{% endif %}
2
3
4
5
6
7
8
9
10
11
这四个关键参数让 Atlas 正确启用 Kerberos 认证的 Solr HTTP 客户端。
| 参数 | 作用 |
|---|---|
solr.kerberos.jaas.appname | 指定 JAAS 模块名称 |
useSubjectCredsOnly=false | 允许 JVM 主动使用 Kerberos 凭据 |
solr.httpclient.builder.factory | 指定新客户端类 Krb5HttpClientBuilder |
sun.security.krb5.debug | 打印调试日志 |
# 2、确认 Solr 版本特性
我们使用的 Solr 为 8.11.3。
在此版本中,只有 solr-solrj-8.11.3.jar 包含 Kerberos 构建器类:

而旧路径下的 janusgraph-solr 与 solr-core 不再包含 Kerberos 支持,因此必须通过上面的环境变量加载新类。
# 3、如 Solr 版本不一致
若 Solr 与 Atlas 所用包版本不匹配,或仍出现连接错误,可同时补充自定义配置:

在 Custom application-properties 中加入以下内容:
atlas.graph.index.search.solr.kerberos.principal=atlas/dev1@TTBIGDATA.COM
atlas.graph.index.search.solr.kerberos.keytab=/etc/security/keytabs/atlas.service.keytab
atlas.graph.index.search.solr.kerberos.enable=true
2
3
这能保证即使 HTTPClient 工厂未加载,也能强制启用 Solr 层的 Kerberos 验证。
如上图所示修改 Custom application-properties
- 01
- [Step2] Ranger Admin HA 自动化安装 自生成凭证02-04
- 02
- 调用 Ranger API 返回 403 问题02-03
- 03
- [Step1] Haproxy 规划与环境安装 Kylin V1002-02