You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by "ninsmiracle (via GitHub)" <gi...@apache.org> on 2024/09/12 03:19:18 UTC
[I] Internal authentication of the cluster failed via Pegasus 2.4 [incubator-pegasus]
ninsmiracle opened a new issue, #2114:
URL: https://github.com/apache/incubator-pegasus/issues/2114
## General Question
when I use Pegasus 2.4 access controller , I can use peagsus-shell access target cluster , but I found all the RPC will be failed in the internal of the cluster.
1. I use `pegasus_prc/pegasus@COMPANY.HADOOP` as my server principal.
Here is my keytab Principal used in target cluster(To check my keytab file is consistent with principal):
```
[work@xxxxxxxx pegasus]$ klist -k pegasus_prc@COMPANY.HADOOP.keytab
Keytab name: FILE:pegasus_prc@COMPANY.HADOOP.keytab
KVNO Principal
---- --------------------------------------------------------------------------
1 pegasus_prc/pegasus@COMPANY.HADOOP
1 pegasus_prc/pegasus@COMPANY.HADOOP
```
2. Here is the config in target cluster's config.ini:
```
[[security]]
enable_acl = false
super_users = u_guoningshen
service_name = pegasus_prc
service_fqdn = pegasus
sasl_plugin_path = /usr/lib64/sasl2
krb5_keytab = /home/work/app/pegasus/pegasus_prc@COMPANY.HADOOP.keytab
krb5_config = /home/work/app/pegasus/krb5.conf
krb5_principal = pegasus_prc/pegasus@COMPANY.HADOOP
mandatory_auth = false
enable_auth = true
```
3. Here is my pegasus-shell ini file , and I use it to access target cluster
```
[apps..default]
run = true
count = 1
[apps.mimic]
type = dsn.app.mimic
arguments =
pools = THREAD_POOL_DEFAULT,THREAD_POOL_META_SERVER
run = true
count = 1
[core]
tool = nativerun
pause_on_start = false
logging_start_level = LOG_LEVEL_DEBUG
logging_factory_name = dsn::tools::simple_logger
logging_flush_on_exit = false
enable_default_app_mimic = true
data_dir = ./pegasus_shell.data
[tools.simple_logger]
short_header = false
fast_flush = true
max_number_of_log_files_on_disk = 10
stderr_start_level = LOG_LEVEL_FATAL
[tools.simulator]
random_seed = 0
[network]
io_service_worker_count = 4
[threadpool..default]
worker_count = 4
partitioned = false
worker_priority = THREAD_xPRIORITY_NORMAL
[threadpool.THREAD_POOL_DEFAULT]
name = default
worker_count = 20
[threadpool.THREAD_POOL_META_SERVER]
name = meta_server
[task..default]
is_trace = false
is_profile = false
allow_inline = false
rpc_call_header_format = NET_HDR_DSN
rpc_call_channel = RPC_CHANNEL_TCP
rpc_timeout_milliseconds = 10000
[pegasus.clusters]
c4tst-function2 = 10.xxx.xx.1:32601,10.xxx.xx.2:32601
[security]
enable_auth = true
krb5_keytab = /home/work/2.4.4_pegasus/pegasus/u_guoningshen.keytab
krb5_config = /etc/krb5.conf
krb5_principal = u_guoningshen@COMPANY.HADOOP
sasl_plugin_path = /home/work/2.4.4_pegasus/pegasus/thirdparty/output/lib/sasl2
service_fqdn = pegasus
service_name = pegasus_prc
```
4. What happen?
* Connected to cluster via pegasus-shell
```
./run.sh shell -c ker.ini
```
* `u_guoningshen` is super user for cluster , so I have full permissions.
```
The cluster name is: c4tst-function2
The cluster meta list is: 10.xxx.xx.1:32601,10.xxx.xx.2:32601
>>> ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
238 AVAILABLE test pegasus 4 3 true 2024-09-11_07:30:20 - - 0
239 AVAILABLE gns pegasus 4 3 true 2024-09-12_02:30:50 - - 0
[summary]
total_app_count : 2
>>> drop gns
reserve_seconds = 0
drop app gns succeed
>>> ls
[general_info]
app_id status app_name app_type partition_count replica_count is_stateful create_time drop_time drop_expire envs_count
238 AVAILABLE test pegasus 4 3 true 2024-09-11_07:30:20 - - 0
[summary]
total_app_count : 1
>>>
```
* But I can not create table, because all the rpc send from master meta to another nodes will `negotiation failed, with err = ERR_UNKNOWN, msg = ERR_UNKNOWN`
```
>>> create gns_test
create app gns_test succeed, waiting for app ready
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
gns_test not ready yet, still waiting... (0/4)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
Re: [I] Internal authentication of the cluster failed via Pegasus 2.4 [incubator-pegasus]
Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on issue #2114:
URL: https://github.com/apache/incubator-pegasus/issues/2114#issuecomment-2345197032
When I check the app status:
```
[replicas]
pidx ballot replica_count primary secondaries
0 0 0/3 - []
1 0 0/3 - []
2 0 0/3 - []
3 0 0/3 - []
```
We can see that even the primary is not successfully established. So I check the log of replica server for this replica:
```
D2024-09-11 15:32:08.284 (1726039928284328485 93977) replica.io-thrd.93977: server_negotiation.cpp:40:start(): SERVER_NEGOTIATION(CLIENT=10.xxx.xx.1:41689): start negotiation
D2024-09-11 15:32:08.284 (1726039928284356416 93977) replica.io-thrd.93977: network.cpp:696:on_server_session_accepted(): server session accepted, remote_client = 10.xxx.xx.1:41689, current_count = 1
D2024-09-11 15:32:08.284 (1726039928284364187 93977) replica.io-thrd.93977: network.cpp:701:on_server_session_accepted(): ip session inserted, remote_client = 10.xxx.xx.1:41689, current_count = 1
W2024-09-11 15:32:08.289 (1726039928289020684 93993) replica.default10.04006f170001004b: server_negotiation.cpp:137:do_challenge(): SERVER_NEGOTIATION(CLIENT=10.xxx.xx.1:41689): negotiation failed, with err = ERR_UNKNOWN, msg = ERR_UNKNOWN
D2024-09-11 15:32:08.289 (1726039928289039571 93993) replica.default10.04006f170001004b: network.cpp:738:on_server_session_disconnected(): session 10.xxx.xx1:41689 disconnected, the total client sessions count remains 0
D2024-09-11 15:32:08.289 (1726039928289046389 93993) replica.default10.04006f170001004b: network.cpp:744:on_server_session_disconnected(): client ip 10.xxx.xx.1:41689 has no more session to this server
E2024-09-11 15:32:08.289 (1726039928289069504 93975) replica.io-thrd.93975: asio_rpc_session.cpp:96:operator()(): asio read from 10.xxx.xx.1:41689 failed: Operation canceled
```
I think the key message is `server_negotiation.cpp:137:do_challenge(): SERVER_NEGOTIATION(CLIENT=10.xxx.xx.1:41689): negotiation failed, with err = ERR_UNKNOWN, msg = ERR_UNKNOWN`
In my opinion, it failed on `SASL_INITIATE` step of SASL process between meta server and replica server.
```
client server
| --- SASL_LIST_MECHANISMS --> |
| <-- SASL_LIST_MECHANISMS_RESP --- |
| -- SASL_SELECT_MECHANISMS --> |
| <-- SASL_SELECT_MECHANISMS_RESP --- |
| |
| --- SASL_INITIATE --> |
| |
| <-- SASL_CHALLENGE --- |
| --- SASL_CHALLENGE_RESP --> |
| |
| ..... |
| |
| <-- SASL_CHALLENGE --- |
| --- SASL_CHALLENGE_RESP --> |
| |
| |
| <-- SASL_SUCC --- |
| |
| |
| --- RPC_CALL ---> |
| <-- RPC_RESP ---- |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
Re: [I] Internal authentication of the cluster failed via Pegasus 2.4 [incubator-pegasus]
Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on issue #2114:
URL: https://github.com/apache/incubator-pegasus/issues/2114#issuecomment-2345207036
I also check the log in KDC:
```
Sep 11 15:26:24 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): AS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): NEEDED_PREAUTH: pegasus_prc/pegasus@COMPANY.HADOOP for krbtgt/COMPANY.HADOOP@COMPANY.HADOOP, Additional pre-authentication required
Sep 11 15:26:24 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): ISSUE: authtime 1726039584, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for pegasus_prc/pegasus@COMPANY.HADOOP
Sep 11 15:26:38 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): TGS_REQ (4 etypes {18 17 16 23}) [10.132.5.3](http://10.132.5.3/): ISSUE: authtime 1726039598, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for pegasus_prc/pegasus@COMPANY.HADOOP
Sep 11 15:26:47 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): ISSUE: authtime 1726039567, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for pegasus_prc/pegasus@COMPANY.HADOOP
Sep 11 15:26:47 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): ISSUE: authtime 1726039567, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for pegasus_prc/pegasus@COMPANY.HADOOP
Sep 11 15:26:48 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): AS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): ISSUE: authtime 1726039608, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for krbtgt/COMPANY.HADOOP@COMPANY.HADOOP
Sep 11 15:27:48 [c4-hadoop-krb02.bj](http://c4-hadoop-krb02.bj/) krb5kdc[59974](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) [10.XXX.XX.1](http://10.XXX.XX.1/): ISSUE: authtime 1726039608, etypes {rep=18 tkt=18 ses=18}, pegasus_prc/pegasus@COMPANY.HADOOP for pegasus_prc/pegasus@COMPANY.HADOOP
```
I can see `TGS_REQ` here, so I think client(here is meta server,use sasl client side to pass the permission check on replica servers) can get TGT from KDC.
And I check backup mate server, still have some error on it:
```
server_negotiation.cpp:137:do_challenge(): SERVER_NEGOTIATION(CLIENT=10.XXX.XX.1:55545): negotiation failed, with err = ERR_UNKNOWN, msg = ERR_UNKNOWN
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
Re: [I] Internal authentication of the cluster failed via Pegasus 2.4 [incubator-pegasus]
Posted by "ninsmiracle (via GitHub)" <gi...@apache.org>.
ninsmiracle commented on issue #2114:
URL: https://github.com/apache/incubator-pegasus/issues/2114#issuecomment-2345215271
I also tried the following. When I changed the principal used by the pegasus-shell to be consistent with the server, I found that the pegasus-shell could not connect to the cluster at all. When you enter the `ls` command, the pegasus-shell will return an `ERR_TIMEOUT`. I checked the information from internet and it shows that when sasl calls kerberos, both ends of sasl cannot be set to the same principal.
So I was confused. If this is the case, different nodes in the cluster must use the same principal, because we cannot assign a principal to each node. In this way, internal verification will definitely fail.
At the same time, I noticed that `mandatory_auth = false` was developed in the code, and it seems that the internal verification between nodes can be skipped from the code logic. But I don’t know why it didn’t work.
There is too little log information in various places, and the only error `ERR_UNKNOW` report is not very usefull, so my work is in trouble...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org