You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by GitBox <gi...@apache.org> on 2022/09/03 00:49:35 UTC
[GitHub] [bigtop] timyuer opened a new pull request, #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
timyuer opened a new pull request, #1000:
URL: https://github.com/apache/bigtop/pull/1000
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'BIGTOP-3638: Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
- [ ] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi merged pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
guyuqi merged PR #1000:
URL: https://github.com/apache/bigtop/pull/1000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1248846681
ZhiguoWu's deploy order is:
1. Deploy Hadoop, zookeeper, Tez by Ambari UI first.
2. Deploy Hive by Ambari UI first.
3. Enabled Kerberos.
This lead to the `/etc/passwd`(source file MapReduce checked) file size changed, and then cause MapReduce Service Check Failed.
This problem maybe caused by a BUG from resource_management in Ambari Server-2.7.5, I will find it and fix it.
All in all, I can make sure this Hive PR is correct.
Can you please try again? @guyuqi
@kevinw66
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] kevinw66 commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
kevinw66 commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1247877540
@guyuqi works fine on my machine.
But I MapReduce service check failed at first, but succeed when I run it separately after kerberos enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1236013544
Metastore in Hive 3.1.3 is not compatible with Hadoop 3.3.x when Kerberos enabled.
Related to https://github.com/apache/bigtop/pull/1001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1248901857
@timyuer @kevinw66
oops, I found that I did not update my local packages repo and failed to fetch the new built Hive rpms.
After using latest Hadoop and Hive rpms, everything is ok.
LGTM, +1.
Thank you guys for working on it. :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1246461504
@timyuer Thanks for working on it.
But after I combined the https://github.com/apache/bigtop/pull/1001 ,https://github.com/apache/bigtop/pull/1003 with this PR, and rebuilt Hive, it seems `HiveServer2` failed to start:
```
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module>
HiveServer().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/HIVE/package/scripts/hive_server.py", line 53, in start
hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service
wait_for_znode()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 62, in wrapper
return function(*args, **kwargs)
File "/var/lib/ambari-agent/cache/stacks/BGTP/1.0/services/HIVE/package/scripts/hive_service.py", line 196, in wait_for_znode
raise Fail(format("ZooKeeper node /{hive_server2_zookeeper_namespace} is not ready yet"))
resource_management.core.exceptions.Fail: ZooKeeper node /hiveserver2 is not ready yet
```
```
................
.........
.....
2022-09-14 08:49:51,350 - Will retry 5 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-09-14 08:50:01,361 - call['/usr/lib/zookeeper/bin/zkCli.sh -server ambari-server:2181,ambari-agent-01:2181,ambari-agent-02:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-09-14 08:50:02,086 - call returned (1, '')
2022-09-14 08:50:02,087 - Will retry 4 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-09-14 08:50:12,098 - call['/usr/lib/zookeeper/bin/zkCli.sh -server ambari-server:2181,ambari-agent-01:2181,ambari-agent-02:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-09-14 08:50:12,926 - call returned (1, '')
2022-09-14 08:50:12,927 - Will retry 3 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-09-14 08:50:22,938 - call['/usr/lib/zookeeper/bin/zkCli.sh -server ambari-server:2181,ambari-agent-01:2181,ambari-agent-02:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-09-14 08:50:23,686 - call returned (1, '')
2022-09-14 08:50:23,687 - Will retry 2 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-09-14 08:50:33,688 - call['/usr/lib/zookeeper/bin/zkCli.sh -server ambari-server:2181,ambari-agent-01:2181,ambari-agent-02:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-09-14 08:50:34,521 - call returned (1, '')
2022-09-14 08:50:34,522 - Will retry 1 time(s), caught exception: ZooKeeper node /hiveserver2 is not ready yet. Sleeping for 10 sec(s)
2022-09-14 08:50:44,529 - call['/usr/lib/zookeeper/bin/zkCli.sh -server ambari-server:2181,ambari-agent-01:2181,ambari-agent-02:2181 ls /hiveserver2 | grep 'serverUri=''] {}
2022-09-14 08:50:45,282 - call returned (1, '')
```
`Hadoop: 3.3.4`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1236008006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1246465303
I deployed Hadoop, zookeeper, Hive, Tez by Ambari UI first, and then enabled Kerberos:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] timyuer commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
timyuer commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1248863910
> > ZhiguoWu's deploy order is:
> >
> > Deploy Hadoop, zookeeper, Tez by Ambari UI first.
> > Deploy Hive by Ambari UI first.
> > Enabled Kerberos.
> > This lead to the /etc/passwd(source file MapReduce checked) file size changed, and then cause MapReduce Service Check Failed.
> >
> > This problem maybe caused by a BUG from resource_management in Ambari Server-2.7.5, I will find it and fix it.
> >
> > All in all, I can make sure this Hive PR is correct.
> >
> > Can you please try again? @guyuqi @kevinw66 Thanks.
>
> May I ask which Hadoop version you deploy?
Hadoop 3.3.4, compiled by https://ci.bigtop.apache.org/job/Bigtop-trunk-snapshot/.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bigtop] guyuqi commented on pull request #1000: BIGTOP-3802: Fix Mpack Hive fail to start when kerberos enabled
Posted by GitBox <gi...@apache.org>.
guyuqi commented on PR #1000:
URL: https://github.com/apache/bigtop/pull/1000#issuecomment-1248859584
> ZhiguoWu's deploy order is:
>
> 1. Deploy Hadoop, zookeeper, Tez by Ambari UI first.
> 2. Deploy Hive by Ambari UI first.
> 3. Enabled Kerberos.
>
> This lead to the `/etc/passwd`(source file MapReduce checked) file size changed, and then cause MapReduce Service Check Failed.
>
> This problem maybe caused by a BUG from resource_management in Ambari Server-2.7.5, I will find it and fix it.
>
> All in all, I can make sure this Hive PR is correct.
>
> Can you please try again? @guyuqi @kevinw66 Thanks.
May I ask which Hadoop version you deploy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bigtop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org