You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/02 08:36:47 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
HeartSaVioR opened a new pull request, #37377:
URL: https://github.com/apache/spark/pull/37377
### What changes were proposed in this pull request?
This PR proposes to use canonical host name for principals in KafkaTestUtils.
This is effectively reverting a part of SPARK-39530, as we used the canonical host name before SPARK-39530. (We used IP as "127.0.0.1" (loopback) and the host part of principal as the canonical host name of "127.0.0.1") Picking up the actual IP to use IPv6 if appropriate remains unchanged from SPARK-39530.
### Why are the changes needed?
KafkaDelegationTokenSuite fails on AWS EC2.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Ran KafkaDelegationTokenSuite in AWS EC2 via following command:
```
build/sbt "sql-kafka-0-10/testOnly *KafkaDelegationTokenSuite"
```
This suite failed on current master branch and succeeds with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1203166822
Failure happens only spark r build - the failure seems to be fairly consistent as I observed the same recently.
https://github.com/HeartSaVioR/spark/runs/7630390731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1203692385
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202191048
@dongjoon-hyun We don't have an environment to verify the functionality for IPv6 - could you please help validating the fix? Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
URL: https://github.com/apache/spark/pull/37377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202189322
cc. @dongjoon-hyun @HyukjinKwon @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1203729055
Thanks all for reviewing and merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202218377
Thank you for pinging me, @HeartSaVioR . Sure, I'll test in 24 hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202244752
Yes, no SPARK related system environment has been set.
> hostname -i
<ipv4>
> hostname -f
ip-<ipv4-with-dot-replaced-with-bar>.<region>.compute.internal
I'll update the PR description to contain the kerberos error log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202276524
Resources I referred on investigation:
https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-user/What-is-a-Kerberos-Principal_003f.html
https://web.mit.edu/kerberos/www/krb5-latest/doc/admin/princ_dns.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1203465399
Filed JIRA for that: https://issues.apache.org/jira/browse/SPARK-39959. We can ignore that for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202225078
BTW, do you mean you have no `SPARK_LOCAL_IP` configuration?
> Ran KafkaDelegationTokenSuite in AWS EC2 via following command:
> build/sbt "sql-kafka-0-10/testOnly *KafkaDelegationTokenSuite"
> This suite failed on current master branch and succeeds with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #37377: [SPARK-39949][SS] Use canonical host name for principals in KafkaTestUtils
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #37377:
URL: https://github.com/apache/spark/pull/37377#issuecomment-1202354609
Failure was from ImageFileFormatSuite which is known to be flaky. It seems to be too much flaky indeed though.
https://github.com/HeartSaVioR/spark/runs/7627422730?check_suite_focus=true
Btw, I just figured out KafkaDelegationTokenSuite in current master branch fails in my MacOS as well with same error message, and this PR fixed the test suite.
I quickly looked into the GA build for the last comment of SPARK-39530, and the host part of the principal was "localhost", which did not trigger the issue. There's no unit-test log file (looks like we don't submit one if the build passes?) so it's not feasible to investigate further.
https://github.com/dongjoon-hyun/spark/runs/6965173075?check_suite_focus=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org