You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "touchida (via GitHub)" <gi...@apache.org> on 2023/06/14 15:34:02 UTC
[GitHub] [kyuubi] touchida opened a new pull request, #4964: [KYUUBI #4951] Add reload4j support
touchida opened a new pull request, #4964:
URL: https://github.com/apache/kyuubi/pull/4964
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
It resolves https://github.com/apache/kyuubi/issues/4951.
It would be also possible to resolve it by adding log4j 1 slf4j binding (e.g., adding `slf4j-api-1.7.30.jar` and `slf4j-log4j12-1.7.30`) to the Spark jars directory, but it would be helpful if Kyuubi could support reload4j to avoid log4j 1's security vulnerabilities.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Assume
- we use Spark SQL engine in YARN cluster deploy mode;
- we use Hadoop-free Spark 3.2.4 and provides the classpath of Hadoop in YARN cluster, which uses reload4j, with `SPARK_DIST_CLASSPATH`.
Before applying the patch, Hive beeline shows operation logs only from Kyuubi server, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 22:55:35.584 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 5409e4cb-abe0-457f-b3c8-c3f1c5467c60
2023-06-14 22:55:35.604 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 22:55:35.662 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[013b8294-b395-4e3d-945e-3594b0603f83] in FINISHED_STATE
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.857 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.113 seconds)
```
After applying the patch, Hive beeline shows operation logs from both Kyuubi server and Spark SQL engine, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 23:05:57.424 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials with epoch 0 to SQL engine through session 146aa800-6047-471e-a480-5683197a84cd
2023-06-14 23:05:57.453 INFO org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session credentials epoch from -1 to 0
2023-06-14 23:05:57.512 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement:
Spark application name: kyuubi_CONNECTION_SPARK_SQL_some_user_146aa800-6047-471e-a480-5683197a84cd
application ID: application_1686295570868_0065
application web UI: some_yarn_application_url
master: yarn
deploy mode: cluster
version: 3.2.4
Start time: 2023-06-14T23:05:02.490
User: some_user
(omitted)
2023-06-14 23:06:00.285 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[1c209bea-6f62-4854-acdc-a1945867a65d] in FINISHED_STATE
2023-06-14 23:06:00.286 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.774 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.156 seconds)
```
- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #4964: [KYUUBI #4951] Add reload4j support
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4964:
URL: https://github.com/apache/kyuubi/pull/4964#issuecomment-1592225616
Thanks, merged to master/1.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] touchida commented on pull request #4964: [KYUUBI #4951] Add reload4j support
Posted by "touchida (via GitHub)" <gi...@apache.org>.
touchida commented on PR #4964:
URL: https://github.com/apache/kyuubi/pull/4964#issuecomment-1592251175
@pan3793 @cxzl25
Thanks for your reviews!
> you may want to link your email to your GitHub account
Oh, I see. Thanks for pointing it out!
I will :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #4964: [KYUUBI #4951] Add reload4j support
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4964:
URL: https://github.com/apache/kyuubi/pull/4964#issuecomment-1591691971
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/4964?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#4964](https://app.codecov.io/gh/apache/kyuubi/pull/4964?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d7444be) into [master](https://app.codecov.io/gh/apache/kyuubi/commit/050b5aa7df148fc11cdbe164f72061739566fdc4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (050b5aa) will **not change** coverage.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #4964 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 563 563
Lines 30964 30965 +1
Branches 4035 4036 +1
======================================
- Misses 30964 30965 +1
```
| [Impacted Files](https://app.codecov.io/gh/apache/kyuubi/pull/4964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...mon/src/main/scala/org/apache/kyuubi/Logging.scala](https://app.codecov.io/gh/apache/kyuubi/pull/4964?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9Mb2dnaW5nLnNjYWxh) | `0.00% <0.00%> (ø)` | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #4964: [KYUUBI #4951] Add reload4j support
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4964:
URL: https://github.com/apache/kyuubi/pull/4964#issuecomment-1592229574
@touchida thanks for your contribution, you may want to link your email to your GitHub account, otherwise, your contribution will not be counted by GitHub :)
https://github.com/apache/kyuubi/commit/a9d4abe64a6fa206274a7e78e705c8f251396370
```
...
Authored-by: toshihiko.uchida <to...@linecorp.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed pull request #4964: [KYUUBI #4951] Add reload4j support
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #4964: [KYUUBI #4951] Add reload4j support
URL: https://github.com/apache/kyuubi/pull/4964
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org