You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2021/06/22 10:43:14 UTC
[GitHub] [storm] wxy929629 opened a new pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
wxy929629 opened a new pull request #3377:
URL: https://github.com/apache/storm/pull/3377
when use SASL_PLAIN kafka JAAS auth, missing sasl.mechanism will lead to KafkaOffsetLagUtil cant pull the offset correctly
kafka version 2.2.0
Apache JIRA url: https://issues.apache.org/jira/browse/STORM-3404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
Ethanlm edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-800633946
@wxy929629 Thanks for contributing!
Can you please help me understand this problem better?
Do you mean when Kafka broker is set up with SASL_PLAIN, missing sasl.mechanism will lead to failures? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
Also did you test this with Kafka-1.x and make sure it still works?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
Ethanlm edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-800633946
@wxy929629 Thanks for contributing!
Can you please help me understand this problem better?
Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
Also did you test this with Kafka-1.x and make sure it still works?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-812418213
> @wxy929629 Thanks for contributing!
>
> Can you please help me understand this problem better?
>
> Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
>
> Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
>
> Also did you test this with Kafka-1.x and make sure it still works?
>
> Thanks
Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, plz review and find if there is any other problem.
Because my test environment is not ready any more, so i perhaps can't provide the log and trace for what happend.But i think the failures is easy to reappearance on any kafka brokers which setup SASL_PLAIN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 commented on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 commented on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-775756787
> @wxy929629 Thanks. Could you fix the checkstyle issues that have been introduced? The build output has the following lines
>
> ```
> [WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:147:86: WhitespaceAround: '{' is not preceded with whitespace. [WhitespaceAround]
>
> [WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:149: 'if rcurly' has incorrect indentation level 14, expected level should be 16. [Indentation]
> ```
>
> I.e. there's some indentation that needs to be fixed in the lines you added.
@srdo
Ok i have done this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 commented on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 commented on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-775602960
@srdo I have already change the target to the master branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
Ethanlm edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-800633946
@wxy929629 Thanks for contributing!
Can you please help me understand this problem better?
Do you mean when Kafka broker is set up with SASL_PLAIN, missing sasl.mechanism will lead to failures (by default, sasl.mechanism is `GSSAPI`)? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
Also did you test this with Kafka-1.x and make sure it still works?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-800633946
@wxy929629 Thanks for contributing!
Can you please help me understand this problem better?
`sasl.mechanism` is set `GSSAPI` if not set. Why would it error when the security protocol `SASL_PLAINTEXT` is used? Does it only fail when the security protocol is set `SASL_PLAINTEXT`? What happens to other protocols like `PLAINTEXT`, `SASL_SSL` and `SSL`?
Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
Also did you test this with Kafka-1.x and make sure it still works?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-812418213
> @wxy929629 Thanks for contributing!
>
> Can you please help me understand this problem better?
>
> Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
>
> Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
>
> Also did you test this with Kafka-1.x and make sure it still works?
>
> Thanks
Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, please review it
Because my test environment is not ready any more, so i perhaps can't provide the log and trace for what happend.But i think the failures is easy to reappearance on any kafka brokers which setup SASL_PLAIN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 closed pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 closed pull request #3377:
URL: https://github.com/apache/storm/pull/3377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-812418213
> @wxy929629 Thanks for contributing!
>
> Can you please help me understand this problem better?
>
> Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
>
> Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
>
> Also did you test this with Kafka-1.x and make sure it still works?
>
> Thanks
Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, plz review and find if there is any other problem.
Because my test environment is not ready any more, so i perhaps can't provide the log and trace for what happend.But i think the failures is easy to reappearance on any kafka setup SASL_PLAIN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] srdo commented on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
srdo commented on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-775747115
@wxy929629 Thanks. Could you fix the checkstyle issues that have been introduced? The build output has the following lines
```
[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:147:86: WhitespaceAround: '{' is not preceded with whitespace. [WhitespaceAround]
[WARN] /home/travis/build/apache/storm/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java:149: 'if rcurly' has incorrect indentation level 14, expected level should be 16. [Indentation]
````
I.e. there's some indentation that needs to be fixed in the lines you added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 edited a comment on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 edited a comment on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-812418213
> @wxy929629 Thanks for contributing!
>
> Can you please help me understand this problem better?
>
> Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
>
> Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
>
> Also did you test this with Kafka-1.x and make sure it still works?
>
> Thanks
Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, plz review and find if there is any other problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] wxy929629 commented on pull request #3377: [STORM-3404]KafkaOffsetLagUtil cant pull the offset correctly
Posted by GitBox <gi...@apache.org>.
wxy929629 commented on pull request #3377:
URL: https://github.com/apache/storm/pull/3377#issuecomment-812418213
> @wxy929629 Thanks for contributing!
>
> Can you please help me understand this problem better?
>
> Do you mean when Kafka broker is set up with `SASL_PLAIN`, missing `sasl.mechanism` will lead to failures (by default, `sasl.mechanism` is [GSSAPI](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/SaslConfigs.java#L31-L32))? If in that case, does it make sense to provide an option for users to choose `sasl.mechanism`, just like what we have for security.protocol ? https://github.com/apache/storm/blob/master/external/storm-kafka-monitor/src/main/java/org/apache/storm/kafka/monitor/KafkaOffsetLagUtil.java#L118
>
> Can you please explain more on what happened, by providing stack trace and logs if possible? Also the jira indicates it is tested on storm 1.2.2. Can you provide the logs and stack trace from the testing from storm master branch? Although I understand this is mostly about kafka client setup, not very related to storm version, but it is still nice to see since this PR is based on storm master branch. It will help reviewers and other storm users to understand this better.
>
> Also did you test this with Kafka-1.x and make sure it still works?
>
> Thanks
Yes, when i set the SASL_PLAIN to connect to the kafka, i can not pull the lag info correctly. So i need to add a property sasl.mechanism=PLAIN, i provide an option for users to choose sasl.mechanism to this PR, plz review and find if there is any other question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org