You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2020/01/23 20:00:20 UTC
[GitHub] [openwhisk] mcdan opened a new pull request #4804: Check to see if
the user can see the topic before creating it,
mcdan opened a new pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804
this allows lower privilege users to be set for the controller
and invoker.
<!--- Provide a concise summary of your changes in the Title -->
## Description
<!--- Provide a detailed description of your changes. -->
<!--- Include details of what problem you are solving and how your changes are tested. -->
Topics in Kafka are not part of the overall functioning of OW. In other words, there is no dynamic topic creation during the normal functioning of any OW component. Therefore there is not reason to have a credential that is powerful enough to make changes to Kafka deployed in the controller or invoker.
This commit allows those pieces to function with "least privilege" under normal operation, meaning you can now set ACLs for their topics in advance and allow them only rights to read / write the topics in question.
## Related issue and scope
<!--- Please include a link to a related issue if there is one. -->
## My changes affect the following components
<!--- Select below all system components are affected by your change. -->
<!--- Enter an `x` in all applicable boxes. -->
- [ ] API
- [ ] Controller
- [X] Message Bus - only Kafka
- [ ] Loadbalancer
- [ ] Invoker
- [ ] Intrinsic actions (e.g., sequences, conductors)
- [ ] Data stores (e.g., CouchDB)
- [ ] Tests
- [ ] Deployment
- [ ] CLI
- [ ] General tooling
- [ ] Documentation
## Types of changes
<!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
- [ ] Bug fix (generally a non-breaking change which closes an issue).
- [X] Enhancement or new feature (adds new functionality).
- [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
## Checklist:
<!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
- [X] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
- [X] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
- [X] I added tests to cover my changes.
- [X] My changes require further changes to the documentation.
- [X] I updated the documentation where necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370672251
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
+ } else {
+ Try(client.createTopics(List(nt).asJava).values().get(topic).get())
+ .map(_ => logging.info(this, s"created topic $topic"))
+ .recoverWith {
+ case CausedBy(_: TopicExistsException) =>
Review comment:
I didn't want to use exceptions for flow control.
Right now if the user can't create a topic it will throw a message that says:
```
[2020-01-23T13:03:59.094Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] completedstandalone already exists and the user can see it, skipping creation.
[2020-01-23T13:04:00.383Z] [ERROR] [#tid_sid_unknown] [KafkaMessagingProvider] ensureTopic for health failed due to java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Topic authorization failed.
```
In this case for this controller startup I didn't pre-create the `health` topic. I think that points to a pretty obvious problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370672251
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
+ } else {
+ Try(client.createTopics(List(nt).asJava).values().get(topic).get())
+ .map(_ => logging.info(this, s"created topic $topic"))
+ .recoverWith {
+ case CausedBy(_: TopicExistsException) =>
Review comment:
I didn't want to use exceptions for flow control.
Right now it will throw a message that says:
```
[2020-01-23T13:03:59.094Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] completedstandalone already exists and the user can see it, skipping creation.
[2020-01-23T13:04:00.383Z] [ERROR] [#tid_sid_unknown] [KafkaMessagingProvider] ensureTopic for health failed due to java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TopicAuthorizationException: Topic authorization failed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] selfxp commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
selfxp commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370410742
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
+ } else {
+ Try(client.createTopics(List(nt).asJava).values().get(topic).get())
+ .map(_ => logging.info(this, s"created topic $topic"))
+ .recoverWith {
+ case CausedBy(_: TopicExistsException) =>
Review comment:
I'm wondering if we could extract the exception that is thrown when the `controllers / invokers` are configured with lower privileged users and display a more personalized message for developer, that would help them realize that the topics have not been created prior to OW deployment but at the same time they can't be created because of permission issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370335825
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
+ } else {
+ Try(client.createTopics(List(nt).asJava).values().get(topic).get())
Review comment:
If it's not there try and create it --- which will keep the same prior flow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan merged pull request #4804: Check to see if the
user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan merged pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370335463
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
Review comment:
List the topics this user can see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on issue #4804: Check to see if the
user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on issue #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#issuecomment-578160720
@chetanmeh
> > there is no dynamic topic creation during the normal functioning of any OW component
>
> Topics like `invokerxxx`, `completedxxx` do get created when a new Invoker or Controller gets provisioned. I believe in low access scenario we would somehow pre-provision the topics and pass known id to controller and invoker at startup
What I mean here is that there isn't something like activation ABC creates Topic ABC in kafka. The sphere of known topics can be computed before any of the components start ( which some limits / orchestration ).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370335567
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
Review comment:
If the topic is there don't bother creating it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] mcdan edited a comment on issue #4804: Check to see if
the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
mcdan edited a comment on issue #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#issuecomment-578160720
@chetanmeh
> > there is no dynamic topic creation during the normal functioning of any OW component
>
> Topics like `invokerxxx`, `completedxxx` do get created when a new Invoker or Controller gets provisioned. I believe in low access scenario we would somehow pre-provision the topics and pass known id to controller and invoker at startup
What I mean here is that there isn't something like activation ABC creates Topic ABC in kafka. The sphere of known topics can be computed before any of the components start ( with some limits / orchestration ).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] chetanmeh commented on issue #4804: Check to see if the
user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
chetanmeh commented on issue #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#issuecomment-577992093
> there is no dynamic topic creation during the normal functioning of any OW component
Topics like `invokerxxx`, `completedxxx` do get created when a new Invoker or Controller gets provisioned. I believe in low access scenario we would somehow pre-provision the topics and pass known id to controller and invoker at startup
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [openwhisk] selfxp commented on a change in pull request #4804:
Check to see if the user can see the topic before creating it,
Posted by GitBox <gi...@apache.org>.
selfxp commented on a change in pull request #4804: Check to see if the user can see the topic before creating it,
URL: https://github.com/apache/openwhisk/pull/4804#discussion_r370798868
##########
File path: common/scala/src/main/scala/org/apache/openwhisk/connector/kafka/KafkaMessagingProvider.scala
##########
@@ -68,19 +68,25 @@ object KafkaMessagingProvider extends MessagingProvider {
val nt = new NewTopic(topic, partitions, kafkaConfig.replicationFactor).configs(topicConfig.asJava)
def createTopic(retries: Int = 5): Try[Unit] = {
- Try(client.createTopics(List(nt).asJava).values().get(topic).get())
- .map(_ => logging.info(this, s"created topic $topic"))
- .recoverWith {
- case CausedBy(_: TopicExistsException) =>
- Success(logging.info(this, s"topic $topic already existed"))
- case CausedBy(t: RetriableException) if retries > 0 =>
- logging.warn(this, s"topic $topic could not be created because of $t, retries left: $retries")
- Thread.sleep(1.second.toMillis)
- createTopic(retries - 1)
- case t =>
- logging.error(this, s"ensureTopic for $topic failed due to $t")
- Failure(t)
- }
+ Try(client.listTopics().names().get())
+ .map(topics =>
+ if (topics.contains(topic)) {
+ Success(logging.info(this, s"$topic already exists and the user can see it, skipping creation."))
+ } else {
+ Try(client.createTopics(List(nt).asJava).values().get(topic).get())
+ .map(_ => logging.info(this, s"created topic $topic"))
+ .recoverWith {
+ case CausedBy(_: TopicExistsException) =>
Review comment:
👍 yeah, it seems that the message is pretty obvious
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services