You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/05/16 00:44:18 UTC
[GitHub] [pulsar] srkukarni opened a new pull request #6972: Add Annotations for config validation checking
srkukarni opened a new pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972
<!--
### Contribution Checklist
- Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
Skip *Issue XYZ* if there is no associated github issue for this pull request.
Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
*(If this PR fixes a github issue, please add `Fixes #<xyz>`.)*
Fixes #<xyz>
*(or if this PR is one task of a github issue, please add `Master Issue: #<xyz>` to link to the master issue.)*
Master Issue: #<xyz>
### Motivation
This pr adds annotations that config classes can use to do member validation. This will be useful for things like source/sink/function submission to check if supplied config is in order before starting the actual source/sink/function
### Modifications
*Describe the modifications you've done.*
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cckellogg commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426102269
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
I like the idea of having type checking. The naming is a bit confusing to me something like makes more sense to me.
```
class TestConfig {
@NotNull
public String stringValue;
@PositiveNumber or @Number(positive=true)
public Integer positiveNumber;
@List(itemType = Integer.class) or @ListType(itemType = Integer.class) or @IntegerList
public List integerList;
@Map(keyType = String.class, valueType = Integer.class) or @MapType(keyType = String.class, valueType = Integer.class)
public Map stringIntegerMap;
@List(itemType = String.class) or @StringList
public List stringList;
@TopicName
public String topic;
@CustomType(validatorClass = TestValidator.class)
public String customString;
}
```
I think annotations should describe what the variable type is as opposed to describing how to test it. The testing is implied.
Another thing to think about is how to handle fields that should be required to have a value.
Thoughts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] srkukarni merged pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
srkukarni merged pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cckellogg commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426734408
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
Sounds good and LGTM +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cckellogg commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426335178
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
Can NotNull be applied in combination with another annotation like this? Or does an annotation like `@TopicName` or `@List` imply not null?
```
@NotNull
@TopicName
public String topic;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] srkukarni commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
srkukarni commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426348720
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
Multiple annotations are possible on a field.
Thus
@NotNull
@TopicName implies that it should be non null and is a topic. However I believe that TopicName already ensures that its not null
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426183847
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
I don't fee strongly about it either way. However, if we feel we can simplify the naming of annotations and still be intuitive to the end user, we should. The current annotations are borrowed from Storm that I created a while ago:
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/validation/ConfigValidationAnnotations.java
Others have name annotations in a more simplified way:
https://www.baeldung.com/javax-validation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cckellogg commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
cckellogg commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426102269
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
I like the idea of having type checking. The naming is a bit confusing to me something like makes more sense to me.
```
class TestConfig {
@NotNull
public String stringValue;
@PositiveNumber or @Number(positive=true)
public Integer positiveNumber;
@List(itemType = Integer.class) or @ListType(itemType = Integer.class) or @IntegerList
public List integerList;
@Map(keyType = String.class, valueType = Integer.class) or @MapType(keyType = String.class, valueType = Integer.class)
public Map stringIntegerMap;
@List(itemType = String.class) or @StringList
public List stringList;
@TopicName
public String topic;
@CustomType(validatorClass = TestValidator.class)
public String customString;
}
```
I think annotations should describe what the variable type is as opposed to describing how to test it. The testing is implied.
Another thing to think about is how to handle fields that should be required to have a value.
Thoughts?
Thoughts? Al
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] srkukarni commented on a change in pull request #6972: Add Annotations for config validation checking
Posted by GitBox <gi...@apache.org>.
srkukarni commented on a change in pull request #6972:
URL: https://github.com/apache/pulsar/pull/6972#discussion_r426188216
##########
File path: pulsar-common/src/test/java/org/apache/pulsar/common/validator/ConfigValidationTest.java
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.common.validator;
+
+import org.testng.annotations.Test;
+
+import java.util.*;
+import static org.testng.Assert.*;
+
+public class ConfigValidationTest {
+
+ private final List<String> testStringList = Arrays.asList(new String[]{"foo", "bar"});
+ private final List<Integer> testIntegerList = Arrays.asList(new Integer[]{0, 1});
+ private final Map<String, Integer> testStringIntegerMap = new HashMap<String, Integer>() {
+ {
+ put("one", 1);
+ put("two", 2);
+ }
+ };
+ private final Map<String, String> testStringStringMap = new HashMap<String, String>() {
+ {
+ put("one", "one");
+ put("two", "two");
+ }
+ };
+ private final String topic = "persistent://public/default/topic";
+
+ public static class TestValidator extends Validator {
+ @Override
+ public void validateField(String name, Object o) {
+ if (o instanceof String) {
+ String value = (String)o;
+ if (!value.startsWith("ABCDE")) {
+ throw new IllegalArgumentException(String.format("Field %s does not start with ABCDE", name));
+ }
+ } else {
+ throw new IllegalArgumentException(String.format("Field %s is not a string", name));
+ }
+ }
+ }
+
+ class TestConfig {
Review comment:
I have changed the annotations. wrt fields that should have a value, isn't that covered by NotNull?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org