You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/15 13:18:14 UTC
[GitHub] [pulsar] SignorMercurio opened a new pull request, #16627: [feature][doc] Generate config docs from source code
SignorMercurio opened a new pull request, #16627:
URL: https://github.com/apache/pulsar/pull/16627
Master issue: #13453
### Motivation
Please checkout the master issue.
### Modifications
The replacement logic in `BaseGenerateDocumentation` is changed. A new config docs for broker is added and the corresponding link in `reference-configuration` is changed.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change is a trivial rework / code cleanup without any test coverage.
### Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API: no
- The schema: no
- The default values of configurations: no
- The wire protocol: no
- The rest endpoints: no
- The admin cli options: no
- Anything that affects deployment: no
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why: This PR only changes GitHub Actions workflow)
- [x] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925840407
##########
site2/docs/reference-configuration-standalone.md:
##########
@@ -0,0 +1,1465 @@
+# Standalone
+
+## Optional
+
+### authenticateOriginalAuthData
+
+If this flag is set to `true`, the broker authenticates the original Auth data; else it just accepts the originalPrincipal and authorizes it (if required).
+
+**Default**: false
+
+### metadataStoreUrl
+
+The quorum connection string for local metadata store
+
+**Default**:
Review Comment:
Docs for standalone is written manually, and I'm not sure about the reason why there's no default value when it should have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1190258138
@Anonymitaet PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925727981
##########
site2/docs/reference-configuration-bookkeeper.md:
##########
@@ -0,0 +1,585 @@
+# BookKeeper
+
+BookKeeper is a replicated log storage system that Pulsar uses for durable storage of all messages.
+
+## Optional
Review Comment:
@gaoran10 are the configurations of `BookKeeper, client, log4j, log4j shell, standalone, ZooKeeper` all optional?
##########
site2/docs/reference-configuration-broker.md:
##########
@@ -0,0 +1,3879 @@
+# Broker
+## Required
+### clusterName
+Name of the cluster to which this broker belongs to
+
+**Default**: `null`
+
+**Dynamic**: `false`
+
+**Category**: Server
+
+## Optional
+### acknowledgmentAtBatchIndexLevelEnabled
+Whether to enable the acknowledge of batch local index
+
Review Comment:
@gaoran10 is this correct?
<img width="569" alt="image" src="https://user-images.githubusercontent.com/50226895/180017827-95e7a9df-2d13-4180-b5e4-ba89e2e00759.png">
##########
site2/docs/reference-configuration-pulsar-proxy.md:
##########
@@ -0,0 +1,934 @@
+# Pulsar proxy
+## Required
Review Comment:
If there is no `required` configuration, can you add an explanation? Something like `All configurations for Pulsar proxy are optional`?
##########
site2/docs/reference-configuration-websocket.md:
##########
@@ -0,0 +1,495 @@
+# WebSocket
+## Required
+### clusterName
+Name of the cluster to which this broker belongs to
+
+**Default**: `null`
+
+**Dynamic**: `false`
+
+**Category**:
Review Comment:
If all configurations do not have category values and will not have in the future, consider not showing `category` because it does not provide useful info to users?
##########
site2/docs/reference-configuration-standalone.md:
##########
@@ -0,0 +1,1465 @@
+# Standalone
+
+## Optional
+
+### authenticateOriginalAuthData
+
+If this flag is set to `true`, the broker authenticates the original Auth data; else it just accepts the originalPrincipal and authorizes it (if required).
+
+**Default**: false
+
+### metadataStoreUrl
+
+The quorum connection string for local metadata store
+
+**Default**:
Review Comment:
Why no value? If it does not have a default value, can you add an explanation? Something like `it does not have a default value`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1192155095
As we discussed just now:
🔹🔹🔹🔹🔹🔹
For Pulsar components (broker, websocket, proxy, client, standalone):
- Categorize them as `Required`, `Optional`, `Deprecated`
- Show all properties no matter if it has a value.
- Add a `note` to the beginning of each doc. Something like:
> This page is automatically generated from code files. If you find something inaccurate, feel free to update [configuration-file-name-placeholder](URL-placeholder). Do not edit this markdown file manually. Manual changes will be overridden by each automatic generation.
🔹🔹🔹🔹🔹🔹
For non-Pulsar components (BK, log4j, log4jshell, ZK)
- Do not categorize them as `Required`, `Optional`, `Deprecated`. It's hard to define them since docs are generated from external files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1196232302
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1196197186
Great work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1196172011
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] urfreespace commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
urfreespace commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r928448702
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/BaseGenerateDocumentation.java:
##########
@@ -80,50 +82,107 @@ public boolean run(String[] args) throws Exception {
protected abstract String generateDocumentByClassName(String className) throws Exception;
+ protected Predicate<Field> isRequired = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.required();
+ };
+
+ protected Predicate<Field> isOptional = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return !fieldContext.deprecated() && !fieldContext.required();
+ };
+
+ protected Predicate<Field> isDeprecated = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.deprecated();
+ };
+
+ protected void writeDocListByFieldContext(List<Field> fieldList, StringBuilder sb, Object obj) throws Exception {
+ for (Field field : fieldList) {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ field.setAccessible(true);
+
+ sb.append("### ").append(field.getName()).append("\n");
+ sb.append(fieldContext.doc().replace(">", "\\>")).append("\n\n");
+ sb.append("**Default**: `").append(field.get(obj)).append("`\n\n");
+ sb.append("**Dynamic**: `").append(fieldContext.dynamic()).append("`\n\n");
+ sb.append("**Category**: ").append(fieldContext.category()).append("\n\n");
+ }
+ }
+
+ protected static class CategoryComparator implements Comparator<Field> {
+ @Override
+ public int compare(Field o1, Field o2) {
+ FieldContext o1Context = o1.getAnnotation(FieldContext.class);
+ FieldContext o2Context = o2.getAnnotation(FieldContext.class);
+
+ if (o1Context.category().equals(o2Context.category())) {
+ return o1.getName().compareTo(o2.getName());
+ }
+ return o1Context.category().compareTo(o2Context.category());
+ }
+ }
+
+ protected String prefix = """
+ :::note
+
+ This page is automatically generated from code files.
+ If you find something inaccurate, feel free to update `""";
+ protected String suffix = """
+ `. Do NOT edit this markdown file manually. Manual changes will be overwritten by automatic generation.
+
+ :::
+ """;
+
protected String generateDocByFieldContext(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.removeIf(f -> f.getAnnotation(FieldContext.class) == null);
+ fieldList.sort(new CategoryComparator());
+ List<Field> requiredFields = fieldList.stream().filter(isRequired).toList();
+ List<Field> optionalFields = fieldList.stream().filter(isOptional).toList();
+ List<Field> deprecatedFields = fieldList.stream().filter(isDeprecated).toList();
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|Dynamic|Category|\n");
- sb.append("|---|---|---|---|---|\n");
- for (Field field : fields) {
- FieldContext fieldContext = field.getAnnotation(FieldContext.class);
- if (fieldContext == null) {
- continue;
- }
- field.setAccessible(true);
- sb.append("| ").append(field.getName()).append(" | ");
- sb.append(fieldContext.doc().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append(fieldContext.dynamic()).append(" | ");
- sb.append(fieldContext.category()).append(" | ");
- sb.append("\n");
- }
Review Comment:
Won't these changes affect the generation of other documents?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1196235866
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1197855724
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925850338
##########
site2/docs/reference-configuration-websocket.md:
##########
@@ -0,0 +1,495 @@
+# WebSocket
+## Required
+### clusterName
+Name of the cluster to which this broker belongs to
+
+**Default**: `null`
+
+**Dynamic**: `false`
+
+**Category**:
Review Comment:
I think so. However, there're some docs like `reference-configuration-broker` that do have values for `Category`, and docs for `broker` and `websocket` are generated by the same code. We could certainly add some additional conditions to eliminate these useless `Category` and do the trick, but it would made the code hard to extend and clumsy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1195214195
@Anonymitaet No, they are not related at all, please remove them. They are for adding a new workflow to delete stale workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1198203540
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged PR #16627:
URL: https://github.com/apache/pulsar/pull/16627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] dave2wave commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
dave2wave commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1186281102
This looks like a good pan if you follow through with all of configuration categories!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1192600376
@urfreespace Ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1194163759
> All related PRs:
The last two PRs are not related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925647256
##########
site2/docs/reference-configuration-client.md:
##########
@@ -0,0 +1,222 @@
+# Client
Review Comment:
Hi @gaoran10 are the configurations of `client` all `optional`? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925724336
##########
site2/docs/reference-configuration-standalone.md:
##########
@@ -0,0 +1,1465 @@
+# Standalone
+
+## Optional
Review Comment:
Hi @gaoran10 are the configurations of `standalone` all `optional`? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1197723086
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1196371680
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] gaoran10 commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
gaoran10 commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r927231288
##########
site2/docs/reference-configuration-broker.md:
##########
@@ -0,0 +1,3879 @@
+# Broker
+## Required
+### clusterName
+Name of the cluster to which this broker belongs to
+
+**Default**: `null`
+
+**Dynamic**: `false`
+
+**Category**: Server
+
+## Optional
+### acknowledgmentAtBatchIndexLevelEnabled
+Whether to enable the acknowledge of batch local index
+
Review Comment:
I think it's ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925724336
##########
site2/docs/reference-configuration-standalone.md:
##########
@@ -0,0 +1,1465 @@
+# Standalone
+
+## Optional
Review Comment:
Hi @gaoran10 are the configurations of `standalone` all `optional`? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1194888066
> > All related PRs:
>
> The last two PRs are not related.
Sorry, I guess these PRs are loosely related as they are prepared for adding a workflow to sync configuration docs from `pulsar` to `pulsar-repo`? If not, I'll remove them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1197717130
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1193519069
Hi @gaoran10 @urfreespace
could you please review these PRs from the technical perspective? Thank you!
All related PRs:
- https://github.com/apache/pulsar/pull/16627
- https://github.com/apache/pulsar-site/pull/147
- https://github.com/apache/pulsar/pull/16539
- https://github.com/apache/pulsar-test-infra/pull/55
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] urfreespace commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
urfreespace commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r928448872
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/BaseGenerateDocumentation.java:
##########
@@ -80,50 +82,107 @@ public boolean run(String[] args) throws Exception {
protected abstract String generateDocumentByClassName(String className) throws Exception;
+ protected Predicate<Field> isRequired = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.required();
+ };
+
+ protected Predicate<Field> isOptional = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return !fieldContext.deprecated() && !fieldContext.required();
+ };
+
+ protected Predicate<Field> isDeprecated = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.deprecated();
+ };
+
+ protected void writeDocListByFieldContext(List<Field> fieldList, StringBuilder sb, Object obj) throws Exception {
+ for (Field field : fieldList) {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ field.setAccessible(true);
+
+ sb.append("### ").append(field.getName()).append("\n");
+ sb.append(fieldContext.doc().replace(">", "\\>")).append("\n\n");
+ sb.append("**Default**: `").append(field.get(obj)).append("`\n\n");
+ sb.append("**Dynamic**: `").append(fieldContext.dynamic()).append("`\n\n");
+ sb.append("**Category**: ").append(fieldContext.category()).append("\n\n");
+ }
+ }
+
+ protected static class CategoryComparator implements Comparator<Field> {
+ @Override
+ public int compare(Field o1, Field o2) {
+ FieldContext o1Context = o1.getAnnotation(FieldContext.class);
+ FieldContext o2Context = o2.getAnnotation(FieldContext.class);
+
+ if (o1Context.category().equals(o2Context.category())) {
+ return o1.getName().compareTo(o2.getName());
+ }
+ return o1Context.category().compareTo(o2Context.category());
+ }
+ }
+
+ protected String prefix = """
+ :::note
+
+ This page is automatically generated from code files.
+ If you find something inaccurate, feel free to update `""";
+ protected String suffix = """
+ `. Do NOT edit this markdown file manually. Manual changes will be overwritten by automatic generation.
+
+ :::
+ """;
+
protected String generateDocByFieldContext(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.removeIf(f -> f.getAnnotation(FieldContext.class) == null);
+ fieldList.sort(new CategoryComparator());
+ List<Field> requiredFields = fieldList.stream().filter(isRequired).toList();
+ List<Field> optionalFields = fieldList.stream().filter(isOptional).toList();
+ List<Field> deprecatedFields = fieldList.stream().filter(isDeprecated).toList();
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|Dynamic|Category|\n");
- sb.append("|---|---|---|---|---|\n");
- for (Field field : fields) {
- FieldContext fieldContext = field.getAnnotation(FieldContext.class);
- if (fieldContext == null) {
- continue;
- }
- field.setAccessible(true);
- sb.append("| ").append(field.getName()).append(" | ");
- sb.append(fieldContext.doc().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append(fieldContext.dynamic()).append(" | ");
- sb.append(fieldContext.category()).append(" | ");
- sb.append("\n");
- }
+
+ sb.append(prefix).append(className).append(suffix);
+ sb.append("## Required\n");
+ writeDocListByFieldContext(requiredFields, sb, obj);
+ sb.append("## Optional\n");
+ writeDocListByFieldContext(optionalFields, sb, obj);
+ sb.append("## Deprecated\n");
+ writeDocListByFieldContext(deprecatedFields, sb, obj);
+
return sb.toString();
}
protected String generateDocByApiModelProperty(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.sort(Comparator.comparing(Field::getName));
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|\n");
- sb.append("|---|---|---|\n");
Review Comment:
Won't these changes affect the generation of other documents?
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/BaseGenerateDocumentation.java:
##########
@@ -80,50 +82,107 @@ public boolean run(String[] args) throws Exception {
protected abstract String generateDocumentByClassName(String className) throws Exception;
+ protected Predicate<Field> isRequired = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.required();
+ };
+
+ protected Predicate<Field> isOptional = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return !fieldContext.deprecated() && !fieldContext.required();
+ };
+
+ protected Predicate<Field> isDeprecated = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.deprecated();
+ };
+
+ protected void writeDocListByFieldContext(List<Field> fieldList, StringBuilder sb, Object obj) throws Exception {
+ for (Field field : fieldList) {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ field.setAccessible(true);
+
+ sb.append("### ").append(field.getName()).append("\n");
+ sb.append(fieldContext.doc().replace(">", "\\>")).append("\n\n");
+ sb.append("**Default**: `").append(field.get(obj)).append("`\n\n");
+ sb.append("**Dynamic**: `").append(fieldContext.dynamic()).append("`\n\n");
+ sb.append("**Category**: ").append(fieldContext.category()).append("\n\n");
+ }
+ }
+
+ protected static class CategoryComparator implements Comparator<Field> {
+ @Override
+ public int compare(Field o1, Field o2) {
+ FieldContext o1Context = o1.getAnnotation(FieldContext.class);
+ FieldContext o2Context = o2.getAnnotation(FieldContext.class);
+
+ if (o1Context.category().equals(o2Context.category())) {
+ return o1.getName().compareTo(o2.getName());
+ }
+ return o1Context.category().compareTo(o2Context.category());
+ }
+ }
+
+ protected String prefix = """
+ :::note
+
+ This page is automatically generated from code files.
+ If you find something inaccurate, feel free to update `""";
+ protected String suffix = """
+ `. Do NOT edit this markdown file manually. Manual changes will be overwritten by automatic generation.
+
+ :::
+ """;
+
protected String generateDocByFieldContext(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.removeIf(f -> f.getAnnotation(FieldContext.class) == null);
+ fieldList.sort(new CategoryComparator());
+ List<Field> requiredFields = fieldList.stream().filter(isRequired).toList();
+ List<Field> optionalFields = fieldList.stream().filter(isOptional).toList();
+ List<Field> deprecatedFields = fieldList.stream().filter(isDeprecated).toList();
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|Dynamic|Category|\n");
- sb.append("|---|---|---|---|---|\n");
- for (Field field : fields) {
- FieldContext fieldContext = field.getAnnotation(FieldContext.class);
- if (fieldContext == null) {
- continue;
- }
- field.setAccessible(true);
- sb.append("| ").append(field.getName()).append(" | ");
- sb.append(fieldContext.doc().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append(fieldContext.dynamic()).append(" | ");
- sb.append(fieldContext.category()).append(" | ");
- sb.append("\n");
- }
+
+ sb.append(prefix).append(className).append(suffix);
+ sb.append("## Required\n");
+ writeDocListByFieldContext(requiredFields, sb, obj);
+ sb.append("## Optional\n");
+ writeDocListByFieldContext(optionalFields, sb, obj);
+ sb.append("## Deprecated\n");
+ writeDocListByFieldContext(deprecatedFields, sb, obj);
+
return sb.toString();
}
protected String generateDocByApiModelProperty(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.sort(Comparator.comparing(Field::getName));
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|\n");
- sb.append("|---|---|---|\n");
- for (Field field : fields) {
+ sb.append(prefix).append(className).append(suffix);
+ sb.append("## Optional\n");
+ for (Field field : fieldList) {
ApiModelProperty fieldContext = field.getAnnotation(ApiModelProperty.class);
if (fieldContext == null) {
continue;
}
field.setAccessible(true);
+
String name = StringUtils.isBlank(fieldContext.name()) ? field.getName() : fieldContext.name();
- sb.append("| ").append(name).append(" | ");
- sb.append(fieldContext.value().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append("\n");
Review Comment:
Won't these changes affect the generation of other documents?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r928839387
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/BaseGenerateDocumentation.java:
##########
@@ -80,50 +82,107 @@ public boolean run(String[] args) throws Exception {
protected abstract String generateDocumentByClassName(String className) throws Exception;
+ protected Predicate<Field> isRequired = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.required();
+ };
+
+ protected Predicate<Field> isOptional = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return !fieldContext.deprecated() && !fieldContext.required();
+ };
+
+ protected Predicate<Field> isDeprecated = field -> {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ return fieldContext.deprecated();
+ };
+
+ protected void writeDocListByFieldContext(List<Field> fieldList, StringBuilder sb, Object obj) throws Exception {
+ for (Field field : fieldList) {
+ FieldContext fieldContext = field.getAnnotation(FieldContext.class);
+ field.setAccessible(true);
+
+ sb.append("### ").append(field.getName()).append("\n");
+ sb.append(fieldContext.doc().replace(">", "\\>")).append("\n\n");
+ sb.append("**Default**: `").append(field.get(obj)).append("`\n\n");
+ sb.append("**Dynamic**: `").append(fieldContext.dynamic()).append("`\n\n");
+ sb.append("**Category**: ").append(fieldContext.category()).append("\n\n");
+ }
+ }
+
+ protected static class CategoryComparator implements Comparator<Field> {
+ @Override
+ public int compare(Field o1, Field o2) {
+ FieldContext o1Context = o1.getAnnotation(FieldContext.class);
+ FieldContext o2Context = o2.getAnnotation(FieldContext.class);
+
+ if (o1Context.category().equals(o2Context.category())) {
+ return o1.getName().compareTo(o2.getName());
+ }
+ return o1Context.category().compareTo(o2Context.category());
+ }
+ }
+
+ protected String prefix = """
+ :::note
+
+ This page is automatically generated from code files.
+ If you find something inaccurate, feel free to update `""";
+ protected String suffix = """
+ `. Do NOT edit this markdown file manually. Manual changes will be overwritten by automatic generation.
+
+ :::
+ """;
+
protected String generateDocByFieldContext(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.removeIf(f -> f.getAnnotation(FieldContext.class) == null);
+ fieldList.sort(new CategoryComparator());
+ List<Field> requiredFields = fieldList.stream().filter(isRequired).toList();
+ List<Field> optionalFields = fieldList.stream().filter(isOptional).toList();
+ List<Field> deprecatedFields = fieldList.stream().filter(isDeprecated).toList();
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|Dynamic|Category|\n");
- sb.append("|---|---|---|---|---|\n");
- for (Field field : fields) {
- FieldContext fieldContext = field.getAnnotation(FieldContext.class);
- if (fieldContext == null) {
- continue;
- }
- field.setAccessible(true);
- sb.append("| ").append(field.getName()).append(" | ");
- sb.append(fieldContext.doc().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append(fieldContext.dynamic()).append(" | ");
- sb.append(fieldContext.category()).append(" | ");
- sb.append("\n");
- }
+
+ sb.append(prefix).append(className).append(suffix);
+ sb.append("## Required\n");
+ writeDocListByFieldContext(requiredFields, sb, obj);
+ sb.append("## Optional\n");
+ writeDocListByFieldContext(optionalFields, sb, obj);
+ sb.append("## Deprecated\n");
+ writeDocListByFieldContext(deprecatedFields, sb, obj);
+
return sb.toString();
}
protected String generateDocByApiModelProperty(String className, String type, StringBuilder sb) throws Exception {
Class<?> clazz = Class.forName(className);
Object obj = clazz.getDeclaredConstructor().newInstance();
Field[] fields = clazz.getDeclaredFields();
+ ArrayList<Field> fieldList = new ArrayList<>(Arrays.asList(fields));
+
+ fieldList.sort(Comparator.comparing(Field::getName));
sb.append("# ").append(type).append("\n");
- sb.append("|Name|Description|Default|\n");
- sb.append("|---|---|---|\n");
- for (Field field : fields) {
+ sb.append(prefix).append(className).append(suffix);
+ sb.append("## Optional\n");
+ for (Field field : fieldList) {
ApiModelProperty fieldContext = field.getAnnotation(ApiModelProperty.class);
if (fieldContext == null) {
continue;
}
field.setAccessible(true);
+
String name = StringUtils.isBlank(fieldContext.name()) ? field.getName() : fieldContext.name();
- sb.append("| ").append(name).append(" | ");
- sb.append(fieldContext.value().replace("\n", "<br>")).append(" | ");
- sb.append(field.get(obj)).append(" | ");
- sb.append("\n");
Review Comment:
@urfreespace No, I've checked all of the above and they won't affect other docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] SignorMercurio commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
SignorMercurio commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925851223
##########
site2/docs/reference-configuration-pulsar-proxy.md:
##########
@@ -0,0 +1,934 @@
+# Pulsar proxy
+## Required
Review Comment:
What about just removing `## Required` and show only `## Optional` and the rest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#discussion_r925647256
##########
site2/docs/reference-configuration-client.md:
##########
@@ -0,0 +1,222 @@
+# Client
Review Comment:
Hi @gaoran10 are the configurations of `client` all `optional`? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #16627: [feature][doc] Generate config docs from source code
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #16627:
URL: https://github.com/apache/pulsar/pull/16627#issuecomment-1197585216
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org