You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/07 18:32:12 UTC
[GitHub] [pulsar] 315157973 opened a new pull request #7193: function to read compacted topics
315157973 opened a new pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193
Fixes #5538
### Motivation
### Modifications
In function mode and sink mode, PulsarSource can read compacted topic。
By `inputSpecs` parameter, each topic can independently decide whether to read compacted
### Verifying this change
unit test:
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction
org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#discussion_r436447865
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdSinks.java
##########
@@ -276,6 +277,9 @@ protected void validateSinkConfigs(SinkConfig sinkConfig) {
@Parameter(names = "--custom-schema-inputs", description = "The map of input topics to Schema types or class names (as a JSON string)")
protected String customSchemaInputString;
+ @Parameter(names = "--input-specs", description = "The map of inputs to custom configuration (as a JSON string)")
Review comment:
Is this related to this PR?
##########
File path: pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
##########
@@ -233,6 +234,8 @@ void processArguments() throws Exception {
protected String customSerdeInputString;
@Parameter(names = "--custom-schema-inputs", description = "The map of input topics to Schema class names (as a JSON string)")
protected String customSchemaInputString;
+ @Parameter(names = "--input-specs", description = "The map of inputs to custom configuration (as a JSON string)")
Review comment:
Is this related to this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#discussion_r437614293
##########
File path: pulsar-functions/proto/src/main/proto/Function.proto
##########
@@ -94,6 +94,7 @@ message ConsumerSpec {
int32 value = 1;
}
ReceiverQueueSize receiverQueueSize = 4;
+ bool readCompacted = 5;
Review comment:
We already have a ConsumerConfig class:
https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/functions/ConsumerConfig.java
Though I am for refactoring this to be generic Map so we don't have to make code changes for new consumer configs that are added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
jiazhai commented on pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#issuecomment-648527253
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jiazhai commented on pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
jiazhai commented on pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#issuecomment-648527077
@jerrypeng @sijie Would you please help review this PR again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#discussion_r438584451
##########
File path: pulsar-functions/proto/src/main/proto/Function.proto
##########
@@ -94,6 +94,7 @@ message ConsumerSpec {
int32 value = 1;
}
ReceiverQueueSize receiverQueueSize = 4;
+ bool readCompacted = 5;
Review comment:
> We already have a ConsumerConfig class:
>
> https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/functions/ConsumerConfig.java
>
> Though I am for refactoring this to be generic Map so we don't have to make code changes for new consumer configs that are added.
Thanks very much. But schemaProperties are properties belonging to the schema. Put the Consumer parameters with them, I don’t think it’s a good way. I think it is necessary to add a new Map to save the general parameters of Consumer.And this make it easier to convert Map to object
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on a change in pull request #7193: function to read compacted topics
Posted by GitBox <gi...@apache.org>.
sijie commented on a change in pull request #7193:
URL: https://github.com/apache/pulsar/pull/7193#discussion_r436539808
##########
File path: pulsar-functions/proto/src/main/proto/Function.proto
##########
@@ -94,6 +94,7 @@ message ConsumerSpec {
int32 value = 1;
}
ReceiverQueueSize receiverQueueSize = 4;
+ bool readCompacted = 5;
Review comment:
I think a more generic approach is to add a `Map<String, String>` to the consumer spec. So we can use https://github.com/apache/pulsar/blob/870a637b4906862a611e418341dd926e21458f08/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBuilderImpl.java#L86 to load the consumer configuration when building the consumers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org