You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/04 12:25:43 UTC

[GitHub] [pulsar] nicoloboschi opened a new pull request, #15426: [feat][elasticsearch-sink] Option to output canonical key fields (JSON and Avro)

nicoloboschi opened a new pull request, #15426:
URL: https://github.com/apache/pulsar/pull/15426

   ### Motivation
   In case of the inbound message is structured (Avro or JSON) the fields order may change overtime and there's no fields order guarantee from the Pulsar function framework.
   The generated _id field of the document is supposed to be the same regardless the input message key fields order.    
   
   ### Modifications
   * New option `canonicalKeyFields` (boolean, default false) to sort the key fields. Both for JSON and Avro we have to parse and rewrite the entire payload. It may increase the CPU overhead even if the sort is only performed on the keys that MUST be lower to 512 bytes in order to suit in the ElasticSearch _id field.
   
   - [x] `doc` 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nicoloboschi merged pull request #15426: [feature][connector] ElasticSearch Sink: option to output canonical key fields (JSON and Avro)

Posted by GitBox <gi...@apache.org>.
nicoloboschi merged PR #15426:
URL: https://github.com/apache/pulsar/pull/15426


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on pull request #15426: [feat][elasticsearch-sink] Option to output canonical key fields (JSON and Avro)

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #15426:
URL: https://github.com/apache/pulsar/pull/15426#issuecomment-1122175568

   can you rename your PR? eg. [feature][doc] xx
   
   guideline: https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.wu6ygjne8e35 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] nicoloboschi commented on pull request #15426: [feature][connector] ElasticSearch Sink: option to output canonical key fields (JSON and Avro)

Posted by GitBox <gi...@apache.org>.
nicoloboschi commented on PR #15426:
URL: https://github.com/apache/pulsar/pull/15426#issuecomment-1128648645

   @lhotari @dlg99 @eolivelli PTAL 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #15426: [feat][elasticsearch-sink] Option to output canonical key fields (JSON and Avro)

Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #15426:
URL: https://github.com/apache/pulsar/pull/15426#discussion_r865891219


##########
site2/docs/io-elasticsearch-sink.md:
##########
@@ -80,6 +80,7 @@ The configuration of the Elasticsearch sink connector has the following properti
 | `compatibilityMode` | enum (AUTO,ELASTICSEARCH,ELASTICSEARCH_7,OPENSEARCH) | AUTO |  | Specify compatibility mode with the ElasticSearch cluster. `AUTO` value will try to auto detect the correct compatibility mode to use. Use `ELASTICSEARCH_7` if the target cluster is running ElasticSearch 7 or prior. Use `ELASTICSEARCH` if the target cluster is running ElasticSearch 8 or higher. Use `OPENSEARCH` if the target cluster is running OpenSearch. |
 | `token` | String| false | " " (empty string)|The token used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured. |
 | `apiKey` | String| false | " " (empty string)|The apiKey used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured. |
+| `canonicalKeyFields` | Boolean | false | false | If canonicalKeyFields is true and record key schema is JSON or AVRO, the serialized object will not consider the properties order. |

Review Comment:
   @momo-jun could you please help review this PR from the technical writing pespective after the technical review? Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #15426: [feat][elasticsearch-sink] Option to output canonical key fields (JSON and Avro)

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #15426:
URL: https://github.com/apache/pulsar/pull/15426#discussion_r868690540


##########
site2/docs/io-elasticsearch-sink.md:
##########
@@ -80,6 +80,7 @@ The configuration of the Elasticsearch sink connector has the following properti
 | `compatibilityMode` | enum (AUTO,ELASTICSEARCH,ELASTICSEARCH_7,OPENSEARCH) | AUTO |  | Specify compatibility mode with the ElasticSearch cluster. `AUTO` value will try to auto detect the correct compatibility mode to use. Use `ELASTICSEARCH_7` if the target cluster is running ElasticSearch 7 or prior. Use `ELASTICSEARCH` if the target cluster is running ElasticSearch 8 or higher. Use `OPENSEARCH` if the target cluster is running OpenSearch. |
 | `token` | String| false | " " (empty string)|The token used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured. |
 | `apiKey` | String| false | " " (empty string)|The apiKey used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured. |
+| `canonicalKeyFields` | Boolean | false | false | If canonicalKeyFields is true and record key schema is JSON or AVRO, the serialized object will not consider the properties order. |

Review Comment:
   ```suggestion
   | `canonicalKeyFields` | Boolean | false | false | Whether to sort the key fields for JSON and Avro or not. If it is set to `true` and the record key schema is `JSON` or `AVRO`, the serialized object does not consider the order of properties. |
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org