You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/08/27 19:46:04 UTC

[GitHub] [pulsar] lhotari opened a new pull request #11822: [IO] Set the output message key when the source record has a key

lhotari opened a new pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822


   ### Motivation
   
   I'm investigating an issue where setting batch-builder for a Debezium Postgres source doesn't seem to have an impact even after applying fixes made by #11706 for supporting `--batch-builder KEY_BASED` for sources.
   When reading the code, I found a special condition for ignoring the source record key.
   https://github.com/apache/pulsar/blob/3b651d910f39a29859434f0c5ba962c3ef8d2504/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L370-L373
   
   ### Modifications
   
   Remove the extra condition. Always set the output message key when the source record has a key.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari closed pull request #11822: [IO] Set the output message key when the source record has a key

Posted by GitBox <gi...@apache.org>.
lhotari closed pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #11822: [IO] Set the output message key when the source record has a key

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822#issuecomment-907471326


   Also found this kind of logic:
   https://github.com/apache/pulsar/blob/0f8aef2b9494cde8e8adc9d97d89a65d73ae8c35/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TypedMessageBuilderImpl.java#L126-L134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #11822: [IO] Set the output message key when the source record has a key

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822#issuecomment-907451096


   If consumed storage space of the key in the output message is the concern, the key should be calculated as a hash from the key.
   I guess the key size might become an issue in certain cases?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #11822: [IO] Set the output message key when the source record has a key

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822#issuecomment-907472965


   Oh now I understand how this works:
   https://github.com/apache/pulsar/blob/0f8aef2b9494cde8e8adc9d97d89a65d73ae8c35/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TypedMessageBuilderImpl.java#L152-L174
   
   closing this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] lhotari commented on pull request #11822: [IO] Set the output message key when the source record has a key

Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #11822:
URL: https://github.com/apache/pulsar/pull/11822#issuecomment-907436981


   This special logic was added by #6034 . @tuteng do we need this special condition? I'm proposing to remove it with this PR. Are you fine with that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org