You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/07 16:57:31 UTC
[GitHub] [pulsar] eolivelli opened a new pull request #8854: PulsarSource does not report original Schema information
eolivelli opened a new pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854
- extract original Schema from the message read from the topic
- expose the schema to the PulsarRecord implement
- add unit test
### Motivation
Pulsar sinks cannot access the original Schema of the Message they are consuming,
this is because currently PulsarSource does not publish the Schema information to downstream processing.
### Modifications
Try to retrieve Schema information from the Message and push it down to the Record.
### Verifying this change
- [X] Make sure that the change passes the CI checks.
- [X[ This change added tests and can be verified as follows:
- added unit test
- tested manually with Pulsar Standalone
### Does this pull request potentially affect one of the following parts: yes
with this change Pulsar function users and especially Pulsar I/O Sinks will be able to inspect the original schema in case the processing starts from a Pulsar topic (PulsarSource)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741701477
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741702429
many jobs failed with
`/__w/_temp/4f45ee2d-1924-45c7-b32d-fd82169e5f0c.sh: 1: /__w/_temp/4f45ee2d-1924-45c7-b32d-fd82169e5f0c.sh: ./build/run_unit_group.sh: not found`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-742001870
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741871052
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741726986
@lhotari @codelipenghui @sijie CI errors seems not related to my patch,
is there something I can do ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: Pulsar IO: Make Source topic Schema information available to downstream Sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-742800764
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740082330
If it is possible to see this also on 2.6.3 it would be greater
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741625949
The problem in the test is that PulsarSink is using the record schema to write to the destination topic, but this is wrong because it must use the schema of the destination topic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741729265
I have rebased onto current master
the commit that changed the CI behavior is
https://github.com/apache/pulsar/commit/a8a79939960fde2ebd4156a811d282adcf898749
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #8854: Pulsar IO: Make Source topic Schema information available to downstream Sinks
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740372251
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740056148
@315157973 @nlu90 @jerrypeng
could you please take a look ?
This can be considered a small breaking behavior change, but I would like to see it cherry picked to 2.7.x branch if possible
cc @sijie @rdhabalia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740602155
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
sijie commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740372206
@codelipenghui let's cherry-pick this change to 2.6.3 release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: Pulsar IO: Make Source topic Schema information available to downstream Sinks
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-742499255
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740707443
There is a test failure related to schemas and Pulsar functions.
I am investigating it now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-741742805
btw I thought that the PR validation is run against the PR branch merged with the target branch, so rebasing onto latest master should not have much effect
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #8854: PulsarSource does not report original Schema information
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #8854:
URL: https://github.com/apache/pulsar/pull/8854#issuecomment-740561925
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org