You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/03/23 00:07:05 UTC
[GitHub] [kafka] dosvath opened a new pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
dosvath opened a new pull request #10375:
URL: https://github.com/apache/kafka/pull/10375
**Problem**
The [current Cast SMT](https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/Cast.java) fails on a null record value (or a null record key), which is problematic for tombstone records. When a tombstone record reaches the transformation the error below is thrown:
With schema:
```
Cannot list fields on non-struct type
org.apache.kafka.connect.errors.DataException: Cannot list fields on non-struct type
at org.apache.kafka.connect.data.ConnectSchema.fields(ConnectSchema.java:179)
at org.apache.kafka.connect.transforms.Cast.getOrBuildSchema(Cast.java:190)
```
For schemaless:
```
Caused by: org.apache.kafka.connect.errors.DataException: Only Map objects supported in absence of schema for [cast types], found: null
at org.apache.kafka.connect.transforms.util.Requirements.requireMap(Requirements.java:38)
```
**Solution**
Null value records should instead be allowed to pass through as there is no cast transformation to be done, with the benefit of allowing the connector to handle the tombstone records as intended.
**Testing**
Added SMT unit tests to verify the records pass through when the keys or values are null.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dosvath commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dosvath commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-833132393
@ewencp @mjsax one more ping on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dongjinleekr commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dongjinleekr commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-804617742
LGTM.
@vvcephei @ewencp @ijuma Could you have a look? IMHO this PR can be included in 2.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison merged pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
mimaison merged pull request #10375:
URL: https://github.com/apache/kafka/pull/10375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dosvath commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dosvath commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-806084689
Thanks @kkonstantine I updated the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dosvath commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dosvath commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-804951684
> LGTM.
>
> @vvcephei @ewencp @ijuma Could you have a look? IMHO this PR can be included in 2.8.
Thanks @dongjinleekr I will update the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kkonstantine commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
kkonstantine commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-804643837
To be more equipped to say whether this issue can be considered a release blocker or not (which is something that should be called out on the dev mailing list for any releases that are in progress) it would be good to know whether this is a regression or bug that has escaped several releases. @dosvath do you happen to know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dosvath commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dosvath commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-819924623
@kkonstantine @dongjinleekr how should we proceed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-841010559
Not familiar with Connect code. I guess @rhauch @kkonstantine @mimaison @C0urante @wicknicks should be able to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dosvath commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
dosvath commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-804950443
> To be more equipped to say whether this issue can be considered a release blocker or not (which is something that should be called out on the dev mailing list for any releases that are in progress) it would be good to know whether this is a regression or bug that has escaped several releases. @dosvath do you happen to know?
It seems it's a bug that has escaped several releases I didn't see any point in the history where null is handled to make this a regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] kkonstantine commented on pull request #10375: KAFKA-12522: Cast SMT should allow null value records to pass through
Posted by GitBox <gi...@apache.org>.
kkonstantine commented on pull request #10375:
URL: https://github.com/apache/kafka/pull/10375#issuecomment-805025698
Thanks for checking @dosvath.
That means that this fix probably won't cut it for a release blocker at this late stage in the release process of 2.8 (or other branches that are in the process of generating release candidates). We'll either merge to trunk and wait before we backport, or wait a bit more altogether.
Finally, in this project we always target `trunk` on our PRs and then we cherry-pick (unless we explicitly need to backport only to a release branch). So your target was correctly pointing to `trunk`. I think @dongjinleekr was suggesting that we should cherry-pick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org