[jira] [Comment Edited] (KAFKA-16858) Flatten SMT throws NPE

["Greg Harris (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850173#comment-17850173 ] 

Greg Harris edited comment on KAFKA-16858 at 5/28/24 8:38 PM:
--------------------------------------------------------------

It really confused me that the stacktrace is in reverse order, is there a tool that does that by default?

I confirmed that ConnectSchema.validateValue() throws NPE when given a null schema, this is replicated with this assertion:

 
{noformat}
assertThrows(NullPointerException.class, () -> ConnectSchema.validateValue(null, new Object()));
{noformat}
I can also confirm that this NPE appears when using SchemaBuilder.arrayOfNull(), which would cause `valueSchema()` to return null. The exception only appears when the array is non-empty.

 

 
{noformat}
final String fieldName = "fieldName";
Schema testSchema = SchemaBuilder.struct()
        .field(fieldName, SchemaBuilder.arrayOfNull());
new Struct(testSchema).put(fieldName, Collections.emptyList());
assertThrows(NullPointerException.class, () -> 
        new Struct(testSchema).put(fieldName, Collections.singletonList(new Object())));
{noformat}
Looking at the way Flatten works, it's just copying schemas from input to output, so an arrayOfNull() can appear if the input contained an arrayOfNull(): [https://github.com/apache/kafka/blob/4d04eb83ea2cf67484d81e30603e976f63616428/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/SchemaUtil.java#L29] 

Since this is a type that SchemaBuilder can emit, we should probably handle it intentionally, either emitting a DataException, or passing the validation.

[~undone] For my own reference: Do you see this fault with another SMT in the chain ahead of Flatten? What Converter (sink connector) or source connector implementation are you using? There's also a very noisy log statement you could use to figure out where this null-array-schema is appearing: [https://github.com/apache/kafka/blob/4d04eb83ea2cf67484d81e30603e976f63616428/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/TransformationChain.java#L53] 

 


was (Author: gharris1727):
It really confused me that the stacktrace is in reverse order, is there a tool that does that by default?

I confirmed that ConnectSchema.validateValue() throws NPE when given a null schema, this is replicated with this assertion:

 
{noformat}
assertThrows(NullPointerException.class, () -> ConnectSchema.validateValue(null, new Object()));
{noformat}
I can also confirm that this NPE appears when using SchemaBuilder.arrayOfNull(), which would cause `valueSchema()` to return null. The exception only appears when the array is non-empty.

 

 
{noformat}
final String fieldName = "fieldName";
Schema testSchema = SchemaBuilder.struct()
        .field(fieldName, SchemaBuilder.arrayOfNull());
new Struct(testSchema).put(fieldName, Collections.emptyList());
assertThrows(NullPointerException.class, () -> 
        new Struct(testSchema).put(fieldName, Collections.singletonList(new Object())));
{noformat}
Looking at the way Flatten works, it's just copying schemas from input to output, so an arrayOfNull() can appear if the input contained an arrayOfNull(): [https://github.com/apache/kafka/blob/4d04eb83ea2cf67484d81e30603e976f63616428/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/util/SchemaUtil.java#L29] 

Since this is a type that SchemaBuilder can emit, we should probably handle it intentionally, either emitting a DataException, or passing the validation.

[~undone] For my own reference: Do you see this fault with another SMT in the chain ahead of Flatten? What Converter implementation are you using? There's also a very noisy log statement you could use to figure out where this null-array-schema is appearing: [https://github.com/apache/kafka/blob/4d04eb83ea2cf67484d81e30603e976f63616428/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/TransformationChain.java#L53] 

 

> Flatten SMT throws NPE
> ----------------------
>
>                 Key: KAFKA-16858
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16858
>             Project: Kafka
>          Issue Type: Bug
>          Components: connect
>    Affects Versions: 3.6.0
>         Environment: Kafka 3.6 by way of CP 7.6.0
>            Reporter: Adam Strickland
>            Priority: Major
>         Attachments: FlattenTest.java
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a call to an SMT chain.  Stack trace snippet:
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error occurs regardless).
> The field being transformed is an array of structs. If the call to {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns {{{}null{}}}, the subsequent call to {{Schema#name()}} at o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen; *sometimes* the struct's schema is found and sometimes it is not. We have been unable to determine the root cause, but have constructed a test that replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)