You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2023/01/17 21:07:35 UTC
[GitHub] [druid] davidgmonical opened a new issue, #13685: JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays
davidgmonical opened a new issue, #13685:
URL: https://github.com/apache/druid/issues/13685
Currently when ingesting json data with multivalue fields, druid parses the input source with logic that includes removing `null` values from the individual arrays. This happens as part of the json flattening [here](https://github.com/apache/druid/blob/22630b0aabbaf683748b65a698ccbf966e1a54a4/core/src/main/java/org/apache/druid/data/input/impl/JsonLineReader.java#L70). An example, supposed we try to ingest:
```
{"list":[0.5, 0.2, null],"timestamp":"2023-01-11T22:57:44.270035398Z"}
```
We end up with a table that looks like (note the lack of `null` in the multivalue field):
<img width="979" alt="Screen Shot 2023-01-17 at 12 06 32 PM" src="https://user-images.githubusercontent.com/22461273/213000830-20ccc51d-c663-4c19-8f1b-5f1dd81bef34.png">
The `null` removal occurs [here](https://github.com/apache/druid/blob/22630b0aabbaf683748b65a698ccbf966e1a54a4/core/src/main/java/org/apache/druid/java/util/common/parsers/JSONFlattenerMaker.java#L177-L179) when we are parsing the individual entries in the multivalue json fields. It's unclear from any documentation that I could find that this is intended behavior. It seems to me like if druid respects `null` values for single-value fields (with regards to `druid.generic.useDefaultValueForNull`), then it should also respect `null` entries in multivalue fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #13685: JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #13685:
URL: https://github.com/apache/druid/issues/13685#issuecomment-1386103922
I've also noticed this recently when working in this area; its not just limited to JSON, but also at least Avro also has this behavior. I'm unsure about the other nested formats (ORC, Parquet, Protobuf), an initial look seems like they might be ok.
I was planning to try to fix this in the near future since I agree it doesn't really make sense for the values to be removed here, or am happy to review if you've already got a fix underway.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] somu-imply commented on issue #13685: JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays
Posted by "somu-imply (via GitHub)" <gi...@apache.org>.
somu-imply commented on issue #13685:
URL: https://github.com/apache/druid/issues/13685#issuecomment-1405761882
The same issue happens with Avro too. I have set up #13712 to tackle both
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13685: JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays
URL: https://github.com/apache/druid/issues/13685
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13685:
URL: https://github.com/apache/druid/issues/13685#issuecomment-1922536653
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13685:
URL: https://github.com/apache/druid/issues/13685#issuecomment-1972195751
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] davidgmonical commented on issue #13685: JSON flattener should respect druid null handling config (druid.generic.useDefaultValueForNull) for arrays
Posted by GitBox <gi...@apache.org>.
davidgmonical commented on issue #13685:
URL: https://github.com/apache/druid/issues/13685#issuecomment-1386316165
Nothing underway but happy to take a stab at it. Would love a review once I have it up thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org