You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rafferty97 (via GitHub)" <gi...@apache.org> on 2025/08/18 01:26:15 UTC
[I] Support top-level arrays when reading JSON files [datafusion]
Rafferty97 opened a new issue, #17223:
URL: https://github.com/apache/datafusion/issues/17223
### Is your feature request related to a problem or challenge?
Datafusion can't read JSONL/ND-JSON files where the rows are arrays rather than objects. e.g.:
```
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]
```
### Describe the solution you'd like
It would be great if Datafusion could directly consume ND-JSON files containing top-level arrays, mapping the array to a single field with a customisable name, like data or items.
I notice that the underlying arrow library seems to support this concept via `datafusion::arrow::json::ReaderBuilder::new_with_field`, but this doesn't seem to be exposed by the JSON `FileFormat` that Datafusion exposes.
### Describe alternatives you've considered
At present I am working around the issue with a pre-processing step that wraps each line with a { "data": ... } wrapper, however this comes with a heavy performance penalty.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support top-level arrays when reading JSON files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17223:
URL: https://github.com/apache/datafusion/issues/17223#issuecomment-3205666515
Thanks @Rafferty97 -- is there any chance you can provide an example dataset / reproducer? I suspect if you did there is a much greater chance of one of the contributors working on this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support top-level arrays when reading JSON files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17223:
URL: https://github.com/apache/datafusion/issues/17223#issuecomment-3211453087
> I'd also be happy to work on this issue myself with a little guidance from the team.
Thanks!
I think this is the code that does the schema inference. The first step would be to figure out if it is doing the right thing
https://github.com/apache/datafusion/blob/2741c60ad9d637e5014d201802abb25562449491/datafusion/datasource-json/src/file_format.rs#L198-L197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support top-level arrays when reading JSON files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #17223:
URL: https://github.com/apache/datafusion/issues/17223#issuecomment-3209112657
@alamb Certainly. What would be the best way to do so? Just an attachment to a github comment on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support top-level arrays when reading JSON files [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17223:
URL: https://github.com/apache/datafusion/issues/17223#issuecomment-3211446064
> [@alamb](https://github.com/alamb) Certainly. What would be the best way to do so? Just an attachment to a github comment on this issue?
Yes, that is what I would suggest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support top-level arrays when reading JSON files [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on issue #17223:
URL: https://github.com/apache/datafusion/issues/17223#issuecomment-3209114350
I'd also be happy to work on this issue myself with a little guidance from the team.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org