You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/05/18 10:26:37 UTC
[GitHub] [druid] josephglanville opened a new pull request #9887: Re-order and document format detection in web console
josephglanville opened a new pull request #9887:
URL: https://github.com/apache/druid/pull/9887
Motivation for this change is to not inadvertently identify binary
formats that contain uncompressed string data as TSV or CSV.
Moving detection of magic byte headers before heuristics should be more
robust in general.
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
maytasm commented on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-633432188
@vogievetsky @josephglanville
I also observed similar issues in the following cases when ingesting from local files:
- Ingesting from directory containing Orc files with file filter set to * auto-detect to csv
- Ingesting from directory containing Parquet files with file filter set to * auto-detect to csv
- Ingesting from directory containing Parquet files with file filter set to *.parquet auto-detect to csv
However, ingesting from directory containing Orc files with file filter set to *.orc correctly auto-detect to orc
I think your PR may fix all of the above issues. I will try checkout the PR and confirm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm edited a comment on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
maytasm edited a comment on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-633432188
@vogievetsky @josephglanville
I also observed similar issues in the following cases when ingesting from local files:
- Ingesting from directory containing Orc files with file filter set to * auto-detect to csv
- Ingesting from directory containing Parquet files with file filter set to * auto-detect to csv
- Ingesting from directory containing Parquet files with file filter set to *.parquet auto-detect to csv
However, ingesting from directory containing Orc files with file filter set to *.orc correctly auto-detect to orc
I think your PR may fix all of the above issues. I will try checkout the PR and confirm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] josephglanville edited a comment on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
josephglanville edited a comment on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-633377531
@vogievetsky sorry for slow reply. Yes I opened an Avro binary file with the compression codec set to `none` that contained JSON data, it was detected as CSV because of the commas. (the reason why it didn't trip the JSON heuristic is that requires the first and last bytes to be ASCII curly braces)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-632308074
Love it. Makes total sense. Out of curiosity was this change motivated by something you saw IRL that tripped up the current system?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-632396654
Oh I guess you saw a binary file that happened to have more than 3 tabs or commas...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky merged pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
vogievetsky merged pull request #9887:
URL: https://github.com/apache/druid/pull/9887
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] josephglanville commented on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
josephglanville commented on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-633377531
@vogievetsky sorry for slow reply. Yes I opened an Avro binary file with the compression codec set to `none` that contained JSON data, it was detected as CSV because of the commas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] josephglanville commented on pull request #9887: Re-order and document format detection in web console
Posted by GitBox <gi...@apache.org>.
josephglanville commented on pull request #9887:
URL: https://github.com/apache/druid/pull/9887#issuecomment-633607219
Yes this will definitely fix those issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org