You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Jason Altekruse (JIRA)" <ji...@apache.org> on 2015/04/08 19:43:12 UTC
[jira] [Commented] (DRILL-1460) JsonReader fails reading files with
decimal numbers and integers in the same field
[ https://issues.apache.org/jira/browse/DRILL-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485619#comment-14485619 ]
Jason Altekruse commented on DRILL-1460:
----------------------------------------
[~dragoncurve] I'm sorry I haven't gotten back to this for so long. I actually put up a discussion on the mailing list a few months ago to just change the default to reading all values as double by default, no configuration needed. These datasets are far too common to have anything else as a default. We do appreciate the patch, but I really think there is very little value in the current behavior, that is until Drill supports the concept of changing type (mentioned in the discussion thread linked below). I am going to make a simpler patch that just changes the reader to always read the values as doubles and post it here for review (this will simplify the JSON reader as well as avoid further cluttering the config options). For anyone with requirements to read the numbers as integers or bigints for precision the option to use all_text_mode with casts will still be there.
Again thanks for the work, this is something we have been needing to solve for a while. Please do check back to review the new patch.
http://mail-archives.apache.org/mod_mbox/drill-dev/201501.mbox/%3CCAMpYv7Bd4sScGdefFsGyZq8bU%3DQH4Tg%2BvYMc8m-BUOz09pp2Dg%40mail.gmail.com%3E
> JsonReader fails reading files with decimal numbers and integers in the same field
> ----------------------------------------------------------------------------------
>
> Key: DRILL-1460
> URL: https://issues.apache.org/jira/browse/DRILL-1460
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Bhallamudi Venkata Siva Kamesh
> Assignee: Jason Altekruse
> Fix For: 0.9.0
>
> Attachments: DRILL-1460.1.patch.txt, DRILL-1460.2.patch.txt
>
>
> Used the following dataset : http://thecodebarbarian.wordpress.com/2014/03/28/plugging-usda-nutrition-data-into-mongodb
> Executed the following query
> {noformat}select t.nutrients from dfs.usda.`usda.json` t limit 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:48:39,421 [b9dfbb9b-29a9-425d-801c-2e418533525f:frag:0:0] ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error 0568d90a-d7df-4a5d-87e9-8b9f718dffa4: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl.
> at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
> {noformat}select t.nutrients[0].units from dfs.usda.`usda.json` t limit 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:50:04,394 [9ee8a529-17fd-492f-9cba-2d1f5842eae1:frag:0:0] ERROR o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error c4c6bffd-b62b-4878-af1e-58db64453307: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you are using a ValueWriter of type NullableFloat8WriterImpl.
> at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:158) ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)