You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Jacques Nadeau (JIRA)" <ji...@apache.org> on 2015/01/04 22:29:36 UTC
[jira] [Updated] (DRILL-947) Add support for mutating JSON files
[ https://issues.apache.org/jira/browse/DRILL-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Nadeau updated DRILL-947:
---------------------------------
Summary: Add support for mutating JSON files (was: Select query with limit failed for json files with mutant)
> Add support for mutating JSON files
> -----------------------------------
>
> Key: DRILL-947
> URL: https://issues.apache.org/jira/browse/DRILL-947
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Xiao Meng
> Fix For: Future
>
> Attachments: drillbit.log, metadata-evolution-test5.json, tableau_tsv_error.PNG
>
>
> Given a json file with mutant like the one in DRILL-835, where the majority has the schema
> {code}
> (name, isDirectory, isFile, isSymlink, length, ownerName, groupName, modificationTime, accessTime, permission)
> {code}
> and there exists a record with different schema
> {code}
> (name, mutant, age, firstName, powers)
> {code}
> the following query generated by Tableau fails:
> {code}
> SELECT 1 AS `Number_of_Records`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`accessTime` AS `accessTime`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`groupName` AS `groupName`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`isDirectory` AS `isDirectory`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`isFile` AS `isFile`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`isSymlink` AS `isSymlink`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`length` AS `length`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`modificationTime` AS `modificationTime`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`name` AS `name`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`ownerName` AS `ownerName`,
> `/opt/drill/raw-files/metadata-evolution-test5.json`.`permission` AS `permission`
> FROM `dfs.default`.`/opt/drill/raw-files/metadata-evolution-test5.json` `/opt/drill/raw-files/metadata-evolution-test5.json`
> LIMIT 10000
> {code}
> The stacktrace is
> {code}
> 2014-06-10 16:20:37,771 [cd37bd9c-bc0b-4e2a-972d-36ec548661f8:frag:0:0] DEBUG o.a.d.e.w.fragment.FragmentExecutor - Caught exception while running fragment
> java.lang.UnsupportedOperationException: Attemped to access a DeadBuf. This would happen if you attempted to interact with a buffer that has been moved or not yet initialized.
> at org.apache.drill.exec.record.DeadBuf.retain(DeadBuf.java:792) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.IntVector.transferTo(IntVector.java:195) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.vector.IntVector$TransferImpl.transfer(IntVector.java:226) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:110) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:71) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.next(LimitRecordBatch.java:99) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.next(ScreenCreator.java:81) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)