You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2018/01/05 22:23:00 UTC
[jira] [Resolved] (IMPALA-6370) Crash when querying nested data in
partitioned Parquet table
[ https://issues.apache.org/jira/browse/IMPALA-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-6370.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
IMPALA-6370: fix partitioned parquet tables with nested types
When materialising a nested collection, has_template_tuple() should use
the template tuple for the collection, not the top-level tuple.
Testing:
Added tests based on nested-types-basic.test that operate on a simple
partitioned table. The tests reliably crashed Impala before the fix.
Change-Id: Ic808b824ce3b31af0539036d8ca23d17b18deab4
Reviewed-on: http://gerrit.cloudera.org:8080/8947
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
> Crash when querying nested data in partitioned Parquet table
> -------------------------------------------------------------
>
> Key: IMPALA-6370
> URL: https://issues.apache.org/jira/browse/IMPALA-6370
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: crash
> Fix For: Impala 2.12.0
>
>
> Header of hs_err file:
> {noformat}
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007f908452d782, pid=1223, tid=0x00007f8fc0d1e700
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [libc.so.6+0x89782] memcpy+0xd2
> #
> # Core dump written. Default location: /data/1/cores/core or core.1223
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> --------------- T H R E A D ---------------
> Current thread is native thread
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000
> Registers:
> RAX=0x00000000276a9000, RBX=0x000000001ae72f00, RCX=0x0000000000000019, RDX=0x0000000000000341
> RSP=0x00007f8fc0d1d058, RBP=0x00007f8fc0d1d150, RSI=0x0000000000000000, RDI=0x00000000276a9000
> R8 =0x00007f8fc0d1d108, R9 =0x00007f8fc0d1d110, R10=0x0000000000000000, R11=0x00007f9084527e5d
> R12=0x000000001b06e418, R13=0x000000002485d2e8, R14=0x0000000000000001, R15=0x000000002485d2a0
> RIP=0x00007f908452d782, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
> TRAPNO=0x000000000000000e
> {noformat}
> Backtrace from core:
> {noformat}
> (gdb) bt
> #0 0x00007f90844d6495 in raise () from sysroot/lib64/libc.so.6
> #1 0x00007f90844d7c75 in abort () from sysroot/lib64/libc.so.6
> #2 0x00007f90867419d5 in os::abort(bool) ()
> from sysroot/usr/java/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so
> #3 0x00007f90868e4033 in VMError::report_and_die() ()
> from sysroot/usr/java/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so
> #4 0x00007f90867471ff in JVM_handle_linux_signal ()
> from sysroot/usr/java/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so
> #5 0x00007f908673d993 in signalHandler(int, siginfo*, void*) ()
> from sysroot/usr/java/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so
> #6 <signal handler called>
> #7 0x00007f908452d782 in memcpy () from sysroot/lib64/libc.so.6
> #8 0x0000000000dbb61b in InitTupleFromTemplate (this=<optimized out>,
> tuple_byte_size=<optimized out>, tuple=<optimized out>, template_tuple=0x0)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/hdfs-scanner.h:446
> #9 InitTuple (tuple=<optimized out>, template_tuple=0x0,
> desc=<optimized out>, this=0x1ae72f00)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/hdfs-scanner.h:432
> #10 impala::HdfsParquetScanner::AssembleCollection (this=<optimized out>,
> column_readers=..., new_collection_rep_level=<optimized out>,
> coll_value_builder=coll_value_builder@entry=0x7f8fc0d1d160)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/hdfs-parquet-scanner.cc:1249
> #11 0x0000000000de3ba1 in ReadSlot (pool=0x1b06e418, tuple=<optimized out>,
> this=0x1ae72f00)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/parquet-column-readers.cc:1280
> #12 impala::CollectionColumnReader::ReadValue (this=0x1ae72f00,
> pool=0x1b06e418, tuple=<optimized out>)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/parquet-column-readers.cc:1256
> #13 0x0000000040bcc682 in ?? ()
> #14 0x0000000027b91c20 in ?? ()
> #15 0x000000001b06e418 in ?? ()
> #16 0x000000003f565000 in ?? ()
> #17 0x0000000000000044 in ?? ()
> #18 0x00007f8fc0d1d2e0 in ?? ()
> #19 0x000000003f565000 in ?? ()
> #20 0x0000000000000003 in ?? ()
> #21 0x0000000000000400 in ?? ()
> #22 0x000000001b06e3d0 in ?? ()
> #23 0x000000001b06e418 in ?? ()
> #24 0x0000000000de36b1 in impala::ParquetColumnReader::ReadValueBatch (
> this=0x2485d2a0, pool=0x2485d2e8, max_values=1086113410, tuple_size=1,
> tuple_mem=0x40bcc000 "", num_values=0x1b06e418)
> at /usr/src/debug/impala-2.11.0-cdh5.14.0/be/src/exec/parquet-column-readers.cc:802
> #25 0x0000000002de8c50 in vtable for parquet::Statistics ()
> #26 0x0000000040bcc000 in ?? ()
> #27 0x000000001ae72f00 in ?? ()
> #28 0x000000001ae72f00 in ?? ()
> #29 0x00000000ffffffff in ?? ()
> #30 0x0000000000000000 in ?? ()
> {noformat}
> I have a repro:
> {code}
> impala-shell.sh -q "
> DROP TABLE IF EXISTS complextypes_partitioned;
> CREATE TABLE complextypes_partitioned (
> id BIGINT,
> int_array ARRAY<INT>,
> int_array_array ARRAY<ARRAY<INT>>,
> int_map MAP<STRING,INT>,
> int_map_array ARRAY<MAP<STRING,INT>>,
> nested_struct STRUCT<a:INT,b:ARRAY<INT>,c:STRUCT<d:ARRAY<ARRAY<STRUCT<e:INT,f:STRING>>>>,g:MAP<STRING,STRUCT<h:STRUCT<i:ARRAY<DOUBLE>>>>>
> )
> PARTITIONED BY (
> part int
> )
> STORED AS PARQUET;
> alter table complextypes_partitioned add partition (part=1);
> "
> hdfs dfs -cp "/test-warehouse/complextypestbl_parquet/*" /test-warehouse/complextypes_partitioned/part=1
> impala-shell.sh -q "
> REFRESH complextypes_partitioned;
> select id, part, a.item from complextypes_partitioned t, t.int_array a
> "
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)