You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Zach Amsden (JIRA)" <ji...@apache.org> on 2017/12/05 00:40:00 UTC

[jira] [Closed] (IMPALA-6068) Dataload does not populate functional_*.complextypes_fileformat correctly

     [ https://issues.apache.org/jira/browse/IMPALA-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zach Amsden closed IMPALA-6068.
-------------------------------
    Resolution: Fixed

IMPALA-6068: Scale back fixing functional-types

I re-created the original patch for IMPALA-6068, but only
performed what I believe to be the limited legal transformation
of data load: DEPENDENT_LOAD -> DEPENDENT_LOAD_HIVE.

Any place that directly uploads via hadoop or hdfs commands
was left alone as changing it can't be proven to be correct.

Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Testing: Doing a full data load.  I mistakenly changed a variable
name causing the first two dry-runs to fail.
Reviewed-on: http://gerrit.cloudera.org:8080/8690
Reviewed-by: Zach Amsden <za...@cloudera.com>
Tested-by: Zach Amsden <za...@cloudera.com>
---
M testdata/bin/generate-schema-statements.py
M testdata/common/widetable.py
M testdata/datasets/functional/functional_schema_template.sql
3 files changed, 61 insertions(+), 38 deletions(-)

> Dataload does not populate functional_*.complextypes_fileformat correctly
> -------------------------------------------------------------------------
>
>                 Key: IMPALA-6068
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6068
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Infrastructure
>    Affects Versions: Impala 2.10.0
>            Reporter: Joe McDonnell
>            Assignee: Joe McDonnell
>            Priority: Critical
>             Fix For: Impala 2.11.0
>
>
> functional.complextypes_fileformat is a text table containing some nested data.
> Data load is supposed to generate functional.complextypes_fileformat in this order:
> 1. Create table functional.complextypes_fileformat
> 2. Populate functional.complextypes_fileformat using
> INSERT OVERWRITE TABLE {db_name}{db_suffix}.{table_name} SELECT id, named_struct("f1",string_col,"f2",int_col), array(1, 2, 3), map("k", cast(0 as bigint)) FROM functional.alltypestiny;
> 3. Create tables functional_*.complextypes_fileformat
> 4. Populate each table using:
> INSERT OVERWRITE TABLE {table_name} SELECT * FROM functional.{table_name};
> However, dataload is doing this in the wrong order. It does #1, #3, #4, and then finally #2. This means that #4 is operating on zero rows, so all the functional_*.complextypes_fileformat tables have zero rows. Oddly enough, dataload also generates #4 to insert into functional.complextypes_fileformat so it is overwriting itself using rows from itself. Dataload should do this in the correct order (and avoid this weirdness). 
> This is only used for frontend tests, but it can cause issues with recent versions of Hive, because Hive seems to skip creating a file when it would be writing zero rows. That can alter the number of files listed in the plan.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)