You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Soumyakanti Das (Jira)" <ji...@apache.org> on 2025/04/30 23:01:00 UTC

[jira] [Updated] (HIVE-28938) Error in LATERAL VIEW with non native tables due to presence of incorrect virtual columns in RowResolver

     [ https://issues.apache.org/jira/browse/HIVE-28938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Soumyakanti Das updated HIVE-28938:
-----------------------------------
    Description: 
To reproduce, run:
{noformat}
create external table test(id int, arr array<string>) stored by iceberg;
insert into test values (1, array("a", "b")), (2, array("c", "d")), (3, array("e", "f"));

select * from test
lateral view explode(arr) tbl1 as name
lateral view explode(arr) tbl2 as name1;
{noformat}
Error:
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column reference 'BLOCK__OFFSET__INSIDE__FILE'
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:14032)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13971)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:5027)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4801)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:11888)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:11827)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12812)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12665)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12696)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12678)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:688)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13572)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:489)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108){noformat}
 

BLOCK_{_}OFFSET{_}{_}INSIDE{_}_FILE should not be present for non native tables, but it is present because of the misclassification of ICEBERG tables as TableType.NATIVE in the method [obtainTableType(Table tabMetaData)|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3272]. This returns NATIVE for ICEBERG tables and subsequently adds incorrect virtual columns in [genTableLogicalPlan|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3079].

Virtual column adding logic should be similar to [here|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L11990].

P.S. This only occurs for multiple lateral views. My guess is that when we just have one lateral view, unnecessary fields are trimmed from the HiveProject on top of the HiveTableFunctionScan. But when we have two or more lateral views, HiveProjects which are not at the top of the plan has all fields as the lateral function on top of the Project (in HiveTableFunctionScan) refers to them.

For example, this is a plan with two lateral views:
{noformat}
query: explain cbo
select * from test
lateral view explode(arr) tbl1 as name
lateral view explode(arr) tbl2 as name1

CBO PLAN:
HiveProject(id=[$0], arr=[$1], name=[$8], name1=[$9])
  HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7, $8)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name, VARCHAR(2147483647) tbl2.name1)])
    HiveProject(id=[$0], arr=[$1], PARTITION__SPEC__ID=[$2], PARTITION__HASH=[$3], FILE__PATH=[$4], ROW__POSITION=[$5], PARTITION__PROJECTION=[$6], SNAPSHOT__ID=[$7], tbl1.name=[$8])
      HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name)])
        HiveTableScan(table=[[default, test]], table:alias=[test]) {noformat}
We can see the second Project has all columns because the lateral function on top of it refers to them. Whereas, this is a plan with just one lateral view:
{noformat}
query: explain cbo
select * from test
lateral view explode(arr) tbl1 as name

CBO PLAN:
HiveProject(id=[$0], arr=[$1], name=[$8])
  HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name)])
    HiveTableScan(table=[[default, test]], table:alias=[test]){noformat}

  was:
To reproduce, run:
{noformat}
create external table test(id int, arr array<string>) stored by iceberg;
insert into test values (1, array("a", "b")), (2, array("c", "d")), (3, array("e", "f"));

select * from test
lateral view explode(arr) tbl1 as name
lateral view explode(arr) tbl2 as name1;
{noformat}
Error:
{noformat}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column reference 'BLOCK__OFFSET__INSIDE__FILE'
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:14032)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13971)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:5027)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4801)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:11888)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:11827)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12812)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12665)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12696)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12678)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:688)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13572)
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:489)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108){noformat}
 

BLOCK__OFFSET__INSIDE__FILE should not be present for non native tables, but it is present because of the misclassification of ICEBERG tables as TableType.NATIVE in the method [obtainTableType(Table tabMetaData)|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3272]. This returns NATIVE for ICEBERG tables and subsequently adds incorrect virtual columns in [genTableLogicalPlan|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3079]. 

Virtual column adding logic should be similar to  [here|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L11990].


> Error in LATERAL VIEW with non native tables due to presence of incorrect virtual columns in RowResolver
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-28938
>                 URL: https://issues.apache.org/jira/browse/HIVE-28938
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 4.0.1
>            Reporter: Soumyakanti Das
>            Assignee: Soumyakanti Das
>            Priority: Major
>
> To reproduce, run:
> {noformat}
> create external table test(id int, arr array<string>) stored by iceberg;
> insert into test values (1, array("a", "b")), (2, array("c", "d")), (3, array("e", "f"));
> select * from test
> lateral view explode(arr) tbl1 as name
> lateral view explode(arr) tbl2 as name1;
> {noformat}
> Error:
> {noformat}
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid column reference 'BLOCK__OFFSET__INSIDE__FILE'
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:14032)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:13971)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:5027)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4801)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:11888)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:11827)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12812)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12665)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12696)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:12678)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:688)
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13572)
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:489)
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:324)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:227)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:108){noformat}
>  
> BLOCK_{_}OFFSET{_}{_}INSIDE{_}_FILE should not be present for non native tables, but it is present because of the misclassification of ICEBERG tables as TableType.NATIVE in the method [obtainTableType(Table tabMetaData)|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3272]. This returns NATIVE for ICEBERG tables and subsequently adds incorrect virtual columns in [genTableLogicalPlan|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L3079].
> Virtual column adding logic should be similar to [here|https://github.com/apache/hive/blob/e7cdc0cac935b8e2c573edad9d6e790d8beeee40/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L11990].
> P.S. This only occurs for multiple lateral views. My guess is that when we just have one lateral view, unnecessary fields are trimmed from the HiveProject on top of the HiveTableFunctionScan. But when we have two or more lateral views, HiveProjects which are not at the top of the plan has all fields as the lateral function on top of the Project (in HiveTableFunctionScan) refers to them.
> For example, this is a plan with two lateral views:
> {noformat}
> query: explain cbo
> select * from test
> lateral view explode(arr) tbl1 as name
> lateral view explode(arr) tbl2 as name1
> CBO PLAN:
> HiveProject(id=[$0], arr=[$1], name=[$8], name1=[$9])
>   HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7, $8)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name, VARCHAR(2147483647) tbl2.name1)])
>     HiveProject(id=[$0], arr=[$1], PARTITION__SPEC__ID=[$2], PARTITION__HASH=[$3], FILE__PATH=[$4], ROW__POSITION=[$5], PARTITION__PROJECTION=[$6], SNAPSHOT__ID=[$7], tbl1.name=[$8])
>       HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name)])
>         HiveTableScan(table=[[default, test]], table:alias=[test]) {noformat}
> We can see the second Project has all columns because the lateral function on top of it refers to them. Whereas, this is a plan with just one lateral view:
> {noformat}
> query: explain cbo
> select * from test
> lateral view explode(arr) tbl1 as name
> CBO PLAN:
> HiveProject(id=[$0], arr=[$1], name=[$8])
>   HiveTableFunctionScan(invocation=[LATERAL(explode($1), $0, $1, $2, $3, $4, $5, $6, $7)], rowType=[RecordType(INTEGER id, VARCHAR(2147483647) ARRAY arr, INTEGER PARTITION__SPEC__ID, BIGINT PARTITION__HASH, VARCHAR(2147483647) FILE__PATH, BIGINT ROW__POSITION, VARCHAR(2147483647) PARTITION__PROJECTION, BIGINT SNAPSHOT__ID, VARCHAR(2147483647) tbl1.name)])
>     HiveTableScan(table=[[default, test]], table:alias=[test]){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)