You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dian Fu (Jira)" <ji...@apache.org> on 2020/04/17 02:17:00 UTC
[jira] [Updated] (FLINK-16860) Failed to push filter into
OrcTableSource when upgrading to 1.9.2
[ https://issues.apache.org/jira/browse/FLINK-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dian Fu updated FLINK-16860:
----------------------------
Summary: Failed to push filter into OrcTableSource when upgrading to 1.9.2 (was: TableException: Failed to push filter into table source! when upgrading flink to 1.9.2)
> Failed to push filter into OrcTableSource when upgrading to 1.9.2
> -----------------------------------------------------------------
>
> Key: FLINK-16860
> URL: https://issues.apache.org/jira/browse/FLINK-16860
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ORC, Table SQL / API
> Affects Versions: 1.9.2, 1.10.0
> Environment: flink 1.8.2
> flink 1.9.2
> Reporter: Nikola
> Assignee: Jingsong Lee
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.3, 1.10.1
>
> Attachments: flink-1.8.2.txt, flink-1.9.2.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We have a batch job which we currently have on a flink cluster running 1.8.2
> The job runs fine. We wanted to upgrade to flink 1.10, but that yielded errors, so we started downgrading until we found that the issue is in flink 1.9.2
> The job on 1.9.2 fails with:
> {code:java}
> Caused by: org.apache.flink.table.api.TableException: Failed to push filter into table source! table source with pushdown capability must override and change explainSource() API to explain the pushdown applied!{code}
> Which is not happening on flink 1.8.2. You can check the logs for the exactly same job, just running on different cluster versions: [^flink-1.8.2.txt] [^flink-1.9.2.txt]
>
> I tried to narrow it down and it seems that this exception has been added in FLINK-12399 and there was a small discussion regarding the exception: [https://github.com/apache/flink/pull/8468#discussion_r329876088]
> Our code looks something like this:
>
> {code:java}
> String tempTableName = "tempTable";
> String sql = SqlBuilder.buildSql(tempTableName);
> BatchTableEnvironment tableEnv = BatchTableEnvironment.create(env);
> OrcTableSource orcTableSource = OrcTableSource.builder()
> .path(hdfsFolder, true)
> .forOrcSchema(ORC.getSchema())
> .withConfiguration(config)
> .build();
> tableEnv.registerTableSource(tempTableName, orcTableSource);
> Table tempTable = tableEnv.sqlQuery(sql);
> return tableEnv.toDataSet(tempTable, Row.class);
> {code}
> Where the sql build is nothing more than
> {code:java}
> SELECT * FROM table WHERE id IN (1,2,3) AND mid IN(4,5,6){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)