You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LsomeYeah (via GitHub)" <gi...@apache.org> on 2023/06/14 08:44:48 UTC
[GitHub] [incubator-paimon] LsomeYeah opened a new pull request, #1371: [bug] Fix: make hive load the correct FileIO set by conf
LsomeYeah opened a new pull request, #1371:
URL: https://github.com/apache/incubator-paimon/pull/1371
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
When creating table in hive, if the scheme of the table location is not consistent with the fs.defaultFs in HiveConf, hive may load false FileIOLoader which will cause IOException.
So, before loading FileIOLoader, it should get the correct path according to the fs.defaultFs in HiveConf.
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
There is an e2e-test named 'testHiveWrite' in HiveE2eTest for fixing this bug.
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] tsreaper merged pull request #1371: [bug] Fix: make hive load the correct FileIO set by conf
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper merged PR #1371:
URL: https://github.com/apache/incubator-paimon/pull/1371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] tsreaper commented on a diff in pull request #1371: [bug] Fix: make hive load the correct FileIO set by conf
Posted by "tsreaper (via GitHub)" <gi...@apache.org>.
tsreaper commented on code in PR #1371:
URL: https://github.com/apache/incubator-paimon/pull/1371#discussion_r1229275560
##########
paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/PaimonMetaHook.java:
##########
@@ -76,10 +77,12 @@ public void preCreateTable(Table table) throws MetaException {
table.getSd().setInputFormat(PaimonInputFormat.class.getCanonicalName());
table.getSd().setOutputFormat(PaimonOutputFormat.class.getCanonicalName());
-
- String location = LocationKeyExtractor.getLocation(table);
+ String location = LocationKeyExtractor.getLocation(table, conf);
if (location == null) {
String warehouse = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
+ org.apache.hadoop.fs.Path hadoopPath =
+ getDnsPath(new org.apache.hadoop.fs.Path(warehouse), conf);
+ warehouse = hadoopPath.toUri().toString();
Identifier identifier = Identifier.create(table.getDbName(), table.getTableName());
location = AbstractCatalog.dataTableLocation(warehouse, identifier).toUri().toString();
table.getSd().setLocation(location);
Review Comment:
Why not merge these implementation into `LocationKeyExtractor.getLocation`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-paimon] LsomeYeah commented on a diff in pull request #1371: [bug] Fix: make hive load the correct FileIO set by conf
Posted by "LsomeYeah (via GitHub)" <gi...@apache.org>.
LsomeYeah commented on code in PR #1371:
URL: https://github.com/apache/incubator-paimon/pull/1371#discussion_r1229299145
##########
paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/PaimonMetaHook.java:
##########
@@ -76,10 +77,12 @@ public void preCreateTable(Table table) throws MetaException {
table.getSd().setInputFormat(PaimonInputFormat.class.getCanonicalName());
table.getSd().setOutputFormat(PaimonOutputFormat.class.getCanonicalName());
-
- String location = LocationKeyExtractor.getLocation(table);
+ String location = LocationKeyExtractor.getLocation(table, conf);
if (location == null) {
String warehouse = conf.get(HiveConf.ConfVars.METASTOREWAREHOUSE.varname);
+ org.apache.hadoop.fs.Path hadoopPath =
+ getDnsPath(new org.apache.hadoop.fs.Path(warehouse), conf);
+ warehouse = hadoopPath.toUri().toString();
Identifier identifier = Identifier.create(table.getDbName(), table.getTableName());
location = AbstractCatalog.dataTableLocation(warehouse, identifier).toUri().toString();
table.getSd().setLocation(location);
Review Comment:
`Identifier.class` and `AbstractCatalog.class` are used here. The two classes are defined in '`paimon-core`' module, if merge these implementation into `LocationKeyExtractor.getLocation`, we need add dependency to the `pom.xml` of the module '`paimon-hive : paimon-hive-common`' in which `LocationKeyExtractor.getLocation` is defined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org