You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "shivjha30 (via GitHub)" <gi...@apache.org> on 2024/01/11 13:14:38 UTC
[PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
shivjha30 opened a new pull request, #5000:
URL: https://github.com/apache/hive/pull/5000
### What changes were proposed in this pull request?
Earlier, the code flows for partitioned and non-partitioned tables were different. The partitioned tables skipped constraints checks before submitting the job for execution. This Pull request ensures that both, the partitioned and non-partitioned tables go through the constraints validations in `applyConstraintsAndGetFiles` function.
### Why are the changes needed?
For partitioned tables, while executing LOAD DATA/ LOAD DATA LOCAL commands, the check for file existence is not executed on HiveServer2, and this in turn throws `java.io.FileNotFoundException` during Runtime once the job is launched.
This PR prevents such cases at compile time.
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
The test cases already exist. The error messages prompted back to the user are now consistent if the file is not found at HiveServer2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1547115174
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
when we create a table **Tstage**, in warehouse directory the table will be created in small case letters like "tstage". While we are performing Load Data we are appending warehousedir + "Tstage" in https://github.com/apache/hive/blob/0bc624ade3020826b4e7101b7ea4789be7e8585b/ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java#L406
As Linux is case sensitive, it couldnt find the path of warehousedir + "Tstage" but there is a path of warehousedir + "tstage" exists.
Previously, the file path wasn't being validated in the Hive server. Now, after implementing a fix, the test has failed because the Hive server is now validating whether the file path exists or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1665939418
##########
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:
##########
@@ -3422,4 +3422,51 @@ public void testHeaderFooterNonTextFiles() throws Exception {
stmt.close();
}
}
+ /**
+ * These test methods validate the error handling when attempting to load data from a non-existent file into different types of Hive tables.
+ * Each test creates a specific type of table ( dynamically partitioned, or bucketed), and then attempts to load data from a non-existent file.
+ * The tests pass if an exception is thrown with a message containing the string "Invalid path".
+ */
+ @Test
+ public void testLoadDataNegativeForDynamicPartition() throws Exception {
+ HiveStatement stmt = (HiveStatement) con.createStatement();
+ try {
+ stmt.execute("drop table if exists T");
+ stmt.execute("create table T (a int, b int) partitioned by (p int) stored as orc");
+ stmt.execute("set hive.exec.dynamic.partition=true");
Review Comment:
Why we need to set these parameters explicitly, we can test with default values only. All the test cases have these parameters every where, we can test with default values unless any explicit case we need test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1557448538
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
why do you need `fileSystem` here? move it to the line where it's used together with `try`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2083711642
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam merged PR #5000:
URL: https://github.com/apache/hive/pull/5000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1547102223
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
The changes were introduced only for Partition tables, there is no change in behaviour for Non Partition.
For partition tables for local , Before the fix the exception thrown was File Not Found from Node Manager, if file would be present at hs2. For hdfs + partition, no exception is getting thrown. After the fix the exception will be thrown from Hive Server itself, and the exception now would be Semantic Exception for both hdfs and local .
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
<img width="394" alt="image" src="https://github.com/apache/hive/assets/149884343/658eb103-9c1a-4fd3-92f6-19228474660c">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1577238980
##########
ql/src/test/results/clientnegative/load_wrong_noof_part.q.out:
##########
@@ -6,4 +6,4 @@ POSTHOOK: query: CREATE TABLE loadpart1(a STRING, b STRING) PARTITIONED BY (ds S
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@loadpart1
-FAILED: SemanticException [Error 10006]: Line 2:82 Partition not found ''2009-05-05''
+#### A masked pattern was here ####
Review Comment:
How it is related to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2082615712
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1667891174
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
For partition case if the the file does exist at hs2, and the container looks for the file at other machines it can throw the execution error from hdfs end, that scenario is not covered in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2222098560
Overall changes looks good to me, But locally when i run newly added test case it falling
`[ERROR] TestNegativeLlapCliDriver.testCliDriver:59 Client Execution succeeded but contained differences (error code = 1) after executing load_data_partition.q
15,21c15
< FAILED: SemanticException Line 2:17 Invalid path ''hdfs://### HDFS PATH ###'': No files matching path hdfs://### HDFS PATH ###
< FAILED: AssertionError java.lang.AssertionError: Client Execution succeeded but contained differences (error code = 1) after executing load_data_partition.q
< 15c15
< < FAILED: SemanticException Line 2:17 Invalid path ''hdfs://### HDFS PATH ###'': No files matching path hdfs://### HDFS PATH ###
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1578076516
##########
ql/src/test/results/clientnegative/load_wrong_noof_part.q.out:
##########
@@ -6,4 +6,4 @@ POSTHOOK: query: CREATE TABLE loadpart1(a STRING, b STRING) PARTITIONED BY (ds S
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@loadpart1
-FAILED: SemanticException [Error 10006]: Line 2:82 Partition not found ''2009-05-05''
+#### A masked pattern was here ####
Review Comment:
In HIVE-5601, the file path in the q file was given an incorrect path, although the purpose of the test was to check for number of partitions. With this PR change that wrong path would throw the semantic exception even before checking the number of partitions.
Instead of aiming for the file path and raising a semantic exception, I have updated the q file with the right file path, fulfilling the task of merely verifying the number of partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1684108398
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
ok, wasn't the issue only with the inpath for Load Data?
````
load data local inpath '" + getWarehouseDir() + "/tstage' into table T
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1685240923
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
Yes lowercased inpath dir should work, but need to comment why we are lowering the table name from camel notation without using as it is, it will help for further extension of the test case. IIUC test case scenario wise camel notation of table name is not needed.
All the test cases in this class uses camel notation, only to make it consistent I agree with denis maintain camel notation of table name here also and lowercased only inpath dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1887705670
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2225495105
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2074919852
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1588090924
##########
ql/src/test/queries/clientnegative/load_data_partition.q:
##########
@@ -0,0 +1,7 @@
+--load data if the file does not exist
+
+CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE;
+LOAD DATA INPATH 'hdfs:///validateload/filedoesnt.txt' INTO TABLE validate_load_data partition (hr);
+SELECT * FROM validate_load_data;
+DROP TABLE validate_load_data;
Review Comment:
Can you check this, without fix also it is throwing error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2046643880
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1557452129
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -325,6 +337,8 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
if(ts.tableHandle.isStoredAsSubDirectories()) {
throw new SemanticException(ErrorMsg.LOAD_INTO_STORED_AS_DIR.getMsg());
}
+ // validate the arguments
+ FileStatus[] srcs = verifyFilePathExists(fromURI);
Review Comment:
why don't we validate `fromURI` sooner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1568283994
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
@deniskuzZ Can you look at the changes and review it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1557448538
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
why do you need `fileSystem` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2093630517
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2084367398
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1683756644
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
The problem at hand pertains to the case sensitivity of the table name, which is currently in uppercase. Despite modifying the directory in the input path to lowercase, the case sensitivity for the table name continues to persist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2207633030
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2211342185
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2051284224
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2092727239
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1671555143
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -325,6 +337,8 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
if(ts.tableHandle.isStoredAsSubDirectories()) {
throw new SemanticException(ErrorMsg.LOAD_INTO_STORED_AS_DIR.getMsg());
}
+ // validate the arguments
+ FileStatus[] srcs = verifyFilePathExists(fromURI);
Review Comment:
Comment is updated and resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2238168629
+1 Thanks for the review Ayush Saxena, Denys Kuzmenko
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1546686701
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
Why these changes are needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1665955733
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
Move this to above the line where it is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1920553543
There is a test failure in Jenkins, i ran the test in in local and that test is passing which can be viewed in the below screenshot:
<img width="1707" alt="image" src="https://github.com/apache/hive/assets/149884343/aa3e7165-8595-4313-b171-29f52f41bcd0">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1924055005
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1943126065
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1547102223
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
The changes were introduced only for Partition tables, there is no change in behaviour for Non Partition.
For partition tables for both local and hdfs path , Before the fix the exception thrown was File Not Found from Node Manager. After the fix the exception will be thrown from Hive Server itself, and the exception now would be Semantic Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1589937210
##########
ql/src/test/queries/clientnegative/load_data_partition.q:
##########
@@ -0,0 +1,7 @@
+--load data if the file does not exist
+
+CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE;
+LOAD DATA INPATH 'hdfs:///validateload/filedoesnt.txt' INTO TABLE validate_load_data partition (hr);
+SELECT * FROM validate_load_data;
+DROP TABLE validate_load_data;
Review Comment:
@chinnaraolalam the following scenario is now handled, can you review this once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1557448538
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
why do you need `fileSystem` here? please move it to the line where it's used together with `try`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1557448538
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
why do you need `fileSystem` here? move it to the line where it's used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1671607076
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
This comment is update and resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1576424351
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
In the partition case if the file doesn't exist then number of splits will be zero, as the splits are zero the mappers also will be equal to zero. As there are no mappers the CombineHiveInputformat will not be called.
Now after the fix we would be validating at the server level, thus in all such cases in the existing applications, those queries needs to be corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1558845820
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -181,7 +184,16 @@ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table) t
ErrorMsg.INVALID_PATH.getMsg(), fromTree,
"No files matching path " + fromURI));
}
-
+ } catch (IOException e) {
+ throw new SemanticException(ASTErrorUtils.getMsg(
+ ErrorMsg.INVALID_PATH.getMsg(), fromTree), e);
+ }
+ return srcs;
+ }
+ private List<FileStatus> applyConstraintsAndGetFiles(URI fromURI, Table table, FileStatus[] srcs)
+ throws SemanticException {
+ try {
+ FileSystem fileSystem = FileSystem.get(fromURI, conf);
Review Comment:
Down the line in the applyConstraintsAndGetFiles, filesystem is getting used in "AcidUtils.validateAcidFiles(table, srcs, fileSystem);"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-2074262864
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5000&metric=new_accepted_issues&view=list)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1922485979
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1943585524
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1924993607
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1680742932
##########
ql/src/test/results/clientnegative/load_data_partition.q.out:
##########
@@ -0,0 +1,15 @@
+PREHOOK: query: drop table if exists validate_load_data
+PREHOOK: type: DROPTABLE
+PREHOOK: Output: database:default
+POSTHOOK: query: drop table if exists validate_load_data
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Output: database:default
+PREHOOK: query: CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@validate_load_data
+POSTHOOK: query: CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@validate_load_data
+#### A masked pattern was here ####
Review Comment:
After masking the failure scenario is coming like this, before that
the error is coming as expected
"FAILED: SemanticException Line 2:17 Invalid path ''/tmp/kv1.txt'': No files matching path hdfs://localhost:65046/tmp/kv1.txt"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1923268567
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
[0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5000&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5000)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "shivjha30 (via GitHub)" <gi...@apache.org>.
shivjha30 commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1920534391
@ayushtkn Thanks for the review, i have added the test case as mentioned above.please review once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on PR #5000:
URL: https://github.com/apache/hive/pull/5000#issuecomment-1968253683
This scenario will fail before fixing this issue at job execution and now it is validated at Hiveserver and it will throw SemanticExceptoin to the client. What is the exception received by client before fixing this. I think exception will be different, can you add what exception client is receiving before fixing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1546692298
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
What is behaviour of below scenarios before and after fix and what exception it is throwing
Partitioned - Local path
Partitioned - HDFS path
Non-Partitioned - Local path
Non-Partitioned - HDFS path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1575895273
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
So earlier in partition case, even though file not exists query was successful. Can you check why it is not throwing earlier.
After fix it was validated at server level and it will throw error for non existing file paths. So if any such cases are there in existing applications, those queries need to corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1665939418
##########
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:
##########
@@ -3422,4 +3422,51 @@ public void testHeaderFooterNonTextFiles() throws Exception {
stmt.close();
}
}
+ /**
+ * These test methods validate the error handling when attempting to load data from a non-existent file into different types of Hive tables.
+ * Each test creates a specific type of table ( dynamically partitioned, or bucketed), and then attempts to load data from a non-existent file.
+ * The tests pass if an exception is thrown with a message containing the string "Invalid path".
+ */
+ @Test
+ public void testLoadDataNegativeForDynamicPartition() throws Exception {
+ HiveStatement stmt = (HiveStatement) con.createStatement();
+ try {
+ stmt.execute("drop table if exists T");
+ stmt.execute("create table T (a int, b int) partitioned by (p int) stored as orc");
+ stmt.execute("set hive.exec.dynamic.partition=true");
Review Comment:
Why we need to set these parameters explicitly, we can test with default values only. All the test cases have these parameters every where, we can test with default values unless any explicit case we need test.
Keep test case name as descriptive. Ex: testLoadNonExistingFileToDynamicPartition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1665952140
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
##########
@@ -344,7 +358,7 @@ private void analyzeLoad(ASTNode ast) throws SemanticException {
}
// make sure the arguments make sense
- List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle);
+ List<FileStatus> files = applyConstraintsAndGetFiles(fromURI, ts.tableHandle, srcs);
Review Comment:
> For partition tables for local ,
In Local path case, If file would be present at hs2 what is the behaviour after fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1678827236
##########
ql/src/test/results/clientnegative/load_data_partition.q.out:
##########
@@ -0,0 +1,15 @@
+PREHOOK: query: drop table if exists validate_load_data
+PREHOOK: type: DROPTABLE
+PREHOOK: Output: database:default
+POSTHOOK: query: drop table if exists validate_load_data
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Output: database:default
+PREHOOK: query: CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@validate_load_data
+POSTHOOK: query: CREATE TABLE validate_load_data(key int, value string) partitioned by (hr int) STORED AS TEXTFILE
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@validate_load_data
+#### A masked pattern was here ####
Review Comment:
Here there is no error, I think it is failing before the expected scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1683375579
##########
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:
##########
@@ -394,16 +394,16 @@ public void loadDataPartitioned() throws Exception {
*/
@Test
public void testValidations() throws Exception {
- runStatementOnDriver("drop table if exists T");
- runStatementOnDriver("drop table if exists Tstage");
Review Comment:
@shivjha30, you should lowercased just the `inpath` dir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27995: Fix inconsistent behavior of LOAD DATA command for partitoned and non-partitioned tables [hive]
Posted by "chinnaraolalam (via GitHub)" <gi...@apache.org>.
chinnaraolalam commented on code in PR #5000:
URL: https://github.com/apache/hive/pull/5000#discussion_r1487125020
##########
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java:
##########
@@ -3422,4 +3422,26 @@ public void testHeaderFooterNonTextFiles() throws Exception {
stmt.close();
}
}
+ /**
Review Comment:
add all the use cases like partitioned , non-partitioned and load data use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org