You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "kokila-19 (via GitHub)" <gi...@apache.org> on 2025/05/06 18:58:49 UTC
[PR] [WIP] Mask File, data sizes in tests [hive]
kokila-19 opened a new pull request, #5803:
URL: https://github.com/apache/hive/pull/5803
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin merged PR #5803:
URL: https://github.com/apache/hive/pull/5803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2077443236
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -353,10 +354,17 @@ private ArrayList<Pair<Pattern, String>> initPatternWithMaskComments() {
{
add(toPatternPair("(pblob|s3.?|swift|wasb.?).*hive-staging.*",
"### BLOBSTORE_STAGING_PATH ###"));
- add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3$4 %s $6%s",
- HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_DATE_MASK, HDFS_MASK)));
+ add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3%s %s $6%s",
+ HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_SIZE_MASK, HDFS_DATE_MASK, HDFS_MASK)));
add(toPatternPair(PATH_HDFS_REGEX, String.format("$1%s", HDFS_MASK)));
add(toPatternPair("(.*totalSize\\s*=*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
+ add(toPatternPair("(.*HDFS_BYTES_READ:*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
Review Comment:
i am not sure we should mask HDFS_BYTES_READ, CACHE_HIT_BYTES, CACHE_MISS_BYTES completely. We might miss issues in the future because of that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2895148068
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102122342
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -44,13 +44,17 @@
public class QOutProcessor {
public static final String PATH_HDFS_REGEX = "(hdfs://)([a-zA-Z0-9:/_\\-\\.=])+";
public static final String PATH_HDFS_WITH_DATE_USER_GROUP_REGEX =
- "([a-z]+) ([a-z]+)([ ]+)([0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ "([a-z]+) ([a-z]+)([ ]+)((?!0\\s)[0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ PATH_HDFS_REGEX;
+ public static final String PATH_HDFS_WITH_DATE_USER_GROUP_ZERO_SIZE_REGEX =
Review Comment:
Because of the way we add mask with patterns
```
add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3%s %s $6%s",
HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_SIZE_MASK, HDFS_DATE_MASK, HDFS_MASK)));
```
Here, we can't distinguish between 0 and non-zero values to decide whether to use HDFS_SIZE_MASK or not. So , I had to add another regex to make it work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2858241339
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102086963
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -44,13 +44,17 @@
public class QOutProcessor {
public static final String PATH_HDFS_REGEX = "(hdfs://)([a-zA-Z0-9:/_\\-\\.=])+";
public static final String PATH_HDFS_WITH_DATE_USER_GROUP_REGEX =
- "([a-z]+) ([a-z]+)([ ]+)([0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ "([a-z]+) ([a-z]+)([ ]+)((?!0\\s)[0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ PATH_HDFS_REGEX;
+ public static final String PATH_HDFS_WITH_DATE_USER_GROUP_ZERO_SIZE_REGEX =
Review Comment:
isn’t `PATH_HDFS_WITH_DATE_USER_GROUP_REGEX` enough? it shouldn’t just mask 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2912819352
Merged. Thanks @kokila-19 for submitting the patch, @deniskuzZ @simhadri-g for your reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2893549669
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2860367238
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] Mask File, data sizes in tests [hive]
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2077937243
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -353,10 +354,17 @@ private ArrayList<Pair<Pattern, String>> initPatternWithMaskComments() {
{
add(toPatternPair("(pblob|s3.?|swift|wasb.?).*hive-staging.*",
"### BLOBSTORE_STAGING_PATH ###"));
- add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3$4 %s $6%s",
- HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_DATE_MASK, HDFS_MASK)));
+ add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3%s %s $6%s",
+ HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_SIZE_MASK, HDFS_DATE_MASK, HDFS_MASK)));
add(toPatternPair(PATH_HDFS_REGEX, String.format("$1%s", HDFS_MASK)));
add(toPatternPair("(.*totalSize\\s*=*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
+ add(toPatternPair("(.*HDFS_BYTES_READ:*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
Review Comment:
Considering if we should not mask it due to possibility of not detecting the issues, then we can only overwrite this for upgrades.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102197874
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: array(const struct(1),const struct(2),const struct(3),const struct(4)) (type: array<struct<col1:int>>)
outputColumnNames: _col0
- Statistics: Num rows: 1/1 Data size: 72 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
I too think its better to correct the data size.
* We can infer quite a bit of info from the actual value and catch issues more easily(such as, if predicate push down is resulting in less data being read?)
* Commit history will also be preserved which will allow us to track which commit modified the data size (Like the orc upgrade mentioned in previous comment by okumin)
Sorry , if i am missing some context as to why we are masking the data size in this PR. It is expected that new optimizations in dependent projects (like orc,parquet etc) will change data size right? That is capture by the this value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2870956263
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102197874
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: array(const struct(1),const struct(2),const struct(3),const struct(4)) (type: array<struct<col1:int>>)
outputColumnNames: _col0
- Statistics: Num rows: 1/1 Data size: 72 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
I too think its better to correct the data size.
* We can infer quite a bit of info from the actual value and catch issues more easily(such as, if predicate push down is resulting in less data being read?)
* Commit history will also be preserved which will allow us to track which commit modified the data size (Like the orc upgrade mentioned in previous comment by okumin)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2079620957
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -353,10 +354,17 @@ private ArrayList<Pair<Pattern, String>> initPatternWithMaskComments() {
{
add(toPatternPair("(pblob|s3.?|swift|wasb.?).*hive-staging.*",
"### BLOBSTORE_STAGING_PATH ###"));
- add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3$4 %s $6%s",
- HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_DATE_MASK, HDFS_MASK)));
+ add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3%s %s $6%s",
+ HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_SIZE_MASK, HDFS_DATE_MASK, HDFS_MASK)));
add(toPatternPair(PATH_HDFS_REGEX, String.format("$1%s", HDFS_MASK)));
add(toPatternPair("(.*totalSize\\s*=*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
+ add(toPatternPair("(.*HDFS_BYTES_READ:*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
Review Comment:
that's ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102117233
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
@kokila-19, can we fallback to ZLIB with "orc.compress"="ZLIB" config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2912409137
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2863977141
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102086963
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -44,13 +44,17 @@
public class QOutProcessor {
public static final String PATH_HDFS_REGEX = "(hdfs://)([a-zA-Z0-9:/_\\-\\.=])+";
public static final String PATH_HDFS_WITH_DATE_USER_GROUP_REGEX =
- "([a-z]+) ([a-z]+)([ ]+)([0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ "([a-z]+) ([a-z]+)([ ]+)((?!0\\s)[0-9]+) ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}) "
+ PATH_HDFS_REGEX;
+ public static final String PATH_HDFS_WITH_DATE_USER_GROUP_ZERO_SIZE_REGEX =
Review Comment:
isn’t `PATH_HDFS_WITH_DATE_USER_GROUP_REGEX` enough, it shouldn’t just mask 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102117233
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
@kokila-19, can we set the "orc.compress"="ZLIB" by default? I don't think it's appropriate to modify this during the JDK migration.
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/TableProperties.java#L209
````
public static final String ORC_COMPRESSION_DEFAULT = "zlib";
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102257943
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
We can configure it using orc.compress, but that's a table-level property in Hive. I'll look into setting a global default and where it should be defined.
If this can be done, I will take up testing ZSTD separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2874002425
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] Mask File, data sizes in tests [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2855813326
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5803&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5803&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5803&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5803)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2080183943
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -353,10 +354,17 @@ private ArrayList<Pair<Pattern, String>> initPatternWithMaskComments() {
{
add(toPatternPair("(pblob|s3.?|swift|wasb.?).*hive-staging.*",
"### BLOBSTORE_STAGING_PATH ###"));
- add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3$4 %s $6%s",
- HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_DATE_MASK, HDFS_MASK)));
+ add(toPatternPair(PATH_HDFS_WITH_DATE_USER_GROUP_REGEX, String.format("%s %s$3%s %s $6%s",
+ HDFS_USER_MASK, HDFS_GROUP_MASK, HDFS_SIZE_MASK, HDFS_DATE_MASK, HDFS_MASK)));
add(toPatternPair(PATH_HDFS_REGEX, String.format("$1%s", HDFS_MASK)));
add(toPatternPair("(.*totalSize\\s*=*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
+ add(toPatternPair("(.*HDFS_BYTES_READ:*\\s*)\\d+\\s*(.*)", "$1#Masked#$2"));
Review Comment:
I have updated the PR, please review it.
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102117233
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
@kokila-19, can we fallback to ZLIB with some Hive/ORC configs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102108827
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java:
##########
@@ -67,6 +71,9 @@ public class QOutProcessor {
private static final PatternReplacementPair MASK_LINEAGE = new PatternReplacementPair(
Pattern.compile("POSTHOOK: Lineage: .*"),
"POSTHOOK: Lineage: ###Masked###");
+
+ public static final String FILE_SIZE_MASK = "$1#Masked#";
+ public static final String MASK_SIZE_PATTERN = "$1#Masked#$2";
Review Comment:
`DATA_SIZE_MASK`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #5803:
URL: https://github.com/apache/hive/pull/5803#discussion_r2102197874
##########
ql/src/test/results/clientpositive/llap/explainanalyze_acid_with_direct_insert.q.out:
##########
@@ -47,21 +47,21 @@ STAGE PLANS:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
- Statistics: Num rows: 1/1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: array(const struct(1),const struct(2),const struct(3),const struct(4)) (type: array<struct<col1:int>>)
outputColumnNames: _col0
- Statistics: Num rows: 1/1 Data size: 72 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1/1 Data size: ###Masked### Basic stats: COMPLETE Column stats: COMPLETE
Review Comment:
I too think its better to correct the data size.
* We can infer quite a bit of info from the actual value and catch issues more easily(such as, if predicate push down is resulting in less data being read?)
* Commit history will also be preserved which will allow us to track which commit modified the data size (Like the orc upgrade mentioned in previous comment by okumin)
Sorry , if i am missing some context as to why we are masking the data size in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28944: Mask File, data sizes in tests [hive]
Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on PR #5803:
URL: https://github.com/apache/hive/pull/5803#issuecomment-2895517532
@deniskuzZ @okumin Thanks for the comments and I have addressed them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org