You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "armitage420 (via GitHub)" <gi...@apache.org> on 2025/09/15 12:07:52 UTC
[PR] [WIP] Flaky test [hive]
armitage420 opened a new pull request, #6075:
URL: https://github.com/apache/hive/pull/6075
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3324298979
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] [WIP] Flaky test [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3291850494
# @check-spelling-bot Report
### :red_circle: Please review
See the [files](6075/files/) view or the [action log](https://github.com/apache/hive/actions/runs/17732473785) for details.
#### Unrecognized words (3)
bucketedtables
languagemanual
teradatabinaryserde
<details><summary>Previously acknowledged words that are now absent
</summary>aarry bytecode cwiki HIVEFETCHOUTPUTSERDE timestamplocal yyyy </details>
<details><summary>To accept these unrecognized words as correct (and remove the previously acknowledged and now absent words),
run the following commands</summary>
... in a clone of the [git@github.com:armitage420/hive.git](https://github.com/armitage420/hive.git) repository
on the `flakyTest` branch:
```
update_files() {
perl -e '
my @expect_files=qw('".github/actions/spelling/expect.txt"');
@ARGV=@expect_files;
my @stale=qw('"$patch_remove"');
my $re=join "|", @stale;
my $suffix=".".time();
my $previous="";
sub maybe_unlink { unlink($_[0]) if $_[0]; }
while (<>) {
if ($ARGV ne $old_argv) { maybe_unlink($previous); $previous="$ARGV$suffix"; rename($ARGV, $previous); open(ARGV_OUT, ">$ARGV"); select(ARGV_OUT); $old_argv = $ARGV; }
next if /^(?:$re)(?:(?:\r|\n)*$| .*)/; print;
}; maybe_unlink($previous);'
perl -e '
my $new_expect_file=".github/actions/spelling/expect.txt";
use File::Path qw(make_path);
use File::Basename qw(dirname);
make_path (dirname($new_expect_file));
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"$patch_add"');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a)."-".$a cmp lc($b)."-".$b} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;
system("git", "add", $new_expect_file);
'
}
comment_json=$(mktemp)
curl -L -s -S \
-H "Content-Type: application/json" \
"https://api.github.com/repos/apache/hive/issues/comments/3291850494" > "$comment_json"
comment_body=$(mktemp)
jq -r ".body // empty" "$comment_json" > $comment_body
rm $comment_json
patch_remove=$(perl -ne 'next unless s{^</summary>(.*)</details>$}{$1}; print' < "$comment_body")
patch_add=$(perl -e '$/=undef; $_=<>; if (m{Unrecognized words[^<]*</summary>\n*```\n*([^<]*)```\n*</details>$}m) { print "$1" } elsif (m{Unrecognized words[^<]*\n\n((?:\w.*\n)+)\n}m) { print "$1" };' < "$comment_body")
update_files
rm $comment_body
git add -u
```
</details>
<!--
{% comment %}
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to you under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{% endcomment %}
-->
<!-- See https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-advice --> <!-- markdownlint-disable MD033 MD041 -->
<details><summary>If the flagged items do not appear to be text</summary>
If items relate to a ...
* well-formed pattern.
If you can write a [pattern](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples:-patterns) that would match it,
try adding it to the `patterns.txt` file.
Patterns are Perl 5 Regular Expressions - you can [test](
https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your lines.
Note that patterns can't match multiline strings.
* binary file.
Please add a file path to the `excludes.txt` file matching the containing file.
File paths are Perl 5 Regular Expressions - you can [test](
https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your files.
`^` refers to the file's path from the root of the repository, so `^README\.md$` would exclude [README.md](
../tree/HEAD/README.md) (on whichever branch you're using).
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix masking and re-enable query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3356953799
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3292923789
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3303898054
> @armitage420
>
> > if the present masked values change
>
> why would they change?
Thank you for your time @deniskuzZ !
Total size properties might change with file format upgrade, and in our case, it's orc here. Here's the jira for reference: [HIVE-25607](https://issues.apache.org/jira/browse/HIVE-25607)
Followed by the above mentioned jira, there was another jira that introduced masking for the same reason in iceberg qfiles: [HIVE-25658](https://issues.apache.org/jira/browse/HIVE-25658)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3330074818
@deniskuzZ I've only seen this flakiness once in one of the PRs. While investigating, I found this JIRA that addressed a similar issue across Iceberg q-tests: [HIVE-25607](https://issues.apache.org/jira/browse/HIVE-25607)
I also encountered other flaky tests later (which seem fairly common) and created JIRAs in case anyone wants to address them: [HIVE-29158](https://issues.apache.org/jira/browse/HIVE-29158), [HIVE-29157](https://issues.apache.org/jira/browse/HIVE-29157)
Regarding the approach of masking before sorting, I have some concerns:
1. Sorting using SORT_QUERY_RESULTS is executed per query, whereas masking currently executes once per file. Changing the architecture to mask per query would increase the number of operations (and likely execution time for the pipeline).
2. The sorting discussed above is only applied to a few qfiles, so according to point 1, masking would still operate per query in those cases.
3. An approach where masking per query is done only when SORT_QUERY_RESULTS is used (but otherwise operates per file) might work better, though I'm not sure if this makes architectural sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix masking and re-enable query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3357572473
@deniskuzZ Thanks for your help :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix masking and re-enable query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #6075:
URL: https://github.com/apache/hive/pull/6075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix masking and re-enable query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3355601973
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3329891597
@armitage420 Every run that fails has a non-negligible cost in terms of money, time, and resources so allowing the flakiness to exist for multiple days/weeks is not OK so I disabled the test for now since the respective PR was approved.
In general, if we cannot solve a flakiness issue reasonably fast we opt to disable the test first and fix it afterwards; that's quite common practice in the Hive project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3329931318
@armitage420, @thomasrebele, I agree that a better approach would be to apply masking before sorting the results.
Since the test is currently disabled, we should aim to re-enable it as soon as possible to avoid unnoticed issues when upgrading to Iceberg 1.10.1 once it’s released.
If the sort refactor will take longer, I’m fine with approving the current PR in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3337230196
@armitage420, i would call this test broken (by some library version bump, etc), not flaky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3312037496
> The masking is only done for HDFS paths, file_size_in_bytes and total file size of table properties, the masking doesn't really effect the validation of the test.
@armitage420 test adds some additional masking as well, try removing and see for yourself. Why do we mask row count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix masking and re-enable query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3357733637
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3311973782
> i think masking in this specific test isn’t very effective, as it bypasses validation for several iceberg metadata fields query_iceberg_metadata_of_partitioned_table.q doesn't suffer from the same issue?
The masking is only done for HDFS paths, file_size_in_bytes and total file size of table properties, the masking doesn't really effect the validation of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3329208163
> Hey everyone, many thanks for working on this. Many recent runs are affected by this flaky test that makes our CI unstable. Since we haven't fully converged on the solution is it reasonable to disable this temporarily till the fix finally lands?
>
> PR for disabling the test: #6098
I hope we can hold off on the following PR for a few days, to get a conclusion on the recent comments if @deniskuzZ has the time! :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3297811277
I would rather not select columns that are going to be masked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3307189021
@thomasrebele Thank you for your input!
You are correct—the lexicographical sorting is done on unmasked values. Therefore, a better (and more accurate) fix would be to apply masking before sorting the results.
Currently, [sorting](https://github.com/apache/hive/blob/96d1635f98d0cfd5b0cd01115bb738e1c6053b94/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L286) is performed for every single query, whereas [masking](https://github.com/apache/hive/blob/96d1635f98d0cfd5b0cd01115bb738e1c6053b94/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L972) is only applied at the end, once we have collected all query results for the entire qfile. To implement the actual fix, we would need to change the test architecture so that masking is done per query, followed by sorting.
I'm not sure if this approach would be agreed upon, but if suggested, I can implement it!
@deniskuzZ Do let me know what you think!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3311869120
i think masking in this specific test isn’t very effective, as it bypasses validation for several iceberg metadata fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3323394112
> > The masking is only done for HDFS paths, file_size_in_bytes and total file size of table properties, the masking doesn't really effect the validation of the test.
>
> @armitage420 test adds some additional masking as well, try removing and see for yourself. Why do we mask row count instead of size_in_bytes? https://iceberg.apache.org/docs/1.9.0/spark-queries/#all-data-files
>
> ```
> 0 hdfs://### HDFS PATH ### ORC 0 5 378 {1:7,2:30}
>
> 0 hdfs://### HDFS PATH ### ORC 0 #Masked# 378 {1:7,2:30}
> ```
My bad, I have fixed the wrong masking related to file format. Although fixing this has no effect on the flaky nature of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3338171314
@armitage420: the QOutProcessor#maskPatterns applies the masking by copying the q.out file, reading the copy and overwriting the old file. Applying the masking on the string in memory will be much faster. I've created [HIVE-29227](https://issues.apache.org/jira/browse/HIVE-29227) to remove the masking from the post processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3329809852
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3329898713
> In general, if we cannot solve a flakiness issue reasonably fast we opt to disable the test first and fix it afterwards; that's quite common practice in the Hive project.
true, however, now we have tons of disabled flaky functional tests no one is looking at.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3306752720
I had a look at this flaky test, too. If you look at the expected [query result](https://github.com/apache/hive/blob/96d1635f98d0cfd5b0cd01115bb738e1c6053b94/iceberg/iceberg-handler/src/test/results/positive/query_iceberg_metadata_of_unpartitioned_table.q.out#L104), the first columns are the same, but the first different column is not sorted lexicographically:
```
0 hdfs://### HDFS PATH ### ORC 0 #Masked# 378 ...
0 hdfs://### HDFS PATH ### ORC 0 #Masked# 365 ...
0 hdfs://### HDFS PATH ### ORC 0 #Masked# 374 ...
```
The problem is that it is sorted on the original value of `### HDFS PATH ###`, and after sorting the value is replaced.
Changing the query to make this deterministic is a workaround for this particular q file. A proposal for a more general fix: refactor the [masking](https://github.com/apache/hive/blob/96d1635f98d0cfd5b0cd01115bb738e1c6053b94/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L972) so that it is done before the [sorting](https://github.com/apache/hive/blob/96d1635f98d0cfd5b0cd01115bb738e1c6053b94/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L286) (`out` is a `org.apache.hadoop.hive.common.io.SortPrintStream`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3308420060
I've been working on a draft of applying the masking before the sorting (in addition to applying the masking at the end of the processing) in https://github.com/thomasrebele/hive/tree/tr/HIVE-29201-v1. The design of `FetchConverter` makes it difficult to implement this cleanly. Alternatively, we could make `FetchConverter` an interface (and the old class would become `FetchConverterImpl`) to simplify the logic of LambdaFetchConverter. What do you think, @armitage420, @deniskuzZ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "armitage420 (via GitHub)" <gi...@apache.org>.
armitage420 commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3297975346
@deniskuzZ Not selecting masked columns is not feasible for this particular test, as the there are part of the column values are masked and not the column as a whole.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3355423176
UPD: rebased and fixed masking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3352710992
since [HIVE-29226](https://issues.apache.org/jira/browse/HIVE-29226) is merged, @armitage420 could you please remove the explicit sorting, but please keep the masking fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3303133309
> @deniskuzZ Not selecting masked columns is not feasible for this particular test, as the there are part of the column values are masked and not the column(related to metadata itself) as a whole.
oh, ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3325131796
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6075&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6075&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6075&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6075)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29201: Fix flaky test query_iceberg_metadata_of_unpartitioned_table.q [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #6075:
URL: https://github.com/apache/hive/pull/6075#issuecomment-3324917664
Hey everyone, many thanks for working on this. Many recent runs are affected by this flaky test that makes our CI unstable. Since we haven't fully converged on the solution is it reasonable to disable this temporarily till the fix finally lands?
PR for disabling the test: https://github.com/apache/hive/pull/6098
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org