You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "thomasrebele (via GitHub)" <gi...@apache.org> on 2025/09/26 14:30:40 UTC
[PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
thomasrebele opened a new pull request, #6105:
URL: https://github.com/apache/hive/pull/6105
### What changes were proposed in this pull request?
When executing qfiles, the query result is first sorted and then masked (search and replace patterns to make the q.out files comparable). The problem: the order of the query results is non-deterministic, so the test may fail.
### Why are the changes needed?
Make the tests more stable and reduce flakiness.
### Does this PR introduce _any_ user-facing change?
The interface `FetchListener` should be preferred over `FetchConverter` wherever possible.
### How was this patch tested?
Executed qfiles locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388267350
##########
itests/util/src/main/java/org/apache/hive/beeline/ConvertedOutputFile.java:
##########
@@ -30,36 +30,36 @@
* FetchConverters.
*/
public class ConvertedOutputFile extends OutputFile {
- private final boolean isActiveFetchConverter;
+ private final boolean isFetchListener;
Review Comment:
minor: maybe `hasFetchCallback` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387505360
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
+
+ @Override
+ public void fetchFinished() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchFinished();
+ }
+ super.flush();
Review Comment:
can't you simply call `flush` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387473284
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
Review Comment:
The name is confusing and doesn’t convey the purpose of the class. A converter is inherently about transforming.
I my opinion QTestFetchConverter would be a better choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388242555
##########
common/src/java/org/apache/hadoop/hive/common/io/QTestFetchConverter.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class QTestFetchConverter extends SessionStream implements FetchCallback {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
Review Comment:
minor: maybe `hasFetchCallback`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2390514024
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:
##########
@@ -658,11 +658,25 @@ private void setSessionOutputs(CliSessionState ss, File outf) throws Exception {
qTestResultProcessor.setOutputs(ss, fo);
+ ss.out = new QTestFetchConverter(ss.out, false, "UTF-8", line -> {
+ notifyOutputLine(line);
+ if(qOutProcessor != null) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #6105:
URL: https://github.com/apache/hive/pull/6105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387469082
##########
common/src/java/org/apache/hadoop/hive/common/io/FetchListener.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common.io;
+
+/**
+ * Callers will use this interface to notify the class that processes the stream
+ * about the content that will be processed. mark the beginning/end
+ * of the output of an executed command, and whether
+ * the command has been a query.
+ */
+public interface FetchListener {
+ /**
+ * Marks the beginning of the output of an executed command.
+ */
+ void fetchStarted();
Review Comment:
maybe onStart, onFinish, onQueryDetected ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388246060
##########
common/src/java/org/apache/hadoop/hive/common/io/QTestFetchConverter.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class QTestFetchConverter extends SessionStream implements FetchCallback {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public QTestFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ps ? ps : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchCallback;
+ this.transformation = transformation;
+ }
+
+ @Override
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchCallback)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
Review Comment:
minor: please format the code, space is missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387483385
##########
common/src/java/org/apache/hadoop/hive/common/io/FetchListener.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common.io;
+
+/**
+ * Callers will use this interface to notify the class that processes the stream
+ * about the content that will be processed. mark the beginning/end
+ * of the output of an executed command, and whether
+ * the command has been a query.
+ */
+public interface FetchListener {
Review Comment:
maybe FetchCallback since foundQuery doesn't really fit the listener lifecycle-event model
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3347441689
@thomasrebele, did the test runtime increase from 2–2.5 hours to 3 hours?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3348934803
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2390512352
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java:
##########
@@ -58,41 +58,32 @@ public TestLocationQueries() {
*/
public static class CheckResults extends QTestUtil {
private final String locationSubdir;
+ private final List<String> outputLines = new ArrayList<>();
/**
* Validate only that the location is correct.
* @return non-zero if it failed
*/
@Override
public QTestProcessExecResult checkCliDriverResults() throws Exception {
- String tname = getInputFile().getName();
- File logFile = new File(logDir, tname + ".out");
-
int failedCount = 0;
StringBuilder fileNames = new StringBuilder("Files failing the location check:");
- FileReader fr = new FileReader(logFile);
- BufferedReader in = new BufferedReader(fr);
- try {
- String line;
- int locationCount = 0;
- Pattern p = Pattern.compile("location:([^,)]+)");
- while((line = in.readLine()) != null) {
- Matcher m = p.matcher(line);
- if (m.find()) {
- File f = new File(m.group(1));
- if (!f.getName().equals(locationSubdir)) {
- failedCount++;
- fileNames.append(f.getName()).append("\r\n");
- }
- locationCount++;
+ int locationCount = 0;
+ Pattern p = Pattern.compile("location:([^,)]+)");
+ for(String line : outputLines) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3350914829
@deniskuzZ, the runtime of the CI is not reliable. I've seen green builds from 2h to 7h. The last build for this PR is 2h 22m 8s, so it's within your expectations. Could you merge the PR if you're OK with the changes, please? I do not have write permission to the Hive repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2390509485
##########
common/src/java/org/apache/hadoop/hive/common/io/QTestFetchConverter.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class QTestFetchConverter extends SessionStream implements FetchCallback {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public QTestFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ps ? ps : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchCallback;
+ this.transformation = transformation;
+ }
+
+ @Override
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchCallback)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
Review Comment:
Sorry, I had forgotten to configure the save actions in my IDE to apply the formatting automatically. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388036948
##########
common/src/java/org/apache/hadoop/hive/common/io/FetchListener.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common.io;
+
+/**
+ * Callers will use this interface to notify the class that processes the stream
+ * about the content that will be processed. mark the beginning/end
+ * of the output of an executed command, and whether
+ * the command has been a query.
+ */
+public interface FetchListener {
Review Comment:
Fixed
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3347718887
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388260193
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java:
##########
@@ -58,41 +58,32 @@ public TestLocationQueries() {
*/
public static class CheckResults extends QTestUtil {
private final String locationSubdir;
+ private final List<String> outputLines = new ArrayList<>();
/**
* Validate only that the location is correct.
* @return non-zero if it failed
*/
@Override
public QTestProcessExecResult checkCliDriverResults() throws Exception {
- String tname = getInputFile().getName();
- File logFile = new File(logDir, tname + ".out");
-
int failedCount = 0;
StringBuilder fileNames = new StringBuilder("Files failing the location check:");
- FileReader fr = new FileReader(logFile);
- BufferedReader in = new BufferedReader(fr);
- try {
- String line;
- int locationCount = 0;
- Pattern p = Pattern.compile("location:([^,)]+)");
- while((line = in.readLine()) != null) {
- Matcher m = p.matcher(line);
- if (m.find()) {
- File f = new File(m.group(1));
- if (!f.getName().equals(locationSubdir)) {
- failedCount++;
- fileNames.append(f.getName()).append("\r\n");
- }
- locationCount++;
+ int locationCount = 0;
+ Pattern p = Pattern.compile("location:([^,)]+)");
+ for(String line : outputLines) {
Review Comment:
nit: space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3346337586
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2390515293
##########
itests/util/src/main/java/org/apache/hive/beeline/ConvertedOutputFile.java:
##########
@@ -30,36 +30,36 @@
* FetchConverters.
*/
public class ConvertedOutputFile extends OutputFile {
- private final boolean isActiveFetchConverter;
+ private final boolean isFetchListener;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387455144
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
Review Comment:
why create techdept when you can address it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3352699801
merged, @thomasrebele please keep an eye on the master build’s runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388037881
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3381468994
I've checked the last three runs of the master build (d46d900cba2b484f033772ec42e62e8f96e0c07a..f75c18b57ae4218db689d1e76fabb1343fd78973). The times were 2h 10m, 2h 22m, 2h 44m. I think that is the runtime is within the expected range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6105:
URL: https://github.com/apache/hive/pull/6105#issuecomment-3339297183
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6105&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6105&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6105&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6105)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387473284
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
Review Comment:
The name is confusing and doesn’t convey the purpose of the class. A converter is inherently about transforming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388036038
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
+
+ @Override
+ public void fetchFinished() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchFinished();
+ }
+ super.flush();
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387455144
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
Review Comment:
why create techdept when you can address it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387469082
##########
common/src/java/org/apache/hadoop/hive/common/io/FetchListener.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common.io;
+
+/**
+ * Callers will use this interface to notify the class that processes the stream
+ * about the content that will be processed. mark the beginning/end
+ * of the output of an executed command, and whether
+ * the command has been a query.
+ */
+public interface FetchListener {
+ /**
+ * Marks the beginning of the output of an executed command.
+ */
+ void fetchStarted();
Review Comment:
maybe onStart, onFinish, onQueryDetected ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2387377630
##########
common/src/java/org/apache/hadoop/hive/common/io/TransformingFetchConverter.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class TransformingFetchConverter extends SessionStream implements FetchListener {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
+
+ public TransformingFetchConverter(OutputStream out, boolean autoFlush, String encoding, UnaryOperator<String> transformation)
+ throws UnsupportedEncodingException {
+ super(out, autoFlush, encoding);
+ inner = out instanceof PrintStream ? (PrintStream) out : new PrintStream(out);
+ innerIsFetchConverter = out instanceof FetchListener;
+ this.transformation = transformation;
+ }
+
+ public void println(String str) {
+ inner.println(transformation.apply(str));
+ }
+
+ @Override
+ public void foundQuery(boolean queryfound) {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).foundQuery(queryfound);
+ }
+ }
+
+ @Override
+ public void fetchStarted() {
+ if(innerIsFetchConverter) {
+ ((FetchListener)inner).fetchStarted();
+ }
+ }
+
+ @Override
+ public void flush() {
+ super.flush();
+ }
Review Comment:
I'm going to remove this leftover from a refactoring in the next round.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2388261424
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java:
##########
@@ -658,11 +658,25 @@ private void setSessionOutputs(CliSessionState ss, File outf) throws Exception {
qTestResultProcessor.setOutputs(ss, fo);
+ ss.out = new QTestFetchConverter(ss.out, false, "UTF-8", line -> {
+ notifyOutputLine(line);
+ if(qOutProcessor != null) {
Review Comment:
nit: space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29226: Make order of qfile query results deterministic when masking and sorting [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6105:
URL: https://github.com/apache/hive/pull/6105#discussion_r2390505847
##########
common/src/java/org/apache/hadoop/hive/common/io/QTestFetchConverter.java:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common.io;
+
+import java.io.OutputStream;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.util.function.UnaryOperator;
+
+/**
+ * Applies a function to the processed lines, before passing it to the wrapped output stream.
+ */
+public class QTestFetchConverter extends SessionStream implements FetchCallback {
+
+ private final UnaryOperator<String> transformation;
+
+ private final PrintStream inner;
+ private final boolean innerIsFetchConverter;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org