You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Viktor Csomor (Jira)" <ji...@apache.org> on 2021/11/29 15:00:00 UTC
[jira] [Updated] (HIVE-25746) Compaction Failure Counter counted incorrectly
[ https://issues.apache.org/jira/browse/HIVE-25746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Viktor Csomor updated HIVE-25746:
---------------------------------
Description:
The count of the below metrics counted incorrectly upon an exception.
- {{compaction_initator_failure_counter}}
- {{compaction_cleaner_failure_counter}}
Reasoning:
In the {{Initator}}/{{Cleaner}} class creates a list of {{CompletableFuture}} which {{Runnable}} core exception is being wrapped to {{RuntimeExceptions}}.
The below code-snippet waits all cleaners to complete (Initiators does it similarly).
{code:java}
try {
....
for (CompactionInfo compactionInfo : readyToClean) {
cleanerList.add(CompletableFuture.runAsync(CompactorUtil.ThrowingRunnable.unchecked(() ->
clean(compactionInfo, cleanerWaterMark, metricsEnabled)), cleanerExecutor));
}
CompletableFuture.allOf(cleanerList.toArray(new CompletableFuture[0])).join();
}
} catch (Throwable t) {
// the lock timeout on AUX lock, should be ignored.
if (metricsEnabled && handle != null) {
failuresCounter.inc();
}
{code}
If the {{CompleteableFututre#join}} throws an Exception then the failure counter is incremented.
Docs:
{code}
/**
* Returns the result value when complete, or throws an
* (unchecked) exception if completed exceptionally. To better
* conform with the use of common functional forms, if a
* computation involved in the completion of this
* CompletableFuture threw an exception, this method throws an
* (unchecked) {@link CompletionException} with the underlying
* exception as its cause.
*
* @return the result value
* @throws CancellationException if the computation was cancelled
* @throws CompletionException if this future completed
* exceptionally or a completion computation threw an exception
*/
public T join() {
Object r;
return reportJoin((r = result) == null ? waitingGet(false) : r);
}
{code}
(!) Let's suppose we have 10 cleaners and the 2nd throws an exception. The {{catch}} block will be initiated and the {{failuresCounter}} will be incremented. If there is any consecutive error amongst the remaining cleaners the counter won't be incremented.
was:
The count of the below metrics counted incorrectly upon an exception.
- {{compaction_initator_failure_counter}}
- {{compaction_cleaner_failure_counter}}
Reasoning:
In the {{Initator}}/{{Cleaner}} class creates a list of {{CompletableFuture}} which {{Runnable}} core exception is being wrapped to {{RuntimeExceptions}}.
The below code-snippet waits all cleaners to complete (Initiators does it similarly).
{code:java}
try {
....
for (CompactionInfo compactionInfo : readyToClean) {
cleanerList.add(CompletableFuture.runAsync(CompactorUtil.ThrowingRunnable.unchecked(() ->
clean(compactionInfo, cleanerWaterMark, metricsEnabled)), cleanerExecutor));
}
CompletableFuture.allOf(cleanerList.toArray(new CompletableFuture[0])).join();
}
} catch (Throwable t) {
// the lock timeout on AUX lock, should be ignored.
if (metricsEnabled && handle != null) {
failuresCounter.inc();
}
{code}
If the {{CompleteableFututre#join}} throws an Exception then the failure counter is incremented.
Let's suppose we have 10 cleaners and the 2nd throws an exception. The {{catch}} block will be initiated and the {{failuresCounter}} will be incremented. If there is any consecutive error amongst the remaining cleaners the counter won't be incremented.
> Compaction Failure Counter counted incorrectly
> ----------------------------------------------
>
> Key: HIVE-25746
> URL: https://issues.apache.org/jira/browse/HIVE-25746
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 4.0.0
> Reporter: Viktor Csomor
> Assignee: Viktor Csomor
> Priority: Minor
>
> The count of the below metrics counted incorrectly upon an exception.
> - {{compaction_initator_failure_counter}}
> - {{compaction_cleaner_failure_counter}}
> Reasoning:
> In the {{Initator}}/{{Cleaner}} class creates a list of {{CompletableFuture}} which {{Runnable}} core exception is being wrapped to {{RuntimeExceptions}}.
> The below code-snippet waits all cleaners to complete (Initiators does it similarly).
> {code:java}
> try {
> ....
> for (CompactionInfo compactionInfo : readyToClean) {
> cleanerList.add(CompletableFuture.runAsync(CompactorUtil.ThrowingRunnable.unchecked(() ->
> clean(compactionInfo, cleanerWaterMark, metricsEnabled)), cleanerExecutor));
> }
> CompletableFuture.allOf(cleanerList.toArray(new CompletableFuture[0])).join();
> }
> } catch (Throwable t) {
> // the lock timeout on AUX lock, should be ignored.
> if (metricsEnabled && handle != null) {
> failuresCounter.inc();
> }
> {code}
> If the {{CompleteableFututre#join}} throws an Exception then the failure counter is incremented.
> Docs:
> {code}
> /**
> * Returns the result value when complete, or throws an
> * (unchecked) exception if completed exceptionally. To better
> * conform with the use of common functional forms, if a
> * computation involved in the completion of this
> * CompletableFuture threw an exception, this method throws an
> * (unchecked) {@link CompletionException} with the underlying
> * exception as its cause.
> *
> * @return the result value
> * @throws CancellationException if the computation was cancelled
> * @throws CompletionException if this future completed
> * exceptionally or a completion computation threw an exception
> */
> public T join() {
> Object r;
> return reportJoin((r = result) == null ? waitingGet(false) : r);
> }
> {code}
> (!) Let's suppose we have 10 cleaners and the 2nd throws an exception. The {{catch}} block will be initiated and the {{failuresCounter}} will be incremented. If there is any consecutive error amongst the remaining cleaners the counter won't be incremented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)