You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Chu Xue (Jira)" <ji...@apache.org> on 2025/08/29 05:46:00 UTC
[jira] [Created] (FLINK-38294) Hbase connector misclassifies failed task as successful
Chu Xue created FLINK-38294:
-------------------------------
Summary: Hbase connector misclassifies failed task as successful
Key: FLINK-38294
URL: https://issues.apache.org/jira/browse/FLINK-38294
Project: Flink
Issue Type: Bug
Components: Connectors / HBase
Affects Versions: 1.20.2, 1.19.3, 1.18.1
Reporter: Chu Xue
Attachments: jobmanager.log, taskmanager.log
The Hbase connector will not flush when write less than 1000(sink.buffer-flush.max-rows) records or execute in less than 1s(sink.buffer-flush.interval).Org.apache.hadoop.hbase.client.BufferedMutator excute flush when close and flush maybe failed.
{code:java}
if (bufferFlushIntervalMillis > 0 && bufferFlushMaxMutations != 1) {
this.executor =
Executors.newScheduledThreadPool(
1, new ExecutorThreadFactory("hbase-upsert-sink-flusher"));
this.scheduledFuture =
this.executor.scheduleWithFixedDelay(
() -> {
if (closed) {
return;
}
try {
flush();
} catch (Exception e) {
// fail the sink and skip the rest of the items
// if the failure handler decides to throw an exception
failureThrowable.compareAndSet(null, e);
}
},
bufferFlushIntervalMillis,
bufferFlushIntervalMillis,
TimeUnit.MILLISECONDS);
} {code}
{code:java}
@SuppressWarnings("rawtypes")@Overridepublic void invoke(T value, Context context) throws Exception { checkErrorAndRethrow();
mutator.mutate(mutationConverter.convertToMutation(value));
// flush when the buffer number of mutations greater than the configured max size. if (bufferFlushMaxMutations > 0 && numPendingRequests.incrementAndGet() >= bufferFlushMaxMutations) { flush(); } else if (bufferFlushMaxMutations == 0 && bufferFlushMaxSizeInBytes == 0) { flush(); }}
{code}
{code:java}
//代码占位符
@Overridepublic void close() throws Exception { closed = true;
if (mutator != null) { try { mutator.close(); } catch (IOException e) { LOG.warn("Exception occurs while closing HBase BufferedMutator.", e); } this.mutator = null; }
if (connection != null) { try { connection.close(); } catch (IOException e) { LOG.warn("Exception occurs while closing HBase Connection.", e); } this.connection = null; }
if (scheduledFuture != null) { scheduledFuture.cancel(false); if (executor != null) { executor.shutdownNow(); } }} {code}
For example, creating a permission denial case where the user does not have permission(ranger) for the hbase namespace.The task will failed ,but return success.
[^jobmanager.log]
[^taskmanager.log]
Modify org.apache.flink.connector.hbase.sink.HBaseSinkFunction#close like this, throw the error.
{code:java}
@Overridepublic void close() throws Exception { closed = true;
if (mutator != null) { try { mutator.close(); } catch (IOException e) { LOG.warn("Exception occurs while closing HBase BufferedMutator.", e); } this.mutator = null; }
if (connection != null) { try { connection.close(); } catch (IOException e) { LOG.warn("Exception occurs while closing HBase Connection.", e); } this.connection = null; }
if (scheduledFuture != null) { scheduledFuture.cancel(false); if (executor != null) { executor.shutdownNow(); } } checkErrorAndRethrow();} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)