You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by GitBox <gi...@apache.org> on 2022/11/20 07:22:25 UTC

[GitHub] [inlong] gong opened a new pull request, #6577: [INLONG-6574][Sort] Add dirty data metric for HBase

gong opened a new pull request, #6577:
URL: https://github.com/apache/inlong/pull/6577

   ### Prepare a Pull Request
   
   - [INLONG-6574][Sort] Add dirty data metric for HBase
   
   - Fixes #6574 
   
   ### Motivation
   
   * Add dirty data metric for HBase
   
   ### Modifications
   
   * Modify base of sort-connector module, add `dirtyRecordsOut` `dirtyBytesOut`.
   * Modify HBase connector for metric computing, exception just print and count dirtyData. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] gong commented on a diff in pull request #6577: [INLONG-6574][Sort] Add dirty data metric for HBase

Posted by GitBox <gi...@apache.org>.
gong commented on code in PR #6577:
URL: https://github.com/apache/inlong/pull/6577#discussion_r1028908828


##########
inlong-sort/sort-connectors/hbase/src/main/java/org/apache/inlong/sort/hbase/sink/HBaseSinkFunction.java:
##########
@@ -225,34 +216,48 @@ private org.apache.hadoop.conf.Configuration prepareRuntimeConfiguration() throw
     private void checkErrorAndRethrow() {
         Throwable cause = failureThrowable.get();
         if (cause != null) {
-            throw new RuntimeException("An error occurred in HBaseSink.", cause);
+            LOG.error("An error occurred in HBaseSink.", cause);
+            failureThrowable.compareAndSet(cause, null);
         }
     }
 
     @SuppressWarnings("rawtypes")
     @Override
-    public void invoke(T value, Context context) throws Exception {
+    public void invoke(T value, Context context) {
         checkErrorAndRethrow();
+        try {
+            mutator.mutate(mutationConverter.convertToMutation(value));
+            rowSize++;
+            dataSize = dataSize + value.toString().getBytes(StandardCharsets.UTF_8).length;
+        } catch (Exception e) {
+            if (sinkMetricData != null) {
+                sinkMetricData.invokeDirty(1, value.toString().getBytes(StandardCharsets.UTF_8).length);

Review Comment:
   other PR add this throw exception flag to handle it 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] yunqingmoswu commented on a diff in pull request #6577: [INLONG-6574][Sort] Add dirty data metric for HBase

Posted by GitBox <gi...@apache.org>.
yunqingmoswu commented on code in PR #6577:
URL: https://github.com/apache/inlong/pull/6577#discussion_r1028902104


##########
inlong-sort/sort-connectors/hbase/src/main/java/org/apache/inlong/sort/hbase/sink/HBaseSinkFunction.java:
##########
@@ -225,34 +216,48 @@ private org.apache.hadoop.conf.Configuration prepareRuntimeConfiguration() throw
     private void checkErrorAndRethrow() {
         Throwable cause = failureThrowable.get();
         if (cause != null) {
-            throw new RuntimeException("An error occurred in HBaseSink.", cause);
+            LOG.error("An error occurred in HBaseSink.", cause);
+            failureThrowable.compareAndSet(cause, null);
         }
     }
 
     @SuppressWarnings("rawtypes")
     @Override
-    public void invoke(T value, Context context) throws Exception {
+    public void invoke(T value, Context context) {
         checkErrorAndRethrow();
+        try {
+            mutator.mutate(mutationConverter.convertToMutation(value));
+            rowSize++;
+            dataSize = dataSize + value.toString().getBytes(StandardCharsets.UTF_8).length;
+        } catch (Exception e) {
+            if (sinkMetricData != null) {
+                sinkMetricData.invokeDirty(1, value.toString().getBytes(StandardCharsets.UTF_8).length);

Review Comment:
   It is necessary to decide whether to throw an exception based on whether dirty data is tolerated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [inlong] dockerzhang merged pull request #6577: [INLONG-6574][Sort] Add dirty data metric for HBase

Posted by GitBox <gi...@apache.org>.
dockerzhang merged PR #6577:
URL: https://github.com/apache/inlong/pull/6577


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org