You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/04/24 12:33:44 UTC

[GitHub] [shardingsphere] azexcy opened a new pull request, #25307: Refactor data record merge at pipeline

azexcy opened a new pull request, #25307:
URL: https://github.com/apache/shardingsphere/pull/25307

   Fixes #ISSUSE_ID.
   
   Changes proposed in this pull request:
     - Refactor data record merge at pipeline
     - Add more E2E log
     - Fix E2E error because of DataRecord hashcode generate method.
   
   ---
   
   Before committing this PR, I'm sure that I have checked the following options:
   - [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
   - [ ] I have self-reviewed the commit code.
   - [ ] I have (or in comment I request) added corresponding labels for the pull request.
   - [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
   - [ ] I have made corresponding changes to the documentation.
   - [ ] I have added corresponding unit tests for my changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz merged pull request #25307: Refactor data record merge at pipeline

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #25307:
URL: https://github.com/apache/shardingsphere/pull/25307


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #25307: Refactor data record merge at pipeline

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #25307:
URL: https://github.com/apache/shardingsphere/pull/25307#discussion_r1175246812


##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataRecordMerger.java:
##########
@@ -39,34 +35,21 @@
 public final class DataRecordMerger {
     
     /**
-     * Merge data record.
-     * <pre>
-     * insert + insert -&gt; exception
-     * update + insert -&gt; exception
-     * delete + insert -&gt; insert
-     * insert + update -&gt; insert
-     * update + update -&gt; update
-     * delete + update -&gt; exception
-     * insert + delete -&gt; delete
-     * update + delete -&gt; delete
-     * delete + delete -&gt; exception
-     * </pre>
+     * Merge data record if the data record has the same key.
      *
      * @param dataRecords data records
      * @return merged data records
      */
-    public List<DataRecord> merge(final List<DataRecord> dataRecords) {
-        Map<DataRecord.Key, DataRecord> result = new HashMap<>();
-        dataRecords.forEach(each -> {
-            if (IngestDataChangeType.INSERT.equals(each.getType())) {
-                mergeInsert(each, result);
-            } else if (IngestDataChangeType.UPDATE.equals(each.getType())) {
-                mergeUpdate(each, result);
-            } else if (IngestDataChangeType.DELETE.equals(each.getType())) {
-                mergeDelete(each, result);
+    public Map<DataRecord.Key, List<DataRecord>> merge(final List<DataRecord> dataRecords) {
+        Map<DataRecord.Key, List<DataRecord>> result = new LinkedHashMap<>();
+        for (DataRecord each : dataRecords) {
+            if (IngestDataChangeType.DELETE.equals(each.getType())) {
+                result.computeIfAbsent(each.getOldKey(), key -> new LinkedList<>()).add(each);
+                continue;
             }
-        });
-        return new ArrayList<>(result.values());
+            result.computeIfAbsent(each.getKey(), key -> new LinkedList<>()).add(each);
+        }
+        return result;
     }

Review Comment:
   1, Compatible without unique key table.
   
   2, Could we return `MergedDataRecords { List<DataRecord> batchDataRecords (3 fields: INSERT, UPDATE, DELETE); List<DataRecord> nonBatchRecords;`, the first ones could be executed in batch, and them could be execute in a transaction.
   To reduce upstream data copy.
   
   3, `DataRecord.Key` could be simplified to `Key`
   



##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataRecordMerger.java:
##########
@@ -77,89 +60,11 @@ public List<DataRecord> merge(final List<DataRecord> dataRecords) {
      */
     public List<GroupedDataRecord> group(final List<DataRecord> dataRecords) {
         List<GroupedDataRecord> result = new ArrayList<>(100);
-        List<DataRecord> mergedDataRecords = dataRecords.get(0).getUniqueKeyValue().isEmpty() ? dataRecords : merge(dataRecords);

Review Comment:
   Looks `dataRecords.get(0).getUniqueKeyValue().isEmpty()` is ignored



##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -242,7 +256,11 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
             for (DataRecord each : dataRecords) {
                 conditionColumns = RecordUtils.extractConditionColumns(each, importerConfig.getShardingColumns(each.getTableName()));
                 for (int i = 0; i < conditionColumns.size(); i++) {
-                    preparedStatement.setObject(i + 1, conditionColumns.get(i).getOldValue());
+                    Object oldValue = conditionColumns.get(i).getOldValue();
+                    if (null == oldValue) {
+                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                    }

Review Comment:
   Could we print `DataRecord each`? For better troubleshooting



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] azexcy commented on a diff in pull request #25307: Refactor data record merge at pipeline

Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #25307:
URL: https://github.com/apache/shardingsphere/pull/25307#discussion_r1177390978


##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -258,18 +242,46 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
                 for (int i = 0; i < conditionColumns.size(); i++) {
                     Object oldValue = conditionColumns.get(i).getOldValue();
                     if (null == oldValue) {
-                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                        log.warn("Record old value is null, record={}", each);
                     }
                     preparedStatement.setObject(i + 1, oldValue);
                 }
                 preparedStatement.addBatch();
             }
-            preparedStatement.executeBatch();
+            int[] executeBatch = preparedStatement.executeBatch();
+            if (executeBatch.length == 0) {
+                log.warn("executeDelete failed, deleteSQL={}, conditionColumns={}", deleteSQL, conditionColumns);
+            }
         } finally {
             batchDeleteStatement = null;
         }
     }
     
+    private void sequentialFlush(final DataSource dataSource, final List<DataRecord> buffer) {

Review Comment:
   I find unique key update, delete event not handle correctly now, i will fix it at next PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [shardingsphere] sandynz commented on a diff in pull request #25307: Refactor data record merge at pipeline

Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #25307:
URL: https://github.com/apache/shardingsphere/pull/25307#discussion_r1177292549


##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -258,18 +242,46 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
                 for (int i = 0; i < conditionColumns.size(); i++) {
                     Object oldValue = conditionColumns.get(i).getOldValue();
                     if (null == oldValue) {
-                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                        log.warn("Record old value is null, record={}", each);
                     }
                     preparedStatement.setObject(i + 1, oldValue);
                 }
                 preparedStatement.addBatch();
             }
-            preparedStatement.executeBatch();
+            int[] executeBatch = preparedStatement.executeBatch();
+            if (executeBatch.length == 0) {
+                log.warn("executeDelete failed, deleteSQL={}, conditionColumns={}", deleteSQL, conditionColumns);
+            }

Review Comment:
   Need to verify every execution result, but not `executeBatch.length == 0`



##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -258,18 +242,46 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
                 for (int i = 0; i < conditionColumns.size(); i++) {
                     Object oldValue = conditionColumns.get(i).getOldValue();
                     if (null == oldValue) {
-                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                        log.warn("Record old value is null, record={}", each);
                     }
                     preparedStatement.setObject(i + 1, oldValue);
                 }
                 preparedStatement.addBatch();
             }
-            preparedStatement.executeBatch();
+            int[] executeBatch = preparedStatement.executeBatch();
+            if (executeBatch.length == 0) {
+                log.warn("executeDelete failed, deleteSQL={}, conditionColumns={}", deleteSQL, conditionColumns);
+            }
         } finally {
             batchDeleteStatement = null;
         }
     }
     
+    private void sequentialFlush(final DataSource dataSource, final List<DataRecord> buffer) {
+        try (Connection connection = dataSource.getConnection()) {
+            // TODO it's better use transaction, but execute delete maybe not effect when open transaction of PostgreSQL
+            if (null != rateLimitAlgorithm) {
+                rateLimitAlgorithm.intercept(JobOperationType.INSERT, 1);
+            }

Review Comment:
   Could not use hard-coded `JobOperationType.INSERT`



##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -258,18 +242,46 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
                 for (int i = 0; i < conditionColumns.size(); i++) {
                     Object oldValue = conditionColumns.get(i).getOldValue();
                     if (null == oldValue) {
-                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                        log.warn("Record old value is null, record={}", each);
                     }
                     preparedStatement.setObject(i + 1, oldValue);
                 }
                 preparedStatement.addBatch();
             }
-            preparedStatement.executeBatch();
+            int[] executeBatch = preparedStatement.executeBatch();
+            if (executeBatch.length == 0) {
+                log.warn("executeDelete failed, deleteSQL={}, conditionColumns={}", deleteSQL, conditionColumns);
+            }
         } finally {
             batchDeleteStatement = null;
         }
     }
     
+    private void sequentialFlush(final DataSource dataSource, final List<DataRecord> buffer) {
+        try (Connection connection = dataSource.getConnection()) {
+            // TODO it's better use transaction, but execute delete maybe not effect when open transaction of PostgreSQL
+            if (null != rateLimitAlgorithm) {
+                rateLimitAlgorithm.intercept(JobOperationType.INSERT, 1);
+            }
+            for (DataRecord each : buffer) {
+                switch (each.getType()) {
+                    case IngestDataChangeType.INSERT:
+                        executeBatchInsert(connection, Collections.singletonList(each));
+                        break;
+                    case IngestDataChangeType.UPDATE:
+                        executeUpdate(connection, each);
+                        break;
+                    case IngestDataChangeType.DELETE:
+                        executeBatchDelete(connection, Collections.singletonList(each));
+                        break;
+                    default:
+                }
+            }
+        } catch (final SQLException ex) {
+            throw new PipelineImporterJobWriteException(ex);
+        }

Review Comment:
   It's better to add DataRecord in exception for better troubleshooting



##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/importer/DataSourceImporter.java:
##########
@@ -258,18 +242,46 @@ private void executeBatchDelete(final Connection connection, final List<DataReco
                 for (int i = 0; i < conditionColumns.size(); i++) {
                     Object oldValue = conditionColumns.get(i).getOldValue();
                     if (null == oldValue) {
-                        log.warn("Old value is null, column={}", conditionColumns.get(i));
+                        log.warn("Record old value is null, record={}", each);
                     }
                     preparedStatement.setObject(i + 1, oldValue);
                 }
                 preparedStatement.addBatch();
             }
-            preparedStatement.executeBatch();
+            int[] executeBatch = preparedStatement.executeBatch();
+            if (executeBatch.length == 0) {
+                log.warn("executeDelete failed, deleteSQL={}, conditionColumns={}", deleteSQL, conditionColumns);
+            }
         } finally {
             batchDeleteStatement = null;
         }
     }
     
+    private void sequentialFlush(final DataSource dataSource, final List<DataRecord> buffer) {

Review Comment:
   If there's no unique key, does it run through `sequentialFlush`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org