You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "schnappi17 (via GitHub)" <gi...@apache.org> on 2023/05/27 18:03:01 UTC

[GitHub] [incubator-paimon] schnappi17 opened a new pull request, #1247: Paimon 1050

schnappi17 opened a new pull request, #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247

   ### Purpose
   
   fix #1050 
   
   Taking `System.nanotime()` as the time source with high precision, make up to the provided `sequence.field` to nanosecond if it's in`Timestamp` data type and with the precision of second or millisecond.
   
   ### Tests
   
   ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampSecond
   ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampMilliSecond
   ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampMicroSecond
   ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnNonTimestampField
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] lppsuixn commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "lppsuixn (via GitHub)" <gi...@apache.org>.
lppsuixn commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1232243474


##########
paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java:
##########
@@ -154,6 +161,219 @@ public void testSequenceNumber() throws Exception {
                                 "1|11|101|binary|varbinary|mapKey:mapVal|multiset"));
     }
 
+    @Test
+    public void testPaddingSequenceNumberOnTimestampSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm0");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.SECOND_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        long sec = ts.getMillisecond() / 1000;
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =
+                    "1|10|100|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a1";
+        }
+        List<Split> splits = toSplits(table.newSnapshotSplitReader().splits());
+        TableRead read = table.newRead();
+        assertThat(getResult(read, splits, binaryRow(1), 0, ROW_WITH_TIMESTAMP_TO_STRING))
+                .isEqualTo(Arrays.asList(expectedResult));
+    }
+
+    @Test
+    public void testPaddingSequenceNumberOnTimestampMilliSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm3");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.MILLIS_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =

Review Comment:
   Currently, when two records have the same sequence number, the latter one is not the final result. #1050  is aimed at resolving this issue.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1212531896


##########
paimon-core/src/main/java/org/apache/paimon/table/sink/SequenceGenerator.java:
##########
@@ -68,6 +73,32 @@ public long generate(InternalRow row) {
         return generator.generate(row, index);
     }
 
+    public long generateWithPadding(InternalRow row, CoreOptions.SequenceAutoPadding autoPadding) {

Review Comment:
   I think we can try to finish this in a simpler way:
   ```
   public long generateWithPadding(InternalRow row, CoreOptions.SequenceAutoPadding autoPadding) {
       switch (autoPadding) {
               case SECOND_TO_MICRO:
                   long value = generate(row); 
                   // timestamp returns mills
                   int second = fieldType.is(DataTypeFamily.TIMESTAMP) ? value / 1000 : value;
                   return secondToMicro(second); 
               case MILLIS_TO_MICRO:
                   return millsToMicro(generate(row)); 
               default:
                   throw new UnsupportedOperationException(
                           "Unknown sequence padding mode " + autoPadding.name());
           }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] schnappi17 commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "schnappi17 (via GitHub)" <gi...@apache.org>.
schnappi17 commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1570686015

   @JingsongLi Updated, thanks for you suggestions~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] lppsuixn commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "lppsuixn (via GitHub)" <gi...@apache.org>.
lppsuixn commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1231918016


##########
paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java:
##########
@@ -154,6 +161,219 @@ public void testSequenceNumber() throws Exception {
                                 "1|11|101|binary|varbinary|mapKey:mapVal|multiset"));
     }
 
+    @Test
+    public void testPaddingSequenceNumberOnTimestampSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm0");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.SECOND_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        long sec = ts.getMillisecond() / 1000;
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =
+                    "1|10|100|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a1";
+        }
+        List<Split> splits = toSplits(table.newSnapshotSplitReader().splits());
+        TableRead read = table.newRead();
+        assertThat(getResult(read, splits, binaryRow(1), 0, ROW_WITH_TIMESTAMP_TO_STRING))
+                .isEqualTo(Arrays.asList(expectedResult));
+    }
+
+    @Test
+    public void testPaddingSequenceNumberOnTimestampMilliSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm3");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.MILLIS_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =

Review Comment:
   Based on   the situation mentioned in #1050, A2 should always be the final result.
   I mean ,the last record should always  as final result.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi merged pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1212532885


##########
docs/content/concepts/primary-key-table.md:
##########
@@ -260,6 +260,8 @@ there will be some cases that lead to data disorder. At this time, you can use a
 {{< hint info >}}
 When the record is updated or deleted, the `sequence.field` must become larger and cannot remain unchanged. For example,
 you can use [Mysql Binlog operation time](https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#available-metadata) as `sequence.field`.
+If the provided `sequence.field` doesn't meet the precision, like a rough second or millisecond, you can set `sequence.auto-padding` to `second-to-micro` or `millis-to-micro` so that the precision of sequence number will be made up to microsecond by system. 
+Note that only fields of data type `Timestamp`, `Integer` and `BigInt` which indicates a `second` or `millisecond` can be padded by microseconds.

Review Comment:
   We can apply this padding to al types.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] schnappi17 commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "schnappi17 (via GitHub)" <gi...@apache.org>.
schnappi17 commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1573697944

   @JingsongLi Thanks, your suggestion simplify the implementation for sure, I updated in the latest commit. Why we need the `writeAndReturnData` is because we can only get the generated sequence number from the origin `KeyValue` but not `SinkRecord`. We need to use the generated sequence number to do comparation and make the assertions. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1233536721


##########
paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java:
##########
@@ -154,6 +161,219 @@ public void testSequenceNumber() throws Exception {
                                 "1|11|101|binary|varbinary|mapKey:mapVal|multiset"));
     }
 
+    @Test
+    public void testPaddingSequenceNumberOnTimestampSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm0");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.SECOND_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        long sec = ts.getMillisecond() / 1000;
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =
+                    "1|10|100|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a1";
+        }
+        List<Split> splits = toSplits(table.newSnapshotSplitReader().splits());
+        TableRead read = table.newRead();
+        assertThat(getResult(read, splits, binaryRow(1), 0, ROW_WITH_TIMESTAMP_TO_STRING))
+                .isEqualTo(Arrays.asList(expectedResult));
+    }
+
+    @Test
+    public void testPaddingSequenceNumberOnTimestampMilliSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm3");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.MILLIS_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =

Review Comment:
   @lppsuixn In real world, the current implementation is possible because it is difficult to have two data with the same key in one microsecond at the same time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1568116365

   Or we can introduce a 'sequence.auto-padding' key, valid values are 'second-to-micro' and 'mills-to-micro' and 'none'. Default is none.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] schnappi17 commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "schnappi17 (via GitHub)" <gi...@apache.org>.
schnappi17 commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1232110180


##########
paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java:
##########
@@ -154,6 +161,219 @@ public void testSequenceNumber() throws Exception {
                                 "1|11|101|binary|varbinary|mapKey:mapVal|multiset"));
     }
 
+    @Test
+    public void testPaddingSequenceNumberOnTimestampSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm0");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.SECOND_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        long sec = ts.getMillisecond() / 1000;
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =
+                    "1|10|100|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a1";
+        }
+        List<Split> splits = toSplits(table.newSnapshotSplitReader().splits());
+        TableRead read = table.newRead();
+        assertThat(getResult(read, splits, binaryRow(1), 0, ROW_WITH_TIMESTAMP_TO_STRING))
+                .isEqualTo(Arrays.asList(expectedResult));
+    }
+
+    @Test
+    public void testPaddingSequenceNumberOnTimestampMilliSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm3");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.MILLIS_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =

Review Comment:
   > Based on the situation mentioned in #1050, a2 should always be the final result.
   
   @lppsuixn In this situation, if you just set the `sequence.field` without auto-padding, the last record is the final result as expected. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1222471272


##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableWriteImpl.java:
##########
@@ -102,6 +103,20 @@ public SinkRecord writeAndReturn(InternalRow row) throws Exception {
         return record;
     }
 
+    @VisibleForTesting
+    public T writeAndReturnData(InternalRow row) throws Exception {
+        keyAndBucketExtractor.setRecord(row);
+        SinkRecord record =

Review Comment:
   you can create a method: `toSinkRecord` to reuse code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] lppsuixn commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "lppsuixn (via GitHub)" <gi...@apache.org>.
lppsuixn commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1670893163

   > In the current implementation, it's possible for the "padding" of later data's sequence to be smaller than that of the earlier data,resulting in data errors. for example: seconds->micros currentNanoTime1:1062154,766334,200 currentSecondsTime1:1062154 currentNanoTime2:1062155,676231,100 currentSecondsTime2:1062155 sourceMillis:1691482327000 result_sequence:1691482327,766334 and 1691482327,676231(smaller)
   > 
   > Possible solutions? set an incremental sequence for the sequence.field, incrementing by one for each incoming data?
   
   You are right. This PR does not solve the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] lppsuixn commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "lppsuixn (via GitHub)" <gi...@apache.org>.
lppsuixn commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1231918016


##########
paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java:
##########
@@ -154,6 +161,219 @@ public void testSequenceNumber() throws Exception {
                                 "1|11|101|binary|varbinary|mapKey:mapVal|multiset"));
     }
 
+    @Test
+    public void testPaddingSequenceNumberOnTimestampSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm0");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.SECOND_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        long sec = ts.getMillisecond() / 1000;
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(sec, TimeUnit.SECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =
+                    "1|10|100|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a1";
+        }
+        List<Split> splits = toSplits(table.newSnapshotSplitReader().splits());
+        TableRead read = table.newRead();
+        assertThat(getResult(read, splits, binaryRow(1), 0, ROW_WITH_TIMESTAMP_TO_STRING))
+                .isEqualTo(Arrays.asList(expectedResult));
+    }
+
+    @Test
+    public void testPaddingSequenceNumberOnTimestampMilliSecond() throws Exception {
+        Timestamp ts =
+                Timestamp.fromLocalDateTime(LocalDateTime.of(2023, 5, 23, 11, 22, 33, 123456000));
+        List<InternalRow> rows = prepareSequencePaddingRows(ts, 1685530987, 1685530987123L);
+        FileStoreTable table =
+                createFileStoreTable(
+                        conf -> {
+                            conf.set(CoreOptions.SEQUENCE_FIELD, "tm3");
+                            conf.set(
+                                    CoreOptions.SEQUENCE_AUTO_PADDING,
+                                    CoreOptions.SequenceAutoPadding.MILLIS_TO_MICRO);
+                        },
+                        ROW_TYPE_WITH_TIMESTAMP);
+        StreamTableWrite write = table.newWrite(commitUser);
+        StreamTableCommit commit = table.newCommit(commitUser);
+        long sequenceNumber1 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(0)).sequenceNumber();
+        long sequenceNumber2 =
+                ((TableWriteImpl<KeyValue>) write).writeAndReturnData(rows.get(1)).sequenceNumber();
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber1, TimeUnit.MICROSECONDS));
+        assertEquals(
+                ts.getMillisecond(),
+                TimeUnit.MILLISECONDS.convert(sequenceNumber2, TimeUnit.MICROSECONDS));
+        commit.commit(0, write.prepareCommit(true, 0));
+        write.close();
+        String expectedResult;
+        if (sequenceNumber2 > sequenceNumber1) {
+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =

Review Comment:
   Based on   the situation mentioned in #1050, a2 should always be the final result.
   I mean ,the last record should always  as final result.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#discussion_r1212532641


##########
paimon-core/src/main/java/org/apache/paimon/table/sink/TableWriteImpl.java:
##########
@@ -102,6 +103,20 @@ public SinkRecord writeAndReturn(InternalRow row) throws Exception {
         return record;
     }
 
+    @VisibleForTesting
+    public T writeAndReturnData(InternalRow row) throws Exception {

Review Comment:
   why not using `writeAndReturn` and get row from `SinkRecord`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1581913568

   @schnappi17 Please ensure test passing first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] schnappi17 commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "schnappi17 (via GitHub)" <gi...@apache.org>.
schnappi17 commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1590997867

   @JingsongLi Updated, and I didn't see any test failed, maybe it's confused by another pr  of mine that UT failed, please help to approve the workflows and let's check it, thanks a lot!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] hzjhjjyy commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "hzjhjjyy (via GitHub)" <gi...@apache.org>.
hzjhjjyy commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1670885102

   In the current implementation, it's possible for the "padding" of later data's sequence to be smaller than that of the earlier data,resulting in data errors.
   for example:
   seconds->micros
   currentNanoTime1:1062154,766334,200
   currentSecondsTime1:1062154
   currentNanoTime2:1062155,676231,100
   currentSecondsTime2:1062155
   sourceMillis:1691482327000
   result_sequence:1691482327,766334 and 1691482327,676231(smaller)
   
   Possible solutions?
   set an incremental sequence for the sequence.field, incrementing by one for each incoming data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] lppsuixn commented on pull request #1247: [core] Support to make up the sequence number to nanosecond precision

Posted by "lppsuixn (via GitHub)" <gi...@apache.org>.
lppsuixn commented on PR #1247:
URL: https://github.com/apache/incubator-paimon/pull/1247#issuecomment-1670892535

   > 
   You are right. This PR does not solve the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org