You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "czy006 (via GitHub)" <gi...@apache.org> on 2024/07/31 10:33:16 UTC

[PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

czy006 opened a new pull request, #3074:
URL: https://github.com/apache/amoro/pull/3074

   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
     2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
   -->
   
   ## Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about its use case.
     2. If you fix a bug, you can clarify why it is a bug.
     3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
   -->
   
   Close #3073.
   
   ## Brief change log
   <!--
   Clearly describe the changes made in modules, classes, methods, etc.
   -->
   
   - Add Paimon Consumer AmoroSnapshotsOfTable of previous Snapshot Id Info
   
   ## How was this patch tested?
   
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [x] Add screenshots for manual tests if appropriate
   
   - [x] Run test locally before making a pull request
   
   ## Documentation
   
   - Does this pull request introduce a new feature? (no)
   - If yes, how is the feature documented? (not documented)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1699283033


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,19 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                Snapshot snapshot = table.snapshotManager().snapshot(nextSnapshotId - 1);

Review Comment:
   If first snapshot is < 0, may be have exp. Flink Checkpoint start id is 0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "huyuanfeng2018 (via GitHub)" <gi...@apache.org>.
huyuanfeng2018 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1699994910


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,23 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                long currentSnapshotId = nextSnapshotId - 1;

Review Comment:
   If can ensure that nextSnapshotId >0, i think don’t need this judgment.
   



##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/model/ConsumerInfo.java:
##########
@@ -48,4 +48,12 @@ public long getNextSnapshotId() {
   public void setNextSnapshotId(long nextSnapshotId) {

Review Comment:
   We don’t seem to need this method. In my opinion, the member variables of this class can be marked as final, WDYT?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "huyuanfeng2018 (via GitHub)" <gi...@apache.org>.
huyuanfeng2018 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1700042833


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,23 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                long currentSnapshotId = nextSnapshotId - 1;

Review Comment:
   If paimon itself can already determine that this is >1, do we not need to judge anymore? If we add judgment, if there is an exception, the stack info will be lost.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #3074:
URL: https://github.com/apache/amoro/pull/3074#issuecomment-2260450433

   #3054 UI with Paimon Consumer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1701100168


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,23 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                long currentSnapshotId = nextSnapshotId - 1;

Review Comment:
   I read this part of the read code, the underlying exception catch, but did not add the SnapshotId judgment range



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 merged PR #3074:
URL: https://github.com/apache/amoro/pull/3074


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "huyuanfeng2018 (via GitHub)" <gi...@apache.org>.
huyuanfeng2018 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1698463525


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,19 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                Snapshot snapshot = table.snapshotManager().snapshot(nextSnapshotId - 1);

Review Comment:
   Will there be any exception when nextSnapshotId=1?
   
   



##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/model/ConsumerInfo.java:
##########
@@ -48,4 +48,13 @@ public long getNextSnapshotId() {
   public void setNextSnapshotId(long nextSnapshotId) {
     this.nextSnapshotId = nextSnapshotId;
   }
+
+  public AmoroSnapshotsOfTable getAmoroPreviousSnapshotsOfTable() {

Review Comment:
   I don’t quite understand why the return the previous snapshot information . Can you give some examples? Why is the current snapshot information not displayed, but the previous one?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #3074:
URL: https://github.com/apache/amoro/pull/3074#issuecomment-2264736772

   @zhoujinsong @huyuanfeng2018 After communicate with the **Apache Paimon community**, there is no interface for querying "current" consumption information in this part. We can only query NextSnapshotId-1 as some information of the current consumption Snapshot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1701077062


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,23 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                long currentSnapshotId = nextSnapshotId - 1;

Review Comment:
   ![image](https://github.com/user-attachments/assets/3693c54b-1179-410f-b309-0ae0f24127b7)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #3074:
URL: https://github.com/apache/amoro/pull/3074#issuecomment-2265068716

   ![image](https://github.com/user-attachments/assets/34179dd3-6749-49a0-9255-084ecba0ac53)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on PR #3074:
URL: https://github.com/apache/amoro/pull/3074#issuecomment-2260202951

   ![image](https://github.com/user-attachments/assets/a4e1e75e-857a-4b7d-92f8-2be5b13dfa53)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1699285774


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/model/ConsumerInfo.java:
##########
@@ -48,4 +48,13 @@ public long getNextSnapshotId() {
   public void setNextSnapshotId(long nextSnapshotId) {
     this.nextSnapshotId = nextSnapshotId;
   }
+
+  public AmoroSnapshotsOfTable getAmoroPreviousSnapshotsOfTable() {

Review Comment:
   I misnamed the description here. nextSnapshotId-1 is the current Snapshot consumption information



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1700015943


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +525,23 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
     ConsumerManager consumerManager = new ConsumerManager(table.fileIO(), table.location());
     List<ConsumerInfo> consumerInfos = new ArrayList<>();
     try {
       consumerManager
           .consumers()
           .forEach(
-              (consumerId, nextSnapshot) ->
-                  consumerInfos.add(new ConsumerInfo(consumerId, nextSnapshot)));
+              (consumerId, nextSnapshotId) -> {
+                long currentSnapshotId = nextSnapshotId - 1;

Review Comment:
   if first snapshot id from flink checkpoint is 1, we need judgment nextSnapshotId > 1,because of get consumer is nextSnapshotId
   
   ![image](https://github.com/user-attachments/assets/fbd52799-4c79-4a44-83bb-0a93802f2e71)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-3073]: Paimon Consumer adds the previous Snapshot Id Info (#3073) [amoro]

Posted by "huyuanfeng2018 (via GitHub)" <gi...@apache.org>.
huyuanfeng2018 commented on code in PR #3074:
URL: https://github.com/apache/amoro/pull/3074#discussion_r1701829153


##########
amoro-ams/amoro-ams-server/src/main/java/org/apache/amoro/server/dashboard/PaimonTableDescriptor.java:
##########
@@ -525,14 +526,26 @@ public List<TagOrBranchInfo> getTableBranches(AmoroTable<?> amoroTable) {
   @Override
   public List<ConsumerInfo> getTableConsumerInfos(AmoroTable<?> amoroTable) {
     FileStoreTable table = getTable(amoroTable);
+    FileStore<?> store = table.store();
+    CoreOptions options = new CoreOptions(table.options());
+    LookupStrategy lookupStrategy = options.lookupStrategy();

Review Comment:
   It seems a bit redundant here.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org