You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ThinkerLei (via GitHub)" <gi...@apache.org> on 2023/04/11 04:57:13 UTC
[GitHub] [hudi] ThinkerLei opened a new issue, #8425: [SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
ThinkerLei opened a new issue, #8425:
URL: https://github.com/apache/hudi/issues/8425
Now, we save the startOffset written to each logfile for this deltacommit. Can we use this data to reduce read amplification when downstream tasks read logfiles?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8425: [SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8425:
URL: https://github.com/apache/hudi/issues/8425#issuecomment-1506420257
Nice idea, currently the reader is assigned with granularity of inputsplits (aka to file slices), need to think through how to manager the offset especially where there are multiple log version in one file slice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ThinkerLei commented on issue #8425: [SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
Posted by "ThinkerLei (via GitHub)" <gi...@apache.org>.
ThinkerLei commented on issue #8425:
URL: https://github.com/apache/hudi/issues/8425#issuecomment-1506246385
@codope thanks for your quick feedback. Currently HoodieLogFileReader starts reading data from the offset 0 of logFile, but for READ_AS_STREAMING mode, we know the starting offset of the logfile written by each deltacommit. With this offset, we can use pread() or seek() to skip unnecessary data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #8425: [SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #8425:
URL: https://github.com/apache/hudi/issues/8425#issuecomment-1504998869
@ThinkerLei Can you please elaborate a bit more on `startOffset` and how you plan to use that to reduce read amplification?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ThinkerLei commented on issue #8425: [SUPPORT] When the downstream tasks read the logfile, use the startoffset of each logfile recorded in the deltacommit metadata to read the logfile
Posted by "ThinkerLei (via GitHub)" <gi...@apache.org>.
ThinkerLei commented on issue #8425:
URL: https://github.com/apache/hudi/issues/8425#issuecomment-1508002140
@danny0405 thanks for your quick feedback. I'm going to make some modifications here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org