You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "YannByron (via GitHub)" <gi...@apache.org> on 2024/03/26 04:02:17 UTC
[PR] [spark][core] Support input_file_name UDF [paimon]
YannByron opened a new pull request, #3094:
URL: https://github.com/apache/paimon/pull/3094
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark][core] Support input_file_name UDF [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3094:
URL: https://github.com/apache/paimon/pull/3094#issuecomment-2019933629
LGTM!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark][core] Support input_file_name UDF [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on PR #3094:
URL: https://github.com/apache/paimon/pull/3094#issuecomment-2029300754
https://github.com/apache/paimon/issues/3057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark][core] Support input_file_name UDF [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #3094:
URL: https://github.com/apache/paimon/pull/3094#issuecomment-2019475572
Thanks @YannByron for the contribution.
Hooks have invaded a lot of core code.
Consider simpler solutions:
## Step 1: Modify paimon-core
Just rename `RecordWithPositionIterator` to `FileRecordIterator`.
```
interface FileRecordIterator<T> extends RecordReader.RecordIterator<T> {
Path filePath();
long returnedPosition();
T next() throws IOException;
}
```
The filePath is a final member for a `FileRecordIterator`. One `FileRecordIterator` should just for only one file.
## Step 2: Modify paimon-spark
We can introduce a `FileHolderRecordReader` to paimon-spark. In this reader, if the returned iterator is a `FileRecordIterator`, we can invoke `InputFileBlockHolder.set` and unset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark][core] Support input_file_name UDF [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #3094:
URL: https://github.com/apache/paimon/pull/3094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org