You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "steFaiz (via GitHub)" <gi...@apache.org> on 2025/11/28 07:26:26 UTC
[PR] [format] introduce SST file format [paimon]
steFaiz opened a new pull request, #6702:
URL: https://github.com/apache/paimon/pull/6702
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### SST File format
<!-- Linking this pull request to the issue -->
Linked issue: None
This PR is about to introduce an SST FileFormat for paimon, which is useful in below scenarios:
1. PK Lookup accelerate
2. BTree Global Index
An SST File is designed to serve:
1. Point queries: lookup a specified key
2. Range queries: seek to somewhere on which the record is exactly greater than or equal to target key, then scan the rest of records (user can decide when to stop)
3. Random access: directly return records specified by selection
<!-- What is the purpose of the change -->
### Tests
Please See:
1. org.apache.paimon.format.sst.SstFileFormatTest
4. org.apache.paimon.format.sst.SstFileTest
<!-- List UT and IT cases to verify this change -->
### API and Format
This PR adds a pair of new interface for FileFormat:
```
public abstract class FileFormat {
public FormatReaderFactory createReaderFactory(
RowType dataSchemaRowType,
RowType projectedRowType,
@Nullable List<Predicate> filters,
RowType keyType,
RowType valueType) {
return createReaderFactory(dataSchemaRowType, projectedRowType, filters);
}
public FormatWriterFactory createWriterFactory(
RowType type, RowType keyType, RowType valueType) {
return createWriterFactory(type);
}
}
```
These two methods will be implemented by File Formats which need to distinguish keys and values from an input row.
<!-- Does this change affect API or storage format -->
### Documentation
Documents will be added ASAP.
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [format] introduce SST file format [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz closed pull request #6702: [format] introduce SST file format
URL: https://github.com/apache/paimon/pull/6702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [format] introduce SST file format [paimon]
Posted by "steFaiz (via GitHub)" <gi...@apache.org>.
steFaiz commented on PR #6702:
URL: https://github.com/apache/paimon/pull/6702#issuecomment-3588225150
This PR is still on working:
1. modify block cache to read bytes from any SeekableInputStream
2. optimize memory using for Scan scenario.
3. optimize file layout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [format] introduce SST file format [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #6702: [format] introduce SST file format
URL: https://github.com/apache/paimon/pull/6702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org