You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/03/26 21:53:29 UTC
[PR] Add Parquet output format to segment converter tool [pinot]
xiangfu0 opened a new pull request, #17990:
URL: https://github.com/apache/pinot/pull/17990
## Summary
- Add `PinotSegmentToParquetConverter` that converts Pinot segments to Parquet format using `AvroParquetWriter` with GZIP compression
- Extend `PinotSegmentConvertCommand` to support `PARQUET` as an output format alongside AVRO, CSV, and JSON
- Handles multi-value columns (Object[]) by converting to lists and byte[] by wrapping in ByteBuffer for Parquet compatibility
## Test plan
- [ ] Build `pinot-tools` module: `./mvnw -pl pinot-tools -am verify`
- [ ] Manually test conversion: `pinot-admin.sh PinotSegmentConvert -segmentDir <dir> -outputDir <dir> -outputFormat PARQUET`
- [ ] Verify generated `.parquet` files can be read back by Parquet-compatible tools (e.g., `parquet-tools`, Spark)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2998232789
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -67,6 +69,14 @@ public class PinotSegmentConvertCommand extends AbstractBaseCommand implements C
@CommandLine.Option(names = {"-csvWithHeader"}, required = false, description = "Print CSV Header (default false).")
private boolean _csvWithHeader;
+ @CommandLine.Option(names = {"-parquetCompression"}, required = false,
+ description = "Parquet compression codec (GZIP/SNAPPY/ZSTD/LZ4/UNCOMPRESSED, default GZIP).")
+ private CompressionCodecName _parquetCompression = CompressionCodecName.GZIP;
+
+ @CommandLine.Option(names = {"-forwardIndexOnly"}, required = false,
+ description = "Load only forward index from the segment, skipping secondary indexes (default false).")
Review Comment:
The `-forwardIndexOnly` CLI option is global, but it is only applied in the `PARQUET` conversion path. If a user passes `-forwardIndexOnly` with `-outputFormat AVRO/CSV/JSON`, it will be silently ignored. Consider either (a) enforcing that this option is only allowed when `-outputFormat PARQUET` (fail fast with a clear error), or (b) renaming/scoping it (e.g., `-parquetForwardIndexOnly`) and updating the help text to explicitly state it only affects Parquet output.
```suggestion
description = "When outputFormat is PARQUET, load only forward index from the segment, skipping secondary "
+ "indexes; ignored for AVRO/CSV/JSON (default false).")
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentRecordReader.java:
##########
@@ -117,9 +117,23 @@ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable Re
*/
public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
boolean skipDefaultNullValues) {
+ init(indexDir, fieldsToRead, sortOrder, skipDefaultNullValues, false);
+ }
+
+ /**
+ * Initializes the record reader from an index directory with an option to load only forward index.
+ *
+ * @param indexDir Index directory
+ * @param fieldsToRead The fields to read from the segment. If null or empty, reads all fields
+ * @param sortOrder List of sorted columns
+ * @param skipDefaultNullValues Whether to skip putting default null values into the record
+ * @param forwardIndexOnly Whether to load only forward index, dictionary, and null value vector
+ */
+ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
+ boolean skipDefaultNullValues, boolean forwardIndexOnly) {
IndexSegment indexSegment;
try {
- indexSegment = ImmutableSegmentLoader.load(indexDir, ReadMode.mmap);
+ indexSegment = ImmutableSegmentLoader.load(indexDir, ReadMode.mmap, forwardIndexOnly);
Review Comment:
The new `forwardIndexOnly` code path (loading segments with only forward index/dictionary/null value vector) isn’t covered by existing unit tests. There are tests for `PinotSegmentRecordReader`, but they use the deprecated constructors and never exercise `init(..., forwardIndexOnly=true)`. Adding a test that creates a segment with at least one secondary index (e.g., inverted index/text index) and verifies `PinotSegmentRecordReader` can still read rows correctly with `forwardIndexOnly=true` would help prevent regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2998337229
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentRecordReader.java:
##########
@@ -117,9 +117,23 @@ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable Re
*/
public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
boolean skipDefaultNullValues) {
+ init(indexDir, fieldsToRead, sortOrder, skipDefaultNullValues, false);
+ }
+
+ /**
+ * Initializes the record reader from an index directory with an option to load only forward index.
+ *
+ * @param indexDir Index directory
+ * @param fieldsToRead The fields to read from the segment. If null or empty, reads all fields
+ * @param sortOrder List of sorted columns
+ * @param skipDefaultNullValues Whether to skip putting default null values into the record
+ * @param forwardIndexOnly Whether to load only forward index, dictionary, and null value vector
+ */
+ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
+ boolean skipDefaultNullValues, boolean forwardIndexOnly) {
IndexSegment indexSegment;
try {
- indexSegment = ImmutableSegmentLoader.load(indexDir, ReadMode.mmap);
+ indexSegment = ImmutableSegmentLoader.load(indexDir, ReadMode.mmap, forwardIndexOnly);
Review Comment:
Done. Added `testPinotSegmentRecordReaderForwardIndexOnly` to `PinotSegmentRecordReaderTest` that verifies all rows are read correctly with `forwardIndexOnly=true`, covering SV, MV, and metric columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2998591500
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java:
##########
@@ -68,8 +68,19 @@ private ImmutableSegmentLoader() {
*/
public static ImmutableSegment load(File indexDir, ReadMode readMode)
throws Exception {
+ return load(indexDir, readMode, false);
+ }
+
+ /**
+ * Loads the segment in read-only mode with an option to load only forward index, dictionary,
+ * and null value vector. This is useful for tools like segment converters that only need to
+ * read data without requiring secondary indexes.
Review Comment:
The new overload is documented as loading “only forward index, dictionary, and null value vector”, but the segment load path still unconditionally loads segment-level indexes when present (e.g., star-tree via `segmentReader.hasStarTreeIndex()` and multi-column text index via `hasMultiColumnTextIndex()`). Either adjust this Javadoc to reflect what is actually skipped, or update the loader to also bypass those segment-level indexes when `forwardIndexOnly=true`.
```suggestion
* Loads the segment in read-only mode with an option to load only column-level forward index, dictionary,
* and null value vector, skipping other column-level secondary indexes. Segment-level indexes (such as
* star-tree or multi-column text index) are still loaded when present. This is useful for tools like segment
* converters that only need to read data without requiring column-level secondary indexes.
```
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentRecordReader.java:
##########
@@ -117,9 +117,23 @@ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable Re
*/
public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
boolean skipDefaultNullValues) {
+ init(indexDir, fieldsToRead, sortOrder, skipDefaultNullValues, false);
+ }
+
+ /**
+ * Initializes the record reader from an index directory with an option to load only forward index.
+ *
+ * @param indexDir Index directory
+ * @param fieldsToRead The fields to read from the segment. If null or empty, reads all fields
+ * @param sortOrder List of sorted columns
+ * @param skipDefaultNullValues Whether to skip putting default null values into the record
+ * @param forwardIndexOnly Whether to load only forward index, dictionary, and null value vector
Review Comment:
This Javadoc says `forwardIndexOnly` loads only forward index/dictionary/null value vector, but `ImmutableSegmentLoader` still loads segment-level indexes like star-tree and multi-column text indexes even when `forwardIndexOnly=true`. Please either narrow the Javadoc claim here or ensure the loader path fully honors the flag so callers aren’t misled about what gets loaded.
```suggestion
* Initializes the record reader from an index directory with an option to prefer loading only forward index.
*
* @param indexDir Index directory
* @param fieldsToRead The fields to read from the segment. If null or empty, reads all fields
* @param sortOrder List of sorted columns
* @param skipDefaultNullValues Whether to skip putting default null values into the record
* @param forwardIndexOnly Whether to request loading forward index, dictionary, and null value vector without
* requiring all other segment-level indexes to be skipped
```
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -127,16 +137,21 @@ public boolean execute()
switch (FileFormat.fromString(_outputFormat)) {
case AVRO:
outputPath += ".avro";
- new PinotSegmentToAvroConverter(inputPath, outputPath).convert();
+ new PinotSegmentToAvroConverter(inputPath, outputPath, _forwardIndexOnly).convert();
break;
case CSV:
outputPath += ".csv";
- new PinotSegmentToCsvConverter(inputPath, outputPath, _csvDelimiter, _csvDelimiter, _csvWithHeader)
- .convert();
+ new PinotSegmentToCsvConverter(inputPath, outputPath, _csvDelimiter, _csvDelimiter, _csvWithHeader,
Review Comment:
`-csvListDelimiter` is defined as a CLI option, but the CSV conversion path passes `_csvDelimiter` for both the row delimiter and the multi-value list delimiter. This makes `-csvListDelimiter` effectively ignored. Pass `_csvListDelimiter` as the list delimiter argument when constructing `PinotSegmentToCsvConverter` so MV columns use the configured separator.
```suggestion
new PinotSegmentToCsvConverter(inputPath, outputPath, _csvDelimiter, _csvListDelimiter, _csvWithHeader,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2999122467
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -127,16 +137,21 @@ public boolean execute()
switch (FileFormat.fromString(_outputFormat)) {
case AVRO:
outputPath += ".avro";
- new PinotSegmentToAvroConverter(inputPath, outputPath).convert();
+ new PinotSegmentToAvroConverter(inputPath, outputPath, _forwardIndexOnly).convert();
break;
case CSV:
outputPath += ".csv";
- new PinotSegmentToCsvConverter(inputPath, outputPath, _csvDelimiter, _csvDelimiter, _csvWithHeader)
- .convert();
+ new PinotSegmentToCsvConverter(inputPath, outputPath, _csvDelimiter, _csvDelimiter, _csvWithHeader,
Review Comment:
Good catch! Fixed — now passing `_csvListDelimiter` instead of `_csvDelimiter` for the list delimiter. This was a pre-existing bug.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java:
##########
@@ -68,8 +68,19 @@ private ImmutableSegmentLoader() {
*/
public static ImmutableSegment load(File indexDir, ReadMode readMode)
throws Exception {
+ return load(indexDir, readMode, false);
+ }
+
+ /**
+ * Loads the segment in read-only mode with an option to load only forward index, dictionary,
+ * and null value vector. This is useful for tools like segment converters that only need to
+ * read data without requiring secondary indexes.
Review Comment:
Done. Updated the Javadoc to clarify that `forwardIndexOnly` skips column-level secondary indexes but segment-level indexes (star-tree, multi-column text) are still loaded when present.
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/readers/PinotSegmentRecordReader.java:
##########
@@ -117,9 +117,23 @@ public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable Re
*/
public void init(File indexDir, @Nullable Set<String> fieldsToRead, @Nullable List<String> sortOrder,
boolean skipDefaultNullValues) {
+ init(indexDir, fieldsToRead, sortOrder, skipDefaultNullValues, false);
+ }
+
+ /**
+ * Initializes the record reader from an index directory with an option to load only forward index.
+ *
+ * @param indexDir Index directory
+ * @param fieldsToRead The fields to read from the segment. If null or empty, reads all fields
+ * @param sortOrder List of sorted columns
+ * @param skipDefaultNullValues Whether to skip putting default null values into the record
+ * @param forwardIndexOnly Whether to load only forward index, dictionary, and null value vector
Review Comment:
Done. Updated the Javadoc to match — clarifies that it skips column-level secondary indexes while segment-level indexes may still be loaded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #17990:
URL: https://github.com/apache/pinot/pull/17990#issuecomment-4143651197
Documentation PR: https://github.com/pinot-contrib/pinot-docs/pull/584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2998350577
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -67,6 +69,14 @@ public class PinotSegmentConvertCommand extends AbstractBaseCommand implements C
@CommandLine.Option(names = {"-csvWithHeader"}, required = false, description = "Print CSV Header (default false).")
private boolean _csvWithHeader;
+ @CommandLine.Option(names = {"-parquetCompression"}, required = false,
+ description = "Parquet compression codec (GZIP/SNAPPY/ZSTD/LZ4/UNCOMPRESSED, default GZIP).")
+ private CompressionCodecName _parquetCompression = CompressionCodecName.GZIP;
+
+ @CommandLine.Option(names = {"-forwardIndexOnly"}, required = false,
+ description = "Load only forward index from the segment, skipping secondary indexes (default false).")
Review Comment:
Updated — `-forwardIndexOnly` now applies to all converters (AVRO, CSV, JSON, PARQUET), since for data dump use cases only the forward index is needed regardless of output format. Also took the opportunity to refactor the Avro converter to reuse `SegmentProcessorAvroUtils.convertGenericRowToAvroRecord()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #17990:
URL: https://github.com/apache/pinot/pull/17990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17990:
URL: https://github.com/apache/pinot/pull/17990#issuecomment-4138939675
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17990?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.61538%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.44%. Comparing base ([`57589a1`](https://app.codecov.io/gh/apache/pinot/commit/57589a1faf1d8c9d38e889bbe45dc14583d6d9b7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ab2b424`](https://app.codecov.io/gh/apache/pinot/commit/ab2b42484d4ff50d1c5f58e468c2ed0499f510c6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 5 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17990?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ent/index/column/PhysicalColumnIndexContainer.java](https://app.codecov.io/gh/apache/pinot/pull/17990?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Fcolumn%2FPhysicalColumnIndexContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2NvbHVtbi9QaHlzaWNhbENvbHVtbkluZGV4Q29udGFpbmVyLmphdmE=) | 60.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17990?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17990 +/- ##
=============================================
+ Coverage 34.20% 55.44% +21.24%
+ Complexity 795 745 -50
=============================================
Files 3198 2502 -696
Lines 193998 142981 -51017
Branches 29865 22937 -6928
=============================================
+ Hits 66348 79277 +12929
+ Misses 122065 56948 -65117
- Partials 5585 6756 +1171
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.44% <84.61%> (-44.56%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.44% <84.61%> (+21.24%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.44% <84.61%> (+21.24%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.44% <84.61%> (?)` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17990/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17990?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2997967541
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -138,6 +144,11 @@ public boolean execute()
outputPath += ".json";
new PinotSegmentToJsonConverter(inputPath, outputPath).convert();
break;
+ case PARQUET:
+ outputPath += ".parquet";
+ new PinotSegmentToParquetConverter(inputPath, outputPath,
+ CompressionCodecName.valueOf(_parquetCompression.toUpperCase())).convert();
Review Comment:
New PARQUET conversion path is added, but the existing `PinotSegmentConverterTest` only covers AVRO/CSV/JSON. Add a unit test that converts a generated segment to Parquet and reads it back (e.g., via `org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader`) to validate SV/MV and BYTES handling.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentToParquetConverter.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.segment.converter;
+
+import java.io.File;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData.Record;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+import org.apache.parquet.io.OutputFile;
+import org.apache.pinot.plugin.inputformat.avro.AvroUtils;
+import org.apache.pinot.plugin.inputformat.parquet.ParquetUtils;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * The <code>PinotSegmentToParquetConverter</code> class is the tool to convert Pinot segment to Parquet format.
+ */
+public class PinotSegmentToParquetConverter implements PinotSegmentConverter {
+ private final String _segmentDir;
+ private final String _outputFile;
+ private final CompressionCodecName _compressionCodec;
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile) {
+ this(segmentDir, outputFile, CompressionCodecName.GZIP);
+ }
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile,
+ CompressionCodecName compressionCodec) {
+ _segmentDir = segmentDir;
+ _outputFile = outputFile;
+ _compressionCodec = compressionCodec;
+ }
+
+ @Override
+ public void convert()
+ throws Exception {
+ File indexDir = new File(_segmentDir);
+ Schema avroSchema = AvroUtils.getAvroSchemaFromPinotSchema(new SegmentMetadataImpl(indexDir).getSchema());
+ OutputFile outputFile =
+ HadoopOutputFile.fromPath(new Path(_outputFile), ParquetUtils.getParquetHadoopConfiguration());
+ try (PinotSegmentRecordReader pinotSegmentRecordReader = new PinotSegmentRecordReader(new File(_segmentDir))) {
+ try (ParquetWriter<Record> parquetWriter =
+ AvroParquetWriter.<Record>builder(outputFile).withSchema(avroSchema)
+ .withCompressionCodec(_compressionCodec)
+ .withConf(ParquetUtils.getParquetHadoopConfiguration()).build()) {
+ GenericRow row = new GenericRow();
+ while (pinotSegmentRecordReader.hasNext()) {
+ row = pinotSegmentRecordReader.next(row);
+ Record record = new Record(avroSchema);
+ for (Map.Entry<String, Object> entry : row.getFieldToValueMap().entrySet()) {
+ String field = entry.getKey();
+ Object value = entry.getValue();
+ if (value instanceof Object[]) {
+ record.put(field, Arrays.asList((Object[]) value));
+ } else if (value instanceof byte[]) {
+ record.put(field, ByteBuffer.wrap((byte[]) value));
+ } else {
+ record.put(field, value);
+ }
+ }
Review Comment:
The GenericRow → Avro `Record` population logic here duplicates the implementation in `pinot-core` (`SegmentProcessorAvroUtils.convertGenericRowToAvroRecord(...)`) and in `PinotSegmentToAvroConverter`. Reusing the shared utility (or extracting a common helper in this package) would reduce duplication and help prevent subtle divergence in type handling over time.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -67,6 +69,10 @@ public class PinotSegmentConvertCommand extends AbstractBaseCommand implements C
@CommandLine.Option(names = {"-csvWithHeader"}, required = false, description = "Print CSV Header (default false).")
private boolean _csvWithHeader;
+ @CommandLine.Option(names = {"-parquetCompression"}, required = false,
+ description = "Parquet compression codec (GZIP/SNAPPY/ZSTD/LZ4/UNCOMPRESSED, default GZIP).")
+ private String _parquetCompression = "GZIP";
Review Comment:
Consider declaring `-parquetCompression` as a `CompressionCodecName` option (with a default value) instead of a `String`. Picocli will then validate the input and generate a clearer usage error for unsupported values, rather than deferring to a later `valueOf(...)` call.
```suggestion
private CompressionCodecName _parquetCompression = CompressionCodecName.GZIP;
```
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentToParquetConverter.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.segment.converter;
+
+import java.io.File;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData.Record;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+import org.apache.parquet.io.OutputFile;
+import org.apache.pinot.plugin.inputformat.avro.AvroUtils;
+import org.apache.pinot.plugin.inputformat.parquet.ParquetUtils;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * The <code>PinotSegmentToParquetConverter</code> class is the tool to convert Pinot segment to Parquet format.
+ */
+public class PinotSegmentToParquetConverter implements PinotSegmentConverter {
+ private final String _segmentDir;
+ private final String _outputFile;
+ private final CompressionCodecName _compressionCodec;
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile) {
+ this(segmentDir, outputFile, CompressionCodecName.GZIP);
+ }
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile,
+ CompressionCodecName compressionCodec) {
+ _segmentDir = segmentDir;
+ _outputFile = outputFile;
+ _compressionCodec = compressionCodec;
+ }
+
+ @Override
+ public void convert()
+ throws Exception {
+ File indexDir = new File(_segmentDir);
+ Schema avroSchema = AvroUtils.getAvroSchemaFromPinotSchema(new SegmentMetadataImpl(indexDir).getSchema());
+ OutputFile outputFile =
+ HadoopOutputFile.fromPath(new Path(_outputFile), ParquetUtils.getParquetHadoopConfiguration());
+ try (PinotSegmentRecordReader pinotSegmentRecordReader = new PinotSegmentRecordReader(new File(_segmentDir))) {
Review Comment:
`ParquetUtils.getParquetHadoopConfiguration()` allocates a new `Configuration` each call, but it’s invoked multiple times here (for `HadoopOutputFile.fromPath(...)` and again when building the writer). Create the configuration once and reuse it to avoid repeated allocation and to ensure both components share the same settings.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -138,6 +144,11 @@ public boolean execute()
outputPath += ".json";
new PinotSegmentToJsonConverter(inputPath, outputPath).convert();
break;
+ case PARQUET:
+ outputPath += ".parquet";
+ new PinotSegmentToParquetConverter(inputPath, outputPath,
+ CompressionCodecName.valueOf(_parquetCompression.toUpperCase())).convert();
+ break;
Review Comment:
`CompressionCodecName.valueOf(_parquetCompression.toUpperCase())` will throw an `IllegalArgumentException` with a relatively opaque message on bad input, and `toUpperCase()` is locale-sensitive. Use `toUpperCase(Locale.ROOT)` and catch/rehrow with a user-focused message that includes the allowed codec names (or rely on picocli enum parsing as suggested above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Add Parquet output format to segment converter tool [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #17990:
URL: https://github.com/apache/pinot/pull/17990#discussion_r2998336691
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentToParquetConverter.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.segment.converter;
+
+import java.io.File;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData.Record;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+import org.apache.parquet.io.OutputFile;
+import org.apache.pinot.plugin.inputformat.avro.AvroUtils;
+import org.apache.pinot.plugin.inputformat.parquet.ParquetUtils;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * The <code>PinotSegmentToParquetConverter</code> class is the tool to convert Pinot segment to Parquet format.
+ */
+public class PinotSegmentToParquetConverter implements PinotSegmentConverter {
+ private final String _segmentDir;
+ private final String _outputFile;
+ private final CompressionCodecName _compressionCodec;
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile) {
+ this(segmentDir, outputFile, CompressionCodecName.GZIP);
+ }
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile,
+ CompressionCodecName compressionCodec) {
+ _segmentDir = segmentDir;
+ _outputFile = outputFile;
+ _compressionCodec = compressionCodec;
+ }
+
+ @Override
+ public void convert()
+ throws Exception {
+ File indexDir = new File(_segmentDir);
+ Schema avroSchema = AvroUtils.getAvroSchemaFromPinotSchema(new SegmentMetadataImpl(indexDir).getSchema());
+ OutputFile outputFile =
+ HadoopOutputFile.fromPath(new Path(_outputFile), ParquetUtils.getParquetHadoopConfiguration());
+ try (PinotSegmentRecordReader pinotSegmentRecordReader = new PinotSegmentRecordReader(new File(_segmentDir))) {
+ try (ParquetWriter<Record> parquetWriter =
+ AvroParquetWriter.<Record>builder(outputFile).withSchema(avroSchema)
+ .withCompressionCodec(_compressionCodec)
+ .withConf(ParquetUtils.getParquetHadoopConfiguration()).build()) {
+ GenericRow row = new GenericRow();
+ while (pinotSegmentRecordReader.hasNext()) {
+ row = pinotSegmentRecordReader.next(row);
+ Record record = new Record(avroSchema);
+ for (Map.Entry<String, Object> entry : row.getFieldToValueMap().entrySet()) {
+ String field = entry.getKey();
+ Object value = entry.getValue();
+ if (value instanceof Object[]) {
+ record.put(field, Arrays.asList((Object[]) value));
+ } else if (value instanceof byte[]) {
+ record.put(field, ByteBuffer.wrap((byte[]) value));
+ } else {
+ record.put(field, value);
+ }
+ }
Review Comment:
Done. Replaced with `SegmentProcessorAvroUtils.convertGenericRowToAvroRecord()` from `pinot-core`.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -138,6 +144,11 @@ public boolean execute()
outputPath += ".json";
new PinotSegmentToJsonConverter(inputPath, outputPath).convert();
break;
+ case PARQUET:
+ outputPath += ".parquet";
+ new PinotSegmentToParquetConverter(inputPath, outputPath,
+ CompressionCodecName.valueOf(_parquetCompression.toUpperCase())).convert();
Review Comment:
Done. Added `testParquetConverter` to `PinotSegmentConverterTest` that verifies SV, MV, and BYTES column handling via round-trip conversion using `ParquetRecordReader`.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -67,6 +69,10 @@ public class PinotSegmentConvertCommand extends AbstractBaseCommand implements C
@CommandLine.Option(names = {"-csvWithHeader"}, required = false, description = "Print CSV Header (default false).")
private boolean _csvWithHeader;
+ @CommandLine.Option(names = {"-parquetCompression"}, required = false,
+ description = "Parquet compression codec (GZIP/SNAPPY/ZSTD/LZ4/UNCOMPRESSED, default GZIP).")
+ private String _parquetCompression = "GZIP";
Review Comment:
Done. Changed to `CompressionCodecName` type directly so picocli handles validation.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -138,6 +144,11 @@ public boolean execute()
outputPath += ".json";
new PinotSegmentToJsonConverter(inputPath, outputPath).convert();
break;
+ case PARQUET:
+ outputPath += ".parquet";
+ new PinotSegmentToParquetConverter(inputPath, outputPath,
+ CompressionCodecName.valueOf(_parquetCompression.toUpperCase())).convert();
+ break;
Review Comment:
Done. Now using `CompressionCodecName` type directly — the `valueOf` call is no longer needed.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentToParquetConverter.java:
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.tools.segment.converter;
+
+import java.io.File;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData.Record;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+import org.apache.parquet.io.OutputFile;
+import org.apache.pinot.plugin.inputformat.avro.AvroUtils;
+import org.apache.pinot.plugin.inputformat.parquet.ParquetUtils;
+import org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * The <code>PinotSegmentToParquetConverter</code> class is the tool to convert Pinot segment to Parquet format.
+ */
+public class PinotSegmentToParquetConverter implements PinotSegmentConverter {
+ private final String _segmentDir;
+ private final String _outputFile;
+ private final CompressionCodecName _compressionCodec;
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile) {
+ this(segmentDir, outputFile, CompressionCodecName.GZIP);
+ }
+
+ public PinotSegmentToParquetConverter(String segmentDir, String outputFile,
+ CompressionCodecName compressionCodec) {
+ _segmentDir = segmentDir;
+ _outputFile = outputFile;
+ _compressionCodec = compressionCodec;
+ }
+
+ @Override
+ public void convert()
+ throws Exception {
+ File indexDir = new File(_segmentDir);
+ Schema avroSchema = AvroUtils.getAvroSchemaFromPinotSchema(new SegmentMetadataImpl(indexDir).getSchema());
+ OutputFile outputFile =
+ HadoopOutputFile.fromPath(new Path(_outputFile), ParquetUtils.getParquetHadoopConfiguration());
+ try (PinotSegmentRecordReader pinotSegmentRecordReader = new PinotSegmentRecordReader(new File(_segmentDir))) {
Review Comment:
Done. Cached `ParquetUtils.getParquetHadoopConfiguration()` in a local variable and reused it for both `HadoopOutputFile.fromPath()` and `AvroParquetWriter.builder().withConf()`.
##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/PinotSegmentConvertCommand.java:
##########
@@ -67,6 +69,14 @@ public class PinotSegmentConvertCommand extends AbstractBaseCommand implements C
@CommandLine.Option(names = {"-csvWithHeader"}, required = false, description = "Print CSV Header (default false).")
private boolean _csvWithHeader;
+ @CommandLine.Option(names = {"-parquetCompression"}, required = false,
+ description = "Parquet compression codec (GZIP/SNAPPY/ZSTD/LZ4/UNCOMPRESSED, default GZIP).")
+ private CompressionCodecName _parquetCompression = CompressionCodecName.GZIP;
+
+ @CommandLine.Option(names = {"-forwardIndexOnly"}, required = false,
+ description = "Load only forward index from the segment, skipping secondary indexes (default false).")
Review Comment:
Good point. Updated the description to clarify it only applies to PARQUET output and is ignored for AVRO/CSV/JSON. Making it work for all formats would require changing the `PinotSegmentConverter` interface — can be a follow-up if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org