You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2025/05/28 02:46:41 UTC
[PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
zhangyue19921010 opened a new pull request, #13365:
URL: https://github.com/apache/hudi/pull/13365
### Change Logs
details are as followed
### Impact
no
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
# Hudi Parquet Binary Stream Copy Optimization
## Background
- Hudi tables primarily use **Parquet format**.
- During **Clustering**, data undergoes redundant "column-to-row-to-column" transformations, involving:
- Compression/decompression
- Encoding/decoding
- Row/column conversions
- Format translations between compute engines and Hudi
- Similar inefficiencies exist in:
- Low-update-ratio **compaction** scenarios
- **CoW write amplification** with minimal data updates
## Optimization Approach
<img width="1314" alt="image" src="https://github.com/user-attachments/assets/aa3385d6-a90e-40b3-9a55-f45b7604c3d4" />
- **HoodieParquetRewriter**: A foundational implementation for:
✅ **Column-chunk-level binary stream copying**
✅ Directly copying unmodified Parquet column chunks to target files
✅ Rewriting Parquet footer for fast replication
- Applied to **Clustering scenarios** in this PR.
## Supported Capabilities
### 1. Schema Evolution
- Handles schema changes (e.g., adding fields, widening numeric types). **Behavior**:
- Uses the table's latest schema as output schema.
- Compares input file schemas against the latest schema.
- Fills missing fields with **null values** during binary copying.
### 2. Hudi Custom Metadata Merging
- Merges critical Hudi metadata from input files:
```plaintext
hoodie_min_record_key
hoodie_max_record_key
org.apache.hudi.bloomfilter
hoodie_bloom_filter_type_code
- Process:
- Collect metadata from all input files.
- Compute aggregated values (e.g., global min/max keys and merge bloom filters).
- Write merged metadata to output files.
## Future Roadmap
Extend binary copy support to Merge-on-Read (MoR) Compaction and CoW Upsert operations.
## Optimization Results
Using identical configurations, Clustering was performed on 100GB of data with both the default strategy and the binary stream copy strategy. The binary stream copy strategy achieved:
- **93% reduction in execution time** (from 18 minutes to 1.2 minutes)
- **95% reduction in computational workload** (from 28.7 task-hours to 1.3 task-hours)
**Before**
<img width="1482" alt="image (1)" src="https://github.com/user-attachments/assets/ac7800c7-f033-43c0-a5e8-277cf9cd2a77" />
**After**
<img width="1638" alt="image (3)" src="https://github.com/user-attachments/assets/c78d9201-0ab9-449d-9d51-3654f3e49fc7" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2935691887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960)
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963)
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2125150949
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SingleSparkJobExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+ if (!supportBinaryStreamCopy()) {
+ String message = "1. Only support for CoW table 2. Not support for sort function";
+ throw new UnsupportedOperationException(message);
+
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ @Override
+ protected List<WriteStatus> performClusteringForGroup(ReaderContextFactory<T> readerContextFactory, ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ return new ArrayList<>(handler.close());
+ }
+
+ private boolean supportBinaryStreamCopy() {
+ return this.getHoodieTable().getMetaClient().getTableType() == COPY_ON_WRITE;
Review Comment:
For MOR with `INSERT` operation, it should also work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2960581832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 40d3cbc23a2ab560fc760fd92145630cf8cad0e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948857645
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 90115910fb597ab25f3d4111d4f184727c5e1b10 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2982899944
@zhangyue19921010 in future, please open PRs from your fork. and not a feature branch on hudi . :). just noticed..
Great to see the perf results. I will hop on the revised PR .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2156495437
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -268,4 +267,106 @@ object AvroConversionUtils {
}
schema
}
+
+ /**
+ * Recursively aligns the nullable property of hoodie table schema, supporting nested structures
+ */
+ def alignFieldsNullability(sourceSchema: StructType, avroSchema: Schema): StructType = {
Review Comment:
Try my best to add UTs for core methods as much as possible to ensure code security :)
Add ut in TestAvroConversionUtils
```
test("test alignFieldsNullability function") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2959413669
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 7d5efa6090bbca6e5d479106f05060cac4be9089 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035)
* 79d067dc6055365a678997b8a6d3a0abccc41122 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2959575781
> @zhangyue19921010 @danny0405 pls check the match of table_schema and parquet schema before do binary copy. Spark BulkInsert will produce parquets with required attribute； however those columns in hudi itself are optional。 Do binary copy will corrupt parquet file [https://github.com/apache/spark/blob/1a3ae66c6c48bb319f0798826085e694fa7d0b58/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java#L389。](https://github.com/apache/spark/blob/1a3ae66c6c48bb319f0798826085e694fa7d0b58/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java#L389%E3%80%82) Suggest: throw exception directly when found above mismatch.
Thanks for your reminding. We notice this so that we use `alignFieldsNullability` func to resolve nullable inconsistencies issue. However, for the existing data file, we still need to do some pre-verification, which I will add later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2960288512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 79d067dc6055365a678997b8a6d3a0abccc41122 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100)
* 40d3cbc23a2ab560fc760fd92145630cf8cad0e2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005205855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448)
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987293530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987273556
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2156498837
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/BloomFilter.java:
##########
@@ -54,4 +54,10 @@ public interface BloomFilter {
* @return the bloom index type code.
**/
BloomFilterTypeCode getBloomFilterTypeCode();
+
+ /**
+ * Performs a logical OR operations with other BloomFilter.
+ * @param other
+ */
+ void or(BloomFilter other);
Review Comment:
Add `TestBloomFilter#testBloomORMethod` cover `SIMPLE` and `DYNAMIC_V0` BloomFilterTypeCode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987406731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2153783195
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/BinaryCopyHandleFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Combine all input file paths quickly into a single large file at block level.
+ * Without performing extra operations like data serialization/deserialization or compression/decompression.
+ */
+public class BinaryCopyHandleFactory<T, I, K, O>
+ extends WriteHandleFactory<T, I, K, O> implements Serializable {
+
+ private final List<StoragePath> inputFilePaths;
+
+ public BinaryCopyHandleFactory(List<StoragePath> inputFilePaths) {
+ this.inputFilePaths = inputFilePaths;
+ }
+
+ @Override
+ public HoodieBinaryCopyHandle<T, I, K, O> create(
+ HoodieWriteConfig config,
+ String commitTime,
+ HoodieTable<T, I, K, O> hoodieTable,
+ String partitionPath,
+ String fileIdPrefix,
+ TaskContextSupplier taskContextSupplier) {
+ return new HoodieBinaryCopyHandle(
+ config,
+ commitTime,
+ partitionPath,
+ getNextFileId(fileIdPrefix),
+ hoodieTable,
Review Comment:
rename: table
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/BinaryCopyHandleFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Combine all input file paths quickly into a single large file at block level.
+ * Without performing extra operations like data serialization/deserialization or compression/decompression.
+ */
+public class BinaryCopyHandleFactory<T, I, K, O>
Review Comment:
like the name "binary copy" way better.. thanks!
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -268,4 +267,106 @@ object AvroConversionUtils {
}
schema
}
+
+ /**
+ * Recursively aligns the nullable property of hoodie table schema, supporting nested structures
+ */
+ def alignFieldsNullability(sourceSchema: StructType, avroSchema: Schema): StructType = {
Review Comment:
lets please ensure each method here has unit tests.
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/BloomFilter.java:
##########
@@ -54,4 +54,10 @@ public interface BloomFilter {
* @return the bloom index type code.
**/
BloomFilterTypeCode getBloomFilterTypeCode();
+
+ /**
+ * Performs a logical OR operations with other BloomFilter.
+ * @param other
+ */
+ void or(BloomFilter other);
Review Comment:
@zhangyue19921010 can we move them to `TestBloomFilter` or the direct UT for the class.
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/HoodieDynamicBoundedBloomFilter.java:
##########
@@ -125,5 +126,42 @@ private void extractAndSetInternalBloomFilter(DataInputStream dis) throws IOExce
internalDynamicBloomFilter = new InternalDynamicBloomFilter();
internalDynamicBloomFilter.readFields(dis);
}
+
+ @Override
+ public void or(BloomFilter other) {
Review Comment:
Same. please move them to the UT for this class. I know it's a bit scattered now . but change starts somewhere :)
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/BinaryCopyHandleFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Combine all input file paths quickly into a single large file at block level.
+ * Without performing extra operations like data serialization/deserialization or compression/decompression.
+ */
+public class BinaryCopyHandleFactory<T, I, K, O>
+ extends WriteHandleFactory<T, I, K, O> implements Serializable {
+
+ private final List<StoragePath> inputFilePaths;
+
+ public BinaryCopyHandleFactory(List<StoragePath> inputFilePaths) {
+ this.inputFilePaths = inputFilePaths;
+ }
+
+ @Override
+ public HoodieBinaryCopyHandle<T, I, K, O> create(
+ HoodieWriteConfig config,
+ String commitTime,
+ HoodieTable<T, I, K, O> hoodieTable,
+ String partitionPath,
+ String fileIdPrefix,
+ TaskContextSupplier taskContextSupplier) {
+ return new HoodieBinaryCopyHandle(
+ config,
+ commitTime,
Review Comment:
nit. rename: commitTime -> instantTime
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/InternalDynamicBloomFilter.java:
##########
@@ -210,13 +210,29 @@ public void readFields(DataInput in) throws IOException {
/**
* Adds a new row to <i>this</i> dynamic Bloom filter.
*/
- private void addRow() {
+ protected void addRow() {
InternalBloomFilter[] tmp = new InternalBloomFilter[matrix.length + 1];
System.arraycopy(matrix, 0, tmp, 0, matrix.length);
tmp[tmp.length - 1] = new InternalBloomFilter(vectorSize, nbHash, hashType);
matrix = tmp;
}
+ protected void addRows(int size) {
Review Comment:
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2936041088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963)
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2935708101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963)
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124171173
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
Review Comment:
> lets keep this one level higher.. hudi.io.rewrite
Put it directly under the hudi.io
> lets stick to one concept across the PR
Using binaryCopy across the PR instead of rewrite since data are not changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2939021263
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3f1ff569f1b1857ec1635febc2733f56581e4896 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974)
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111565080
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriter.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import java.io.IOException;
+
+public interface HoodieFileRewriter {
Review Comment:
we can just extend the existing `HoodieFileWriter` and add a new interface method: `HoodieFileWriter#deepCopy` and only add a impl for parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948969565
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 90115910fb597ab25f3d4111d4f184727c5e1b10 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033)
* 7d5efa6090bbca6e5d479106f05060cac4be9089 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2149182005
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkBinaryCopyClusteringAndValidationMeta.java:
##########
@@ -192,4 +219,115 @@ private void checkFileFooter(List<String> clusteredFiles, List<HoodieRecord> all
assertTrue(rows.select("_hoodie_file_name").collectAsList().stream()
.allMatch(t -> t.getAs(0).equals(clusteredFileName)));
}
+
+ @Test
+ public void testSupportBinaryStreamCopy() throws IOException {
+ HoodieWriteConfig writeConfig = new HoodieWriteConfig.Builder()
+ .withPath(basePath)
+ .withSchema(TRIP_EXAMPLE_SCHEMA)
+ .withEmbeddedTimelineServerEnabled(false)
+ .build();
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieTable table = HoodieSparkTable.create(writeConfig, engineContext, metaClient);
+ SparkBinaryCopyClusteringExecutionStrategy strategy = new SparkBinaryCopyClusteringExecutionStrategy(table, engineContext, writeConfig);
+
+ MessageType legacySchema = new AvroSchemaConverter().convert(AVRO_SCHEMA);
+ Configuration conf = new Configuration();
+ conf.set("parquet.avro.write-old-list-structure", "false");
+ MessageType standardSchema = new AvroSchemaConverter(conf).convert(AVRO_SCHEMA);
+
+ BloomFilter simpleBloomFilter = BloomFilterFactory.createBloomFilter(1000, 0.0001, 10000, SIMPLE.name());
+ BloomFilter dynmicBloomFilter = BloomFilterFactory.createBloomFilter(1000, 0.0001, 10000, DYNAMIC_V0.name());
+ String file1 = makeTestFile("file-1.parquet", AVRO_SCHEMA, legacySchema, simpleBloomFilter);
+ String file2 = makeTestFile("file-2.parquet", AVRO_SCHEMA, legacySchema, dynmicBloomFilter);
+ String file3 = makeTestFile("file-3.parquet", AVRO_SCHEMA, standardSchema, dynmicBloomFilter);
+ String file4 = makeTestFile("file-4.parquet", AVRO_SCHEMA, standardSchema, dynmicBloomFilter);
+
+ // input file contains multiple bloom filter code type, should return false
+ List<ClusteringGroupInfo> groups = makeClusteringGroup(file1, file2);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ // input file contains legacy schema, should return false
+ groups = makeClusteringGroup(file2, file3);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ // otherwise should return true
+ groups = makeClusteringGroup(file3, file4);
+ Assertions.assertTrue(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ metaClient = HoodieTestUtils.init(basePath, HoodieTableType.MERGE_ON_READ);
+ table = HoodieSparkTable.create(writeConfig, engineContext, metaClient);
+ strategy = new SparkBinaryCopyClusteringExecutionStrategy(table, engineContext, writeConfig);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+ }
+
+ private String makeTestFile(String fileName, Schema schema, MessageType messageType, BloomFilter filter) throws IOException {
+ HoodieAvroWriteSupport writeSupport = new HoodieAvroWriteSupport(messageType, schema, Option.of(filter), new Properties());
+ StoragePath filePath = new StoragePath(tempDir.resolve(fileName).toAbsolutePath().toString());
+ HoodieConfig hoodieConfig = new HoodieConfig();
+ hoodieConfig.setValue("hoodie.base.path", basePath);
+ HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig =
+ new HoodieParquetConfig(
+ writeSupport,
+ CompressionCodecName.GZIP,
+ ParquetWriter.DEFAULT_BLOCK_SIZE,
+ ParquetWriter.DEFAULT_PAGE_SIZE,
+ 1024 * 1024 * 1024,
+ storageConf,
+ 0.1,
+ true);
+
+ HoodieAvroParquetWriter writer =
+ new HoodieAvroParquetWriter(filePath, parquetConfig, "001", new DummyTaskContextSupplier(), true);
+ writer.close();
+ return filePath.toString();
+ }
+
+ private List<ClusteringGroupInfo> makeClusteringGroup(String... files) {
+ List<ClusteringOperation> operations = Arrays.stream(files)
+ .map(file -> {
+ ClusteringOperation op = new ClusteringOperation();
+ op.setDataFilePath(file);
+ return op;
+ })
+ .collect(Collectors.toList());
+ ClusteringGroupInfo group = new ClusteringGroupInfo();
+ group.setNumOutputGroups(1);
+ group.setOperations(operations);
+ return Collections.singletonList(group);
+ }
+
+ static class DummyTaskContextSupplier extends TaskContextSupplier {
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2146292275
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestSparkBinaryCopyClusteringAndValidationMeta.java:
##########
@@ -192,4 +219,115 @@ private void checkFileFooter(List<String> clusteredFiles, List<HoodieRecord> all
assertTrue(rows.select("_hoodie_file_name").collectAsList().stream()
.allMatch(t -> t.getAs(0).equals(clusteredFileName)));
}
+
+ @Test
+ public void testSupportBinaryStreamCopy() throws IOException {
+ HoodieWriteConfig writeConfig = new HoodieWriteConfig.Builder()
+ .withPath(basePath)
+ .withSchema(TRIP_EXAMPLE_SCHEMA)
+ .withEmbeddedTimelineServerEnabled(false)
+ .build();
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+ HoodieTable table = HoodieSparkTable.create(writeConfig, engineContext, metaClient);
+ SparkBinaryCopyClusteringExecutionStrategy strategy = new SparkBinaryCopyClusteringExecutionStrategy(table, engineContext, writeConfig);
+
+ MessageType legacySchema = new AvroSchemaConverter().convert(AVRO_SCHEMA);
+ Configuration conf = new Configuration();
+ conf.set("parquet.avro.write-old-list-structure", "false");
+ MessageType standardSchema = new AvroSchemaConverter(conf).convert(AVRO_SCHEMA);
+
+ BloomFilter simpleBloomFilter = BloomFilterFactory.createBloomFilter(1000, 0.0001, 10000, SIMPLE.name());
+ BloomFilter dynmicBloomFilter = BloomFilterFactory.createBloomFilter(1000, 0.0001, 10000, DYNAMIC_V0.name());
+ String file1 = makeTestFile("file-1.parquet", AVRO_SCHEMA, legacySchema, simpleBloomFilter);
+ String file2 = makeTestFile("file-2.parquet", AVRO_SCHEMA, legacySchema, dynmicBloomFilter);
+ String file3 = makeTestFile("file-3.parquet", AVRO_SCHEMA, standardSchema, dynmicBloomFilter);
+ String file4 = makeTestFile("file-4.parquet", AVRO_SCHEMA, standardSchema, dynmicBloomFilter);
+
+ // input file contains multiple bloom filter code type, should return false
+ List<ClusteringGroupInfo> groups = makeClusteringGroup(file1, file2);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ // input file contains legacy schema, should return false
+ groups = makeClusteringGroup(file2, file3);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ // otherwise should return true
+ groups = makeClusteringGroup(file3, file4);
+ Assertions.assertTrue(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+
+ metaClient = HoodieTestUtils.init(basePath, HoodieTableType.MERGE_ON_READ);
+ table = HoodieSparkTable.create(writeConfig, engineContext, metaClient);
+ strategy = new SparkBinaryCopyClusteringExecutionStrategy(table, engineContext, writeConfig);
+ Assertions.assertFalse(strategy.supportBinaryStreamCopy(groups, new HashMap<>()));
+ }
+
+ private String makeTestFile(String fileName, Schema schema, MessageType messageType, BloomFilter filter) throws IOException {
+ HoodieAvroWriteSupport writeSupport = new HoodieAvroWriteSupport(messageType, schema, Option.of(filter), new Properties());
+ StoragePath filePath = new StoragePath(tempDir.resolve(fileName).toAbsolutePath().toString());
+ HoodieConfig hoodieConfig = new HoodieConfig();
+ hoodieConfig.setValue("hoodie.base.path", basePath);
+ HoodieParquetConfig<HoodieAvroWriteSupport> parquetConfig =
+ new HoodieParquetConfig(
+ writeSupport,
+ CompressionCodecName.GZIP,
+ ParquetWriter.DEFAULT_BLOCK_SIZE,
+ ParquetWriter.DEFAULT_PAGE_SIZE,
+ 1024 * 1024 * 1024,
+ storageConf,
+ 0.1,
+ true);
+
+ HoodieAvroParquetWriter writer =
+ new HoodieAvroParquetWriter(filePath, parquetConfig, "001", new DummyTaskContextSupplier(), true);
+ writer.close();
+ return filePath.toString();
+ }
+
+ private List<ClusteringGroupInfo> makeClusteringGroup(String... files) {
+ List<ClusteringOperation> operations = Arrays.stream(files)
+ .map(file -> {
+ ClusteringOperation op = new ClusteringOperation();
+ op.setDataFilePath(file);
+ return op;
+ })
+ .collect(Collectors.toList());
+ ClusteringGroupInfo group = new ClusteringGroupInfo();
+ group.setNumOutputGroups(1);
+ group.setOperations(operations);
+ return Collections.singletonList(group);
+ }
+
+ static class DummyTaskContextSupplier extends TaskContextSupplier {
Review Comment:
we have `LocalTaskContextSupplier` for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2138025106
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
This schema correction is necessary. The existing "bulk insert as row" operation fetches the DataFrame's schema(may inconsistencies with table schema). This inconsistencies in the nullable properties of identical columns lead to file corruption/unreadability after binary copying. Therefore, we've implemented a nullable type correction for the bulk insert as row operation to resolve this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2960283188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 79d067dc6055365a678997b8a6d3a0abccc41122 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100)
* 40d3cbc23a2ab560fc760fd92145630cf8cad0e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914803454
Please block one review from either me or @yihua .. to ensure the changes play well with how we are thinking about changing the spark writer path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111562217
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
> I guess it is impossible to do so, because the copy breaks the sort layout of the target file.
Agreed. So it might be a bit hard to understand that this class inherits from `SparkSortAndSizeExecutionStrategy`. How about introducing a new strategy that is only used for merging small files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111553608
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
Review Comment:
`HoodieCreateRewriteHandle` -> `HoodieDeepCopyHandle`?
should we just make it extend from `HoodieWriteHandle`, the core method `HoodieCreateHandle#doWrite` does not even been utilitized here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938175838
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965)
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2122348417
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieCreateRewriteHandle.class);
+
+ protected final HoodieFileRewriter rewriter;
+
+ private final List<StoragePath> inputFiles;
+
+ public HoodieCreateRewriteHandle(
+ HoodieWriteConfig config,
+ String instantTime,
+ String partitionPath,
+ String fileId,
+ HoodieTable<T, I, K, O> hoodieTable,
+ TaskContextSupplier taskContextSupplier,
+ List<StoragePath> inputFilePaths,
+ boolean preserveMetadata) {
+ super(
+ config,
+ instantTime,
+ hoodieTable,
+ partitionPath,
+ fileId,
+ Option.empty(),
+ taskContextSupplier,
+ preserveMetadata,
+ false);
+ try {
+ this.inputFiles = inputFilePaths;
+ HoodieFileMetadataMerger fileMetadataMerger = new HoodieFileMetadataMerger();
+ this.rewriter = HoodieFileRewriterFactory.getFileRewriter(
+ inputFilePaths,
+ path,
+ hoodieTable.getStorageConf().unwrapAs(Configuration.class),
+ hoodieTable.getConfig(),
+ fileMetadataMerger,
+ config.getRecordMerger().getRecordType(), this.writeSchemaWithMetaFields);
+ } catch (IOException e) {
+ LOG.error("Fail to create file rewriter, cause: ", e);
+ throw new HoodieException(e);
+ }
+ }
+
+ public void rewrite() {
+ LOG.info("Start to rewrite source files " + this.inputFiles + " into target file: " + this.path);
+ long start = System.currentTimeMillis();
+ long records = 0;
+ try {
+ records = this.rewriter.rewrite();
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ } finally {
+ this.recordsWritten = records;
+ this.insertRecordsWritten = records;
+ }
+ LOG.info("Finish rewriting " + this.path + ". Using " + (System.currentTimeMillis() - start) + " mills");
+ }
+
+ @Override
+ public List<WriteStatus> close() {
+ try {
+ this.rewriter.close();
+ return super.close();
+ } catch (IOException e) {
+ LOG.error("Fail to close the rewrite handle for path: " + path);
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ }
+
+ @Override
+ public boolean canWrite(HoodieRecord record) {
Review Comment:
So we don't do any file sizing here? we need logger.warn(..) etc noting that this change of behavior.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieCreateRewriteHandle.class);
+
+ protected final HoodieFileRewriter rewriter;
+
+ private final List<StoragePath> inputFiles;
+
+ public HoodieCreateRewriteHandle(
+ HoodieWriteConfig config,
+ String instantTime,
+ String partitionPath,
+ String fileId,
+ HoodieTable<T, I, K, O> hoodieTable,
+ TaskContextSupplier taskContextSupplier,
+ List<StoragePath> inputFilePaths,
+ boolean preserveMetadata) {
+ super(
+ config,
+ instantTime,
+ hoodieTable,
+ partitionPath,
+ fileId,
+ Option.empty(),
+ taskContextSupplier,
+ preserveMetadata,
+ false);
+ try {
+ this.inputFiles = inputFilePaths;
+ HoodieFileMetadataMerger fileMetadataMerger = new HoodieFileMetadataMerger();
+ this.rewriter = HoodieFileRewriterFactory.getFileRewriter(
+ inputFilePaths,
+ path,
+ hoodieTable.getStorageConf().unwrapAs(Configuration.class),
+ hoodieTable.getConfig(),
+ fileMetadataMerger,
+ config.getRecordMerger().getRecordType(), this.writeSchemaWithMetaFields);
+ } catch (IOException e) {
+ LOG.error("Fail to create file rewriter, cause: ", e);
+ throw new HoodieException(e);
+ }
+ }
+
+ public void rewrite() {
+ LOG.info("Start to rewrite source files " + this.inputFiles + " into target file: " + this.path);
+ long start = System.currentTimeMillis();
Review Comment:
use HoodieTimer
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Version;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.CompressionConverter.TransParquetFileReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.internal.column.columnindex.ColumnIndex;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.FILENAME_METADATA_FIELD;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class TestHoodieParquetFileRewriter {
+
+ private final int numRecord = 1;
+ private Configuration conf = new Configuration();
+ private List<TestFile> inputFiles = null;
+ private String outputFile = null;
+ private HoodieParquetFileRewriter rewriter = null;
+
+ @BeforeEach
+ public void setUp() {
+ outputFile = TestFileBuilder.createTempFile("test");
+ }
+
+ @AfterEach
+ public void after() {
+ if (outputFile != null) {
+ TestFileBuilder.deleteTempFile(outputFile);
+ }
+ if (inputFiles != null) {
+ inputFiles.stream().map(TestFile::getFileName).forEach(TestFileBuilder::deleteTempFile);
+ }
+ }
+
+ @Test
+ public void testBasic() throws Exception {
+ MessageType schema = createSchema();
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+
+ rewriter = parquetFileRewriter(schema, "GZIP");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+ validateSchema(fileSchema);
+
+ // Verify codec
+ verifyCodec(outputFile, CompressionCodecName.GZIP);
+
+ // Verify the merged data are not changed
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2, 3, 4);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testTranslateCodec() throws Exception {
+ MessageType schema = createSchema();
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "UNCOMPRESSED"));
+
+ rewriter = parquetFileRewriter(schema, "ZSTD");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+ validateSchema(fileSchema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.ZSTD);
+
+ // Verify the data are not changed for the columns not pruned
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2, 3, 4);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testDifferentSchema() throws Exception {
+ MessageType schema1 = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"),
+ new GroupType(OPTIONAL, "Links",
+ new PrimitiveType(REPEATED, BINARY, "Backward"),
+ new PrimitiveType(REPEATED, BINARY, "Forward")));
+ MessageType schema2 = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"));
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema1, "UNCOMPRESSED"));
+ inputFiles.add(makeTestFile(schema2, "UNCOMPRESSED"));
+
+ rewriter = parquetFileRewriter(schema1, "UNCOMPRESSED");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType schema = pmd.getFileMetaData().getSchema();
+ validateSchema(schema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.UNCOMPRESSED);
+
+ // Verify the data are not changed
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testHoodieMetaColumn() throws Exception {
+ MessageType schema = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, BINARY, FILENAME_METADATA_FIELD),
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"),
+ new GroupType(OPTIONAL, "Links",
+ new PrimitiveType(REPEATED, BINARY, "Backward"),
+ new PrimitiveType(REPEATED, BINARY, "Forward")));
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+
+ rewriter = parquetFileRewriter(schema, "GZIP");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.GZIP);
+
+ // Verify the data are not changed
+ validateColumnData();
Review Comment:
would n't some meta fields like `_hoodie_file_name` change.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/SingleFileHandleRewriteFactory.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Rewrite all inputFilePaths related files into one huge file.
Review Comment:
"rewrite" is lil confusing, since no data is changing..
"Combine all input file paths quickly into a single large file" ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##########
@@ -103,9 +103,9 @@ public HoodieCreateHandle(HoodieWriteConfig config, String instantTime, HoodieTa
partitionMetadata.trySave();
createMarkerFile(partitionPath,
FSUtils.makeBaseFileName(this.instantTime, this.writeToken, this.fileId, hoodieTable.getBaseFileExtension()));
- this.fileWriter =
- HoodieFileWriterFactory.getFileWriter(instantTime, path, hoodieTable.getStorage(), config,
- writeSchemaWithMetaFields, this.taskContextSupplier, config.getRecordMerger().getRecordType());
+ this.fileWriter = initWriter
Review Comment:
not a fan of using `null` .. can we instead make the overall `fileWriter` initialization lazy?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
Review Comment:
javadocs
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Version;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.CompressionConverter.TransParquetFileReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.internal.column.columnindex.ColumnIndex;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.FILENAME_METADATA_FIELD;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class TestHoodieParquetFileRewriter {
Review Comment:
can we add more tests from the hudi context.. covering meta fields, stats and so forth. bloom filters..
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/HoodieDynamicBoundedBloomFilter.java:
##########
@@ -125,5 +126,42 @@ private void extractAndSetInternalBloomFilter(DataInputStream dis) throws IOExce
internalDynamicBloomFilter = new InternalDynamicBloomFilter();
internalDynamicBloomFilter.readFields(dis);
}
+
+ @Override
+ public void or(BloomFilter other) {
Review Comment:
unit tests
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/BloomFilter.java:
##########
@@ -54,4 +54,10 @@ public interface BloomFilter {
* @return the bloom index type code.
**/
BloomFilterTypeCode getBloomFilterTypeCode();
+
+ /**
+ * Performs a logical OR operations with other BloomFilter.
+ * @param other
+ */
+ void or(BloomFilter other);
Review Comment:
unit tests.
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriterFactory.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+
+public class HoodieFileRewriterFactory {
+
+ private static HoodieFileRewriterFactory getWriterFactory(HoodieRecordType recordType, String extension) {
+ if (PARQUET.getFileExtension().equals(extension)) {
+ try {
+ Class<?> clazz = ReflectionUtils.getClass("org.apache.hudi.parquet.io.HoodieParquetRewriterFactory");
+ return (HoodieFileRewriterFactory) clazz.newInstance();
+ } catch (IllegalAccessException | IllegalArgumentException | InstantiationException e) {
+ throw new HoodieException("Unable to create hoodie avro parquet file writer factory", e);
+ }
+ }
+ throw new UnsupportedOperationException(extension + " file format not supported yet.");
+ }
+
+ public static <T, I, K, O> HoodieFileRewriter getFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ HoodieRecordType recordType,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ String extension = FSUtils.getFileExtension(targetFilePath.getName());
+ HoodieFileRewriterFactory factory = getWriterFactory(recordType, extension);
+ return factory.newFileRewriter(inputFilePaths, targetFilePath, conf, config, metadataMerger, writeSchemaWithMetaFields);
+ }
+
+ protected <T> HoodieFileRewriter newFileRewriter(
Review Comment:
can we avoid this
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkStreamCopyClusteringExecutionStrategy.class);
+
+ public SparkStreamCopyClusteringExecutionStrategy(
Review Comment:
lets please add UTs for this class. and all the new files introduced
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY;
+
+public class HoodieFileMetadataMerger {
+
+ private final HashMap<String, String> mergedMateData = new HashMap<>();
+
+ private BloomFilter bloomFilter;
+
+ private String minRecordKey;
+
+ private String maxRecordKey;
+
+ public HoodieFileMetadataMerger() {
+
+ }
+
+ public Map<String, String> mergeMetaData(Map<String, String> metaMap) {
Review Comment:
UT
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
Review Comment:
lets keep this one level higher.. `hudi.io.rewrite` ? naming wise : lets stick to one concept across the PR.
fastcopy or rewrite. and rename this `..hudi.io.fastcopy` ?
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.util.CompressionConverter;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.schema.MessageType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+
+public class HoodieParquetFileRewriter extends HoodieParquetRewriterBase implements HoodieFileRewriter {
Review Comment:
is this adapted from parquet-mr/parquet-java?
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestFile.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.parquet.example.data.simple.SimpleGroup;
+
+public class TestFile {
Review Comment:
same.. please consolidate this clas and TestFileBuilder into sth maintainable/minimize.
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriterFactory.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+
+public class HoodieFileRewriterFactory {
+
+ private static HoodieFileRewriterFactory getWriterFactory(HoodieRecordType recordType, String extension) {
+ if (PARQUET.getFileExtension().equals(extension)) {
+ try {
+ Class<?> clazz = ReflectionUtils.getClass("org.apache.hudi.parquet.io.HoodieParquetRewriterFactory");
+ return (HoodieFileRewriterFactory) clazz.newInstance();
+ } catch (IllegalAccessException | IllegalArgumentException | InstantiationException e) {
+ throw new HoodieException("Unable to create hoodie avro parquet file writer factory", e);
+ }
+ }
+ throw new UnsupportedOperationException(extension + " file format not supported yet.");
+ }
+
+ public static <T, I, K, O> HoodieFileRewriter getFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ HoodieRecordType recordType,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ String extension = FSUtils.getFileExtension(targetFilePath.getName());
+ HoodieFileRewriterFactory factory = getWriterFactory(recordType, extension);
+ return factory.newFileRewriter(inputFilePaths, targetFilePath, conf, config, metadataMerger, writeSchemaWithMetaFields);
+ }
+
+ protected <T> HoodieFileRewriter newFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ throw new UnsupportedOperationException();
Review Comment:
msg?
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestFileBuilder.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.crypto.ParquetCipher;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.example.GroupWriteSupport;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+
+public class TestFileBuilder {
Review Comment:
This seems like a test util and not an actual test? if this is borrowed from parquet.. can we move this along with other test utils.. and see if we can avoid copying over this much code.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
Review Comment:
+1. lets extend from HoodieWriteHandle and avoid the `initWriter` flags.
Naming: Prefer to call this sth like `HoodieFastCopyHandle` given all this is doing is creating a file out of input files. The CreateHandle takes input itr and create a file. So don't want to overload that naming here.
idk if Deep vs Shallow copy is clear in this context, since shallow copy is not an option. we need to copy the actual bytes/records
##########
hudi-common/src/main/java/org/apache/hudi/common/config/TypedProperties.java:
##########
@@ -39,7 +39,7 @@ public TypedProperties() {
super(null);
}
- protected TypedProperties(Properties defaults) {
+ public TypedProperties(Properties defaults) {
Review Comment:
+1
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
Review Comment:
Good point
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
+1 on a new strategy. if sorting is not supported, this should not extend the SortAndSize..Strategy.. Let's cleanup code in a separate PR if you prefer to get this landed in a good place.
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriter.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import java.io.IOException;
+
+public interface HoodieFileRewriter {
Review Comment:
The interface shoud at-least cover `rewrite(inputFiles, outputFiles, cfg)` type methods?
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriter.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import java.io.IOException;
+
+public interface HoodieFileRewriter {
Review Comment:
Lets not overload the `FileWriter` with this feature, since its not clear if we can handle this for all file types? FileWriter works at the record iterator level . IMO deepCopy/fastCopy like implementations work one level lower than record level iterators writers do..
I question if we need an interface esp in its current form where its just two methods that don't take any args. What other implementations do we want to have for the `HoodieFileRewriter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948537671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981)
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2146293689
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -175,30 +173,17 @@ public boolean supportBinaryStreamCopy(List<ClusteringGroupInfo> inputGroups, Ma
}
JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
- List<Tuple2<BloomFilterTypeCode, Boolean>> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
+
+ List<ParquetBinaryCopyChecker.ParquetFileInfo> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
.flatMap(group -> group.getOperations().iterator())
.map(op -> {
String filePath = op.getDataFilePath();
- ParquetUtils fileUtils = new ParquetUtils();
- BloomFilter filter = fileUtils.readBloomFilterFromMetadata(getHoodieTable().getStorage(), new StoragePath(filePath));
- boolean isLegacySchema = !fileUtils.getFormat().equals(PARQUET) || fileUtils.containLegacy2LevelArrayType(getHoodieTable().getStorage(), filePath);
- return Tuple2.apply(filter == null ? null : filter.getBloomFilterTypeCode(), isLegacySchema);
+ if (!filePath.endsWith(PARQUET.getFileExtension())) {
+ return new ParquetBinaryCopyChecker.ParquetFileInfo(false, "", Collections.emptyMap());
+ }
+ return ParquetBinaryCopyChecker.verifyFile(getHoodieTable().getStorageConf().unwrapAs(Configuration.class), filePath);
Review Comment:
these files would be checked twice with line 187?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2149191571
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -175,30 +173,17 @@ public boolean supportBinaryStreamCopy(List<ClusteringGroupInfo> inputGroups, Ma
}
JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
- List<Tuple2<BloomFilterTypeCode, Boolean>> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
+
+ List<ParquetBinaryCopyChecker.ParquetFileInfo> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
.flatMap(group -> group.getOperations().iterator())
.map(op -> {
String filePath = op.getDataFilePath();
- ParquetUtils fileUtils = new ParquetUtils();
- BloomFilter filter = fileUtils.readBloomFilterFromMetadata(getHoodieTable().getStorage(), new StoragePath(filePath));
- boolean isLegacySchema = !fileUtils.getFormat().equals(PARQUET) || fileUtils.containLegacy2LevelArrayType(getHoodieTable().getStorage(), filePath);
- return Tuple2.apply(filter == null ? null : filter.getBloomFilterTypeCode(), isLegacySchema);
+ if (!filePath.endsWith(PARQUET.getFileExtension())) {
+ return new ParquetBinaryCopyChecker.ParquetFileInfo(false, "", Collections.emptyMap());
+ }
+ return ParquetBinaryCopyChecker.verifyFile(getHoodieTable().getStorageConf().unwrapAs(Configuration.class), filePath);
Review Comment:
Inappropriate file naming, verifyFile has been changed to collectFileInfo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2979971425
> @zhangyue19921010 nice work. Test with 1.7TB data, 300 columns binary copy achived `7.7 mins` vs normal write `35 mins`; and cpu cost cut down from `90% -> 60%`
>
> i have add some buffer for inputStream
>
> ```
> public class HoodieInputFile implements InputFile {
> @Override
> public SeekableInputStream newStream() throws IOException {
> byte[] buffer = new byte[(int) stat.getLen()];
> FSDataInputStream fsDataInputStream = fs.open(stat.getPath());
> fsDataInputStream.readFully(buffer, 0, (int) stat.getLen);
> return new DelegatingSeekableInputStream(new ByteBufferBackedInputStream(buffer)) {
> public long getPos() ....
> public void seek....
> }
> }
> }
> ```
Thanks @xiarixiaoyao for your help in verifying the functionality and performance of this PR. It's great to see the approximately **5x performance improvement**. cc: @vinothchandar @danny0405 ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987870492
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2166975089
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFieldsNullability(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
### Why We Need `alignFieldsNullability`
Currently, Hudi retrieves the Table Schema from three sources in order of priority:
1. Schema from commit file
2. Schema from table config
3. Schema from data file
```java
private Option<Schema> getTableAvroSchemaInternal(boolean includeMetadataFields, Option<HoodieInstant> instantOpt) {
Option<Schema> schema =
(instantOpt.isPresent()
? getTableSchemaFromCommitMetadata(instantOpt.get(), includeMetadataFields)
: getTableSchemaFromLatestCommitMetadata(includeMetadataFields))
.or(() ->
metaClient.getTableConfig().getTableCreateSchema()
.map(tableSchema ->
includeMetadataFields
? HoodieAvroUtils.addMetadataFields(tableSchema, hasOperationField.get())
: tableSchema)
)
.or(() -> {
Option<Schema> schemaFromDataFile = getTableAvroSchemaFromDataFileInternal();
return includeMetadataFields
? schemaFromDataFile
: schemaFromDataFile.map(HoodieAvroUtils::removeMetadataFields);
});
// TODO: Append partition columns in all read-paths
if (metaClient.getTableConfig().shouldDropPartitionColumns() && schema.isPresent()) {
return metaClient.getTableConfig().getPartitionFields()
.map(partitionFields -> appendPartitionColumns(schema.get(), Option.ofNullable(partitionFields)))
.or(() -> schema);
}
return schema;
}
```
However, when `using bulk insert as row,` the schema of the generated Parquet data files may have `inconsistent Optional/Required attributes` compared to the table schema. This occurs because bulk insert as row directly uses the DataFrame's schema as the write schema:
```scala
def bulkInsert(dataset: Dataset[Row],
instantTime: String,
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
shouldPreserveHoodieMetadata: Boolean,
operation: WriteOperationType): HoodieData[WriteStatus] = {
val schema = operation match {
case WriteOperationType.CLUSTER =>
alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
case _ =>
dataset.schema // ← Uses raw DataFrame schema!
}
```
**Generally**, we downplay the distinction between `Optional` and `Required` in schemas, making it seem like a minor issue. **However**, Binary Copy performs **pre-validation of schema alignment** – and these nullability mismatches cause Binary Copy to fail.(Without this validation, the copied files become unreadable due to deserialization failures.)
So that we introduced `alignFieldsNullability` to achieve two critical goals:
1. ✅ Enable Binary Copy for **all Hudi write operations** (including `bulk insert`)
2. ✅ Synchronize **Table Schema** and **Data File Schema** nullability
---
### Experimental Verification
The experiments below demonstrate how `bulk insert as row` creates data files with **inconsistent Optional/Required attributes** :
Spark bulk insert as row
```
Schema:
message spark_schema {
optional binary _hoodie_commit_time (STRING);
optional binary _hoodie_commit_seqno (STRING);
optional binary _hoodie_record_key (STRING);
optional binary _hoodie_partition_path (STRING);
optional binary _hoodie_file_name (STRING);
required int32 id;
required int32 id2;
required group maptype (MAP) {
repeated group key_value {
required binary key (STRING);
optional binary value (STRING);
}
}
required group structtype {
optional binary name (STRING);
optional int32 age;
}
required binary arraytype (STRING);
required group newarraytype (LIST) {
repeated group list {
optional int32 element;
}
}
}
```
Spark insert/upsert
```
Schema:
message hoodie.app_jdr_normal_struct_and_map_a_d_d.app_jdr_normal_struct_and_map_a_d_d_record {
optional binary _hoodie_commit_time (STRING);
optional binary _hoodie_commit_seqno (STRING);
optional binary _hoodie_record_key (STRING);
optional binary _hoodie_partition_path (STRING);
optional binary _hoodie_file_name (STRING);
required int32 id;
optional int32 id2;
optional group maptype (MAP) {
repeated group key_value (MAP_KEY_VALUE) {
required binary key (STRING);
optional binary value (STRING);
}
}
optional group structtype {
optional binary name (STRING);
optional int32 age;
}
optional binary arraytype (STRING);
optional group newarraytype (LIST) {
repeated int32 array;
}
}
```
Flink insert/upsert
```
Schema:
message flink_schema {
optional binary _hoodie_commit_time (STRING);
optional binary _hoodie_commit_seqno (STRING);
optional binary _hoodie_record_key (STRING);
optional binary _hoodie_partition_path (STRING);
optional binary _hoodie_file_name (STRING);
required int32 id;
optional int32 id2;
optional group maptype (MAP) {
repeated group key_value {
required binary key (STRING);
optional binary value (STRING);
}
}
optional group structtype {
optional binary name (STRING);
optional int32 age;
}
optional binary arraytype (STRING);
optional group newarraytype (LIST) {
repeated group list {
optional int32 element;
}
}
}
```
**Overall**, Spark `insert`/`upsert` and Flink `insert`/`upsert` produce **consistent schemas**, while **Spark `bulk insert as row`** (and similar operations like Clustering) exhibit **inconsistent Optional/Required field attributes**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938400076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938151460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965)
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938914928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3f1ff569f1b1857ec1635febc2733f56581e4896 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974)
* 3327ebc046921fff138b889b8110a45b63b04ee2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914731714
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46aa20c12464c31375fadbe8c396adb5f4b008dd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2915747444
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2a61d993d8784144df7d28f80f0610aaad8bb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800)
* b8e88905c1ee696a98750b86c9b09edd2c19eb98 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124173633
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriterFactory.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+
+public class HoodieFileRewriterFactory {
+
+ private static HoodieFileRewriterFactory getWriterFactory(HoodieRecordType recordType, String extension) {
+ if (PARQUET.getFileExtension().equals(extension)) {
+ try {
+ Class<?> clazz = ReflectionUtils.getClass("org.apache.hudi.parquet.io.HoodieParquetRewriterFactory");
+ return (HoodieFileRewriterFactory) clazz.newInstance();
+ } catch (IllegalAccessException | IllegalArgumentException | InstantiationException e) {
+ throw new HoodieException("Unable to create hoodie avro parquet file writer factory", e);
+ }
+ }
+ throw new UnsupportedOperationException(extension + " file format not supported yet.");
+ }
+
+ public static <T, I, K, O> HoodieFileRewriter getFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ HoodieRecordType recordType,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ String extension = FSUtils.getFileExtension(targetFilePath.getName());
+ HoodieFileRewriterFactory factory = getWriterFactory(recordType, extension);
+ return factory.newFileRewriter(inputFilePaths, targetFilePath, conf, config, metadataMerger, writeSchemaWithMetaFields);
+ }
+
+ protected <T> HoodieFileRewriter newFileRewriter(
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124144940
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java:
##########
@@ -103,9 +103,9 @@ public HoodieCreateHandle(HoodieWriteConfig config, String instantTime, HoodieTa
partitionMetadata.trySave();
createMarkerFile(partitionPath,
FSUtils.makeBaseFileName(this.instantTime, this.writeToken, this.fileId, hoodieTable.getBaseFileExtension()));
- this.fileWriter =
- HoodieFileWriterFactory.getFileWriter(instantTime, path, hoodieTable.getStorage(), config,
- writeSchemaWithMetaFields, this.taskContextSupplier, config.getRecordMerger().getRecordType());
+ this.fileWriter = initWriter
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948829344
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981)
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2979533060
@zhangyue19921010 nice work.
Test with 1.7TB data, 300 columns
binary copy achived `7.7 mins` vs normal write `35 mins`; and cpu cost cut down from `90% -> 60%`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2975321819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190)
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005168783
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354)
* 7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2166977493
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java:
##########
@@ -81,7 +80,7 @@ public HoodieData<WriteStatus> performClusteringWithRecordsAsRow(Dataset<Row> in
Dataset<Row> repartitionedRecords = partitioner.repartitionRecords(inputRecords, numOutputGroups);
return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, instantTime, getHoodieTable(), newConfig,
- partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata, WriteOperationType.CLUSTER);
Review Comment:
No impact actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2969783638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 40d3cbc23a2ab560fc760fd92145630cf8cad0e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103)
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2156488261
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/BinaryCopyHandleFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Combine all input file paths quickly into a single large file at block level.
+ * Without performing extra operations like data serialization/deserialization or compression/decompression.
+ */
+public class BinaryCopyHandleFactory<T, I, K, O>
+ extends WriteHandleFactory<T, I, K, O> implements Serializable {
+
+ private final List<StoragePath> inputFilePaths;
+
+ public BinaryCopyHandleFactory(List<StoragePath> inputFilePaths) {
+ this.inputFilePaths = inputFilePaths;
+ }
+
+ @Override
+ public HoodieBinaryCopyHandle<T, I, K, O> create(
+ HoodieWriteConfig config,
+ String commitTime,
+ HoodieTable<T, I, K, O> hoodieTable,
+ String partitionPath,
+ String fileIdPrefix,
+ TaskContextSupplier taskContextSupplier) {
+ return new HoodieBinaryCopyHandle(
+ config,
+ commitTime,
Review Comment:
changed.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/BinaryCopyHandleFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import java.io.Serializable;
+import java.util.List;
+
+/**
+ * Combine all input file paths quickly into a single large file at block level.
+ * Without performing extra operations like data serialization/deserialization or compression/decompression.
+ */
+public class BinaryCopyHandleFactory<T, I, K, O>
+ extends WriteHandleFactory<T, I, K, O> implements Serializable {
+
+ private final List<StoragePath> inputFilePaths;
+
+ public BinaryCopyHandleFactory(List<StoragePath> inputFilePaths) {
+ this.inputFilePaths = inputFilePaths;
+ }
+
+ @Override
+ public HoodieBinaryCopyHandle<T, I, K, O> create(
+ HoodieWriteConfig config,
+ String commitTime,
+ HoodieTable<T, I, K, O> hoodieTable,
+ String partitionPath,
+ String fileIdPrefix,
+ TaskContextSupplier taskContextSupplier) {
+ return new HoodieBinaryCopyHandle(
+ config,
+ commitTime,
+ partitionPath,
+ getNextFileId(fileIdPrefix),
+ hoodieTable,
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3006913085
Comments addressed And all ci passed. PTAL @vinothchandar :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987876974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987251628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2987619831
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124161594
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/HoodieDynamicBoundedBloomFilter.java:
##########
@@ -125,5 +126,42 @@ private void extractAndSetInternalBloomFilter(DataInputStream dis) throws IOExce
internalDynamicBloomFilter = new InternalDynamicBloomFilter();
internalDynamicBloomFilter.readFields(dis);
}
+
+ @Override
+ public void or(BloomFilter other) {
Review Comment:
In TestHoodieFileMetadataMerger#testBloomFilter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2125157054
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetFileBinaryCopier.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.HoodieFileBinaryCopier;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.util.CompressionConverter;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.schema.MessageType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Queue;
+import java.util.Set;
+
+/**
+ * HoodieParquetFileBinaryCopier is a high-performance utility designed for efficient merging of Parquet files at the binary level.
+ * Unlike conventional Parquet writers, it bypasses costly data processing operations through a block-based approach:
+ *
+ * Core Capabilities:
+ * 1. Zero-Processing Merge
+ * Directly concatenates raw Parquet data blocks (row groups) from input files
+ * Avoids:
+ * 1) Data serialization/deserialization
+ * 2) Compression/decompression cycles
+ * 3) Record-level reprocessing
+ *
+ * 2. Metadata Reconstruction
+ * Dynamically rebuilds file metadata:
+ * 1) New footer with merged statistics
+ * 2) Updated row group offsets
+ * 3) Validated schema consistency
+ */
+public class HoodieParquetFileBinaryCopier extends HoodieParquetBinaryCopyBase implements HoodieFileBinaryCopier {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieParquetFileBinaryCopier.class);
+ private final CompressionCodecName codecName;
+
+ // Reader and relevant states of the in-processing input file
+ private Queue<CompressionConverter.TransParquetFileReader> inputFiles = new LinkedList<>();
+
+ private Map<String, String> extraMetaData = new HashMap<>();
+
+ // The reader for the current input file
+ private CompressionConverter.TransParquetFileReader reader = null;
+
+ private HoodieFileMetadataMerger metadataMerger;
+
+ public HoodieParquetFileBinaryCopier(
+ Configuration conf,
+ CompressionCodecName codecName,
+ HoodieFileMetadataMerger metadataMerger) {
+ super(conf);
+ this.metadataMerger = metadataMerger;
+ this.codecName = codecName;
+ }
+
+ @Override
+ protected Map<String, String> finalizeMetadata() {
+ return this.extraMetaData;
+ }
+
+ /**
+ * Merge all inputFilePaths to outputFilePath at block level
+ */
+ @Override
+ public long binaryCopy(List<StoragePath> inputFilePaths,
+ List<StoragePath> outputFilePath,
+ MessageType writeSchema,
+ Properties props) throws IOException {
+ openInputFiles(inputFilePaths, conf);
+ initFileWriter(new Path(outputFilePath.get(0).toUri()), codecName, writeSchema);
+ initNextReader();
+
+ Set<String> allOriginalCreatedBys = new HashSet<>();
+ while (reader != null) {
+ List<BlockMetaData> rowGroups = reader.getRowGroups();
+ FileMetaData fileMetaData = reader.getFooter().getFileMetaData();
+ String createdBy = fileMetaData.getCreatedBy();
+ allOriginalCreatedBys.add(createdBy);
+ Map<String, String> metaMap = fileMetaData.getKeyValueMetaData();
+ metadataMerger.mergeMetaData(metaMap);
+
+ for (BlockMetaData block : rowGroups) {
+ processBlocksFromReader(reader, reader.readNextRowGroup(), block, createdBy);
+ }
+ initNextReader();
+ }
+ extraMetaData.putAll(metadataMerger.getMergedMetaData());
+ extraMetaData.put(ORIGINAL_CREATED_BY_KEY, String.join("\n", allOriginalCreatedBys));
+ return totalRecordsWritten;
+ }
+
+ // Open all input files to validate their schemas are compatible to merge
+ private void openInputFiles(List<StoragePath> inputFiles, Configuration conf) {
+ Preconditions.checkArgument(inputFiles != null && !inputFiles.isEmpty(), "No input files");
+
+ for (StoragePath inputFile : inputFiles) {
+ try {
+ CompressionConverter.TransParquetFileReader reader = new CompressionConverter.TransParquetFileReader(
+ HadoopInputFile.fromPath(new Path(inputFile.toUri()), conf),
+ HadoopReadOptions.builder(conf).build());
+ this.inputFiles.add(reader);
+ } catch (IOException e) {
+ throw new IllegalArgumentException("Failed to open input file: " + inputFile, e);
+ }
+ }
+ }
+
+ // Routines to get reader of next input file and set up relevant states
+ private void initNextReader() {
+ if (reader != null) {
Review Comment:
did we close the last reader properly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2166979297
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java:
##########
@@ -1017,6 +1017,14 @@ public void testSimpleClustering(boolean populateMetaFields) throws Exception {
false, SqlQueryEqualityPreCommitValidator.class.getName(), COUNT_SQL_QUERY_FOR_VALIDATION, "");
}
+ @Test
+ public void testBinaryCopyClustering() throws Exception {
+ String strategy = "org.apache.hudi.client.clustering.run.strategy.SparkBinaryCopyClusteringExecutionStrategy";
+ HoodieClusteringConfig config = createClusteringBuilder(true, 1).withClusteringExecutionStrategyClass(strategy).build();
+ testInsertAndClustering(config, true, true,
+ false, SqlQueryEqualityPreCommitValidator.class.getName(), COUNT_SQL_QUERY_FOR_VALIDATION, "");
Review Comment:
Already have records check in `testInsertAndClustering` func when set complete clustering true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005404595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448)
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3092504417
@zhangyue19921010 I just add a change to my personal branch https://github.com/shangxinli/hudi/commit/5fbfea4a319018ed0c107aca21ba4928b55db471. This change provides the Parquet API that takes list of files as input and output one file. Internally, it validates all files need to have the same schema.
mergeFiles(List<Path> inputFiles, Path outputFile)
You can port your change of the clustering strategy code on top it and implement a groupBy function to facilitate it. Let me know if you want me to check in the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2939811926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914727819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46aa20c12464c31375fadbe8c396adb5f4b008dd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111580622
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
+1, the naming is confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111561557
##########
hudi-common/src/main/java/org/apache/hudi/common/config/TypedProperties.java:
##########
@@ -39,7 +39,7 @@ public TypedProperties() {
super(null);
}
- protected TypedProperties(Properties defaults) {
+ public TypedProperties(Properties defaults) {
Review Comment:
The scope was restricted intentionally to avoid unnecessary copy overhead, use `TypedProperties.copy` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2957287583
> @zhangyue19921010 please mention me again, once you have addressed the feedback.
hi @vinothchandar all comments are addressed, added uts passed PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2138025106
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
This data class correction is necessary. The existing "bulk insert as row" operation fetches the DataFrame's schema(may inconsistencies with table schema). This inconsistencies in the nullable properties of identical columns lead to file corruption/unreadability after binary copying. Therefore, we've implemented a nullable type correction for the bulk insert as row operation to resolve this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2969892092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 40d3cbc23a2ab560fc760fd92145630cf8cad0e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103)
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111556699
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
I guess it is impossible to do so, because the copy breaks the sort layout of the target file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2915940439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8e88905c1ee696a98750b86c9b09edd2c19eb98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2935818577
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963)
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124124861
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
Thanks for your attention here. In this new strategy It only supports file-level sorting. For instance, in our internally implemented Hudi-LSM, data files are strictly ordered and deduplicated. For such files, this new strategy can be applied to perform sorted merging – prioritizing files with smaller record keys during reading.
Also let us make this strategy extends `SingleSparkJobExecutionStrategy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124195103
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Version;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.CompressionConverter.TransParquetFileReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.internal.column.columnindex.ColumnIndex;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.FILENAME_METADATA_FIELD;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class TestHoodieParquetFileRewriter {
+
+ private final int numRecord = 1;
+ private Configuration conf = new Configuration();
+ private List<TestFile> inputFiles = null;
+ private String outputFile = null;
+ private HoodieParquetFileRewriter rewriter = null;
+
+ @BeforeEach
+ public void setUp() {
+ outputFile = TestFileBuilder.createTempFile("test");
+ }
+
+ @AfterEach
+ public void after() {
+ if (outputFile != null) {
+ TestFileBuilder.deleteTempFile(outputFile);
+ }
+ if (inputFiles != null) {
+ inputFiles.stream().map(TestFile::getFileName).forEach(TestFileBuilder::deleteTempFile);
+ }
+ }
+
+ @Test
+ public void testBasic() throws Exception {
+ MessageType schema = createSchema();
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+
+ rewriter = parquetFileRewriter(schema, "GZIP");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+ validateSchema(fileSchema);
+
+ // Verify codec
+ verifyCodec(outputFile, CompressionCodecName.GZIP);
+
+ // Verify the merged data are not changed
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2, 3, 4);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testTranslateCodec() throws Exception {
+ MessageType schema = createSchema();
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "UNCOMPRESSED"));
+
+ rewriter = parquetFileRewriter(schema, "ZSTD");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+ validateSchema(fileSchema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.ZSTD);
+
+ // Verify the data are not changed for the columns not pruned
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2, 3, 4);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testDifferentSchema() throws Exception {
+ MessageType schema1 = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"),
+ new GroupType(OPTIONAL, "Links",
+ new PrimitiveType(REPEATED, BINARY, "Backward"),
+ new PrimitiveType(REPEATED, BINARY, "Forward")));
+ MessageType schema2 = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"));
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema1, "UNCOMPRESSED"));
+ inputFiles.add(makeTestFile(schema2, "UNCOMPRESSED"));
+
+ rewriter = parquetFileRewriter(schema1, "UNCOMPRESSED");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType schema = pmd.getFileMetaData().getSchema();
+ validateSchema(schema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.UNCOMPRESSED);
+
+ // Verify the data are not changed
+ validateColumnData();
+
+ // Verify the page index
+ validatePageIndex(0, 1, 2);
+
+ // Verify original.created.by is preserved
+ validateCreatedBy();
+ }
+
+ @Test
+ public void testHoodieMetaColumn() throws Exception {
+ MessageType schema = new MessageType("schema",
+ new PrimitiveType(OPTIONAL, BINARY, FILENAME_METADATA_FIELD),
+ new PrimitiveType(OPTIONAL, INT64, "DocId"),
+ new PrimitiveType(REQUIRED, BINARY, "Name"),
+ new PrimitiveType(OPTIONAL, BINARY, "Gender"),
+ new GroupType(OPTIONAL, "Links",
+ new PrimitiveType(REPEATED, BINARY, "Backward"),
+ new PrimitiveType(REPEATED, BINARY, "Forward")));
+ inputFiles = new ArrayList<>();
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+ inputFiles.add(makeTestFile(schema, "GZIP"));
+
+ rewriter = parquetFileRewriter(schema, "GZIP");
+ rewriter.rewrite();
+ rewriter.close();
+
+ // Verify the schema are not changed for the columns not pruned
+ ParquetMetadata pmd = ParquetFileReader.readFooter(conf, new Path(outputFile), ParquetMetadataConverter.NO_FILTER);
+ MessageType fileSchema = pmd.getFileMetaData().getSchema();
+ assertEquals(schema, fileSchema);
+
+ // Verify codec has been translated
+ verifyCodec(outputFile, CompressionCodecName.GZIP);
+
+ // Verify the data are not changed
+ validateColumnData();
Review Comment:
_hoodie_file_name will changed to new parquet file name, but _hoodie_commit_time will not be changed. This behavior is consistent with traditional clustering
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2935944716
Sorry for the late response because of preparing JD's Hudi system for the 618 Shopping Festival over the past several days. Thank you all for your reviews and assistance. All comments have been addressed except for more unit tests related to `HoodieParquetFileBinaryCopier`—still working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2939576945
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2138015213
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -268,4 +267,106 @@ object AvroConversionUtils {
}
schema
}
+
+ /**
+ * Recursively aligns the nullable property of hoodie table schema, supporting nested structures
+ */
+ def alignFields(sourceSchema: StructType, avroSchema: Schema): StructType = {
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2131850415
##########
pom.xml:
##########
@@ -2604,6 +2609,7 @@
<orc.flink.version>1.5.6</orc.flink.version>
<flink.avro.version>1.11.4</flink.avro.version>
<flink.format.parquet.version>1.12.2</flink.format.parquet.version>
+ <parquet.version>${flink.format.parquet.version}</parquet.version>
Review Comment:
Make sense Danny. Resolve this issue through https://github.com/apache/hudi/pull/13396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2939576681
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2975362067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190)
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005174082
> Is the feature able to be used by Flink clustering? Any thing needs to be changed in hudi-flink-datasource?
After this PR, **Binary Copy capability** can be easily enabled for Flink Clustering operations by simply using the `BinaryCopierHandler` during clustering execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2166976796
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.parquet.io.ParquetBinaryCopyChecker;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+import static org.apache.hudi.config.HoodieClusteringConfig.PLAN_STRATEGY_SORT_COLUMNS;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+
+ List<ClusteringGroupInfo> clusteringGroupInfos = clusteringPlan.getInputGroups()
+ .stream()
+ .map(ClusteringGroupInfo::create)
+ .collect(Collectors.toList());
+ if (!supportBinaryStreamCopy(clusteringGroupInfos, clusteringPlan.getStrategy().getStrategyParams())) {
+ LOG.info("Required conditions for binary stream copy are currently not satisfied, falling back to default clustering behavior");
+ HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder().withProperties(writeConfig.getProps())
+ .withStorageConfig(HoodieStorageConfig.newBuilder().parquetWriteLegacyFormat("false").build()).build();
+ resetWriteConfig(newConfig);
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+ TaskContextSupplier taskContextSupplier = getEngineContext().getTaskContextSupplier();
+ SerializableSchema serializableSchema = new SerializableSchema(schema);
+ boolean shouldPreserveMetadata = Option.ofNullable(clusteringPlan.getPreserveHoodieMetadata()).orElse(false);
+ JavaRDD<ClusteringGroupInfo> groupInfoJavaRDD = engineContext.parallelize(clusteringGroupInfos, clusteringGroupInfos.size());
+ LOG.info("number of partitions for clustering " + groupInfoJavaRDD.getNumPartitions());
+ JavaRDD<WriteStatus> writeStatusRDD = groupInfoJavaRDD
+ .mapPartitions(clusteringOps -> {
+ Iterable<ClusteringGroupInfo> clusteringOpsIterable = () -> clusteringOps;
+ return StreamSupport.stream(clusteringOpsIterable.spliterator(), false)
+ .flatMap(clusteringOp ->
+ runClusteringForGroup(
+ clusteringOp,
+ clusteringPlan.getStrategy().getStrategyParams(),
+ shouldPreserveMetadata,
+ serializableSchema,
+ taskContextSupplier,
+ instantTime))
+ .iterator();
+ });
+
+ HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = new HoodieWriteMetadata<>();
+ writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
+ return writeMetadata;
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ private Stream<WriteStatus> runClusteringForGroup(ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<WriteStatus> statuses = new ArrayList<>();
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ statuses.addAll(handler.close());
+ return statuses.stream();
+ }
+
+ /**
+ * 1. Check Table type
+ * 2. Check bloom filter type code
+ * 3. Check Array Type Schema consistency affected by hoodie.parquet.writelegacyformat.enabled and spark.hadoop.parquet.avro.write-old-list-structure
+ * 4. Check Schema Optional or Required consistency for the same field
+ */
+ public boolean supportBinaryStreamCopy(List<ClusteringGroupInfo> inputGroups, Map<String, String> strategyParams) {
+ if (getHoodieTable().getMetaClient().getTableType() != COPY_ON_WRITE) {
+ LOG.warn("Only support CoW table. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+ Option<String[]> orderByColumnsOpt =
+ Option.ofNullable(strategyParams.get(PLAN_STRATEGY_SORT_COLUMNS.key()))
+ .map(listStr -> listStr.split(","));
+
+ if (orderByColumnsOpt.isPresent()) {
+ LOG.warn("Not support to sort input records. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+
+ List<ParquetBinaryCopyChecker.ParquetFileInfo> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
+ .flatMap(group -> group.getOperations().iterator())
+ .map(op -> {
+ String filePath = op.getDataFilePath();
+ if (!filePath.endsWith(PARQUET.getFileExtension())) {
+ return new ParquetBinaryCopyChecker.ParquetFileInfo(false, null, null);
Review Comment:
Yes we need to fall back for not parquet format for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2166986439
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
Review Comment:
rename hudi-common/src/main/java/org/apache/hudi/{io/storage => util}/HoodieFileMetadataMerger.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111377439
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
Review Comment:
@zhangyue19921010 Hi, will `sort` be supported in the future?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111576068
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
Review Comment:
How about inherited from ` SingleSparkJobExecutionStrategy `? It seems that you inherited from `MultipleSparkJobExecutionStrategy` and rewrote `performClustering` to submit only one spark job, and then mapped it into individual tasks according to the `ClusteringGroupInfo`. This logic has actually been implemented in `SingleSparkJobExecutionStrategy`, and you only need to rewrite the execution logic of each group, `SingleSparkJobExecutionStrategy#performClusteringForGroup`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914839009
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46aa20c12464c31375fadbe8c396adb5f4b008dd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2915753756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2a61d993d8784144df7d28f80f0610aaad8bb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800)
* b8e88905c1ee696a98750b86c9b09edd2c19eb98 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111559799
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/HoodieDynamicBoundedBloomFilter.java:
##########
@@ -125,5 +126,42 @@ private void extractAndSetInternalBloomFilter(DataInputStream dis) throws IOExce
internalDynamicBloomFilter = new InternalDynamicBloomFilter();
internalDynamicBloomFilter.readFields(dis);
}
+
+ @Override
+ public void or(BloomFilter other) {
+ if (other != null) {
+ ValidationUtils.checkArgument(other instanceof HoodieDynamicBoundedBloomFilter,
+ "HoodieDynamicBoundedBloomFilter can only perform OR operations with other HoodieDynamicBoundedBloomFilter.");
+ HoodieDynamicBoundedBloomFilter otherFilter = (HoodieDynamicBoundedBloomFilter) other;
+ int sourceMatrixLength = this.getMatrixLength();
+ int targetMatrixLength = otherFilter.getMatrixLength();
+ if (targetMatrixLength > sourceMatrixLength) {
+ this.rescaleFromTarget(targetMatrixLength);
+ } else {
+ otherFilter.rescaleFromTarget(sourceMatrixLength);
+ }
+ this.internalDynamicBloomFilter.or(otherFilter.internalDynamicBloomFilter);
+ }
+ }
+
+ /**
+ * rescale the internal dynamic bloom filter by length
+ * @param targetMatrixLength
+ * @return
+ */
+ private HoodieDynamicBoundedBloomFilter rescaleFromTarget(int targetMatrixLength) {
+ int initMatrixLength = this.internalDynamicBloomFilter.getMatrixLength();
+ int needAddRowNum = targetMatrixLength - initMatrixLength;
+ if (needAddRowNum > 0) {
+ for (int i = 0; i < needAddRowNum; i++) {
+ this.internalDynamicBloomFilter.addRow();
Review Comment:
Not sure if we can rebuild the filter in one shot instead of invoking the `addRow` multiple times?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2957208413
@zhangyue19921010 please mention me again, once you have addressed the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2138134779
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
Review Comment:
After careful consideration, I believe we still need to extend the SparkSortAndSizeExecutionStrategy method. The reasons are as follows:
Currently, due to the need to maintain compatibility with various existing data types (e.g., legacy array types), we require a pre-check named supportBinaryStreamCopy. In cases where streaming copy cannot be used, we need to automatically fall back to SparkSortAndSizeExecutionStrategy to avoid task failures while remaining transparent to users.
Regarding the sorting issue, it is true that the new strategy only supports file-level sorting and cannot achieve record-level sorting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2144688749
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SingleSparkJobExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+ if (!supportBinaryStreamCopy()) {
+ String message = "1. Only support for CoW table 2. Not support for sort function";
+ throw new UnsupportedOperationException(message);
+
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ @Override
+ protected List<WriteStatus> performClusteringForGroup(ReaderContextFactory<T> readerContextFactory, ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ return new ArrayList<>(handler.close());
+ }
+
+ private boolean supportBinaryStreamCopy() {
+ return this.getHoodieTable().getMetaClient().getTableType() == COPY_ON_WRITE;
Review Comment:
@gudladona Sorry for missing this comment.
For Mor + insert, maybe we need to check all the clustering operations are base file only, so that we can trigger this binary copy. Will support in followed pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3166251041
> @shangxinli @zhangyue19921010 : you both got the follow ups?
Yep, PR https://github.com/apache/hudi/pull/13674 is under discuss and review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2126313492
##########
pom.xml:
##########
@@ -2604,6 +2609,7 @@
<orc.flink.version>1.5.6</orc.flink.version>
<flink.avro.version>1.11.4</flink.avro.version>
<flink.format.parquet.version>1.12.2</flink.format.parquet.version>
+ <parquet.version>${flink.format.parquet.version}</parquet.version>
Review Comment:
it's about changed the parquet version for other model like hudi-hadoop-common. Without this config compile will failed with followed errors
```
....
Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-common: Compilation failure: Compilation failure:
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[37,52] package org.apache.parquet.column.values.bloomfilter does not exist
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[39,33] package org.apache.parquet.crypto does not exist
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[40,33] cannot find symbol
Error: symbol: class BlockCipher
Error: location: package org.apache.parquet.format
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[47,33] org.apache.parquet.hadoop.ColumnChunkPageWriteStore is not public in org.apache.parquet.hadoop; cannot be accessed from outside package
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[54,38] cannot find symbol
Error: symbol: class CompressionConverter
Error: location: package org.apache.parquet.hadoop.util
...
```
##########
pom.xml:
##########
@@ -2604,6 +2609,7 @@
<orc.flink.version>1.5.6</orc.flink.version>
<flink.avro.version>1.11.4</flink.avro.version>
<flink.format.parquet.version>1.12.2</flink.format.parquet.version>
+ <parquet.version>${flink.format.parquet.version}</parquet.version>
Review Comment:
it's about changed the parquet version for other model like hudi-hadoop-common. Without this config compile will failed with followed errors
```
....
Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-hadoop-common: Compilation failure: Compilation failure:
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[37,52] package org.apache.parquet.column.values.bloomfilter does not exist
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[39,33] package org.apache.parquet.crypto does not exist
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[40,33] cannot find symbol
Error: symbol: class BlockCipher
Error: location: package org.apache.parquet.format
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[47,33] org.apache.parquet.hadoop.ColumnChunkPageWriteStore is not public in org.apache.parquet.hadoop; cannot be accessed from outside package
Error: /home/runner/work/hudi/hudi/hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetBinaryCopyBase.java:[54,38] cannot find symbol
Error: symbol: class CompressionConverter
Error: location: package org.apache.parquet.hadoop.util
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124172890
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY;
+
+public class HoodieFileMetadataMerger {
+
+ private final HashMap<String, String> mergedMateData = new HashMap<>();
+
+ private BloomFilter bloomFilter;
+
+ private String minRecordKey;
+
+ private String maxRecordKey;
+
+ public HoodieFileMetadataMerger() {
+
+ }
+
+ public Map<String, String> mergeMetaData(Map<String, String> metaMap) {
Review Comment:
in `hudi-common/src/test/java/org/apache/hudi/io/storage/TestHoodieFileMetadataMerger.java`
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriterFactory.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import org.apache.hudi.common.config.HoodieConfig;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieRecord.HoodieRecordType;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+
+public class HoodieFileRewriterFactory {
+
+ private static HoodieFileRewriterFactory getWriterFactory(HoodieRecordType recordType, String extension) {
+ if (PARQUET.getFileExtension().equals(extension)) {
+ try {
+ Class<?> clazz = ReflectionUtils.getClass("org.apache.hudi.parquet.io.HoodieParquetRewriterFactory");
+ return (HoodieFileRewriterFactory) clazz.newInstance();
+ } catch (IllegalAccessException | IllegalArgumentException | InstantiationException e) {
+ throw new HoodieException("Unable to create hoodie avro parquet file writer factory", e);
+ }
+ }
+ throw new UnsupportedOperationException(extension + " file format not supported yet.");
+ }
+
+ public static <T, I, K, O> HoodieFileRewriter getFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ HoodieRecordType recordType,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ String extension = FSUtils.getFileExtension(targetFilePath.getName());
+ HoodieFileRewriterFactory factory = getWriterFactory(recordType, extension);
+ return factory.newFileRewriter(inputFilePaths, targetFilePath, conf, config, metadataMerger, writeSchemaWithMetaFields);
+ }
+
+ protected <T> HoodieFileRewriter newFileRewriter(
+ List<StoragePath> inputFilePaths,
+ StoragePath targetFilePath,
+ Configuration conf,
+ HoodieConfig config,
+ HoodieFileMetadataMerger metadataMerger,
+ Schema writeSchemaWithMetaFields) throws IOException {
+ throw new UnsupportedOperationException();
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124198769
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestFileBuilder.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.crypto.ParquetCipher;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.example.GroupWriteSupport;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ThreadLocalRandom;
+
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT32;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+
+public class TestFileBuilder {
Review Comment:
Move into `hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileBinaryCopier.java`
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestFile.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.parquet.example.data.simple.SimpleGroup;
+
+public class TestFile {
Review Comment:
Move into `hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileBinaryCopier.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124131176
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
Review Comment:
1. Rename this class `public class HoodieBinaryCopyHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O> `
2. Java doc added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2125153276
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieFileBinaryCopier.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.parquet.schema.MessageType;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * HoodieFileBinaryCopier is a high-performance utility designed for efficient merging of data files at the binary level.
+ * Unlike conventional hoodie writers, it bypasses costly data processing operations through a block-based approach, like:
+ * 1. Merge multi-parquet files at rowgroup level
+ * 2. Merge multi-orc files at stripe level
+ */
+public interface HoodieFileBinaryCopier {
Review Comment:
HoodieFileBinaryCopier -> HoodieFileBinaryCopyWriter?
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieFileBinaryCopier.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.parquet.schema.MessageType;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Properties;
+
+/**
+ * HoodieFileBinaryCopier is a high-performance utility designed for efficient merging of data files at the binary level.
+ * Unlike conventional hoodie writers, it bypasses costly data processing operations through a block-based approach, like:
+ * 1. Merge multi-parquet files at rowgroup level
+ * 2. Merge multi-orc files at stripe level
+ */
+public interface HoodieFileBinaryCopier {
Review Comment:
HoodieFileBinaryCopier -> HoodieFileBinaryCopyWriter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2939286493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948841777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 90115910fb597ab25f3d4111d4f184727c5e1b10 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948973780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 90115910fb597ab25f3d4111d4f184727c5e1b10 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033)
* 7d5efa6090bbca6e5d479106f05060cac4be9089 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2156505752
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/InternalDynamicBloomFilter.java:
##########
@@ -210,13 +210,29 @@ public void readFields(DataInput in) throws IOException {
/**
* Adds a new row to <i>this</i> dynamic Bloom filter.
*/
- private void addRow() {
+ protected void addRow() {
InternalBloomFilter[] tmp = new InternalBloomFilter[matrix.length + 1];
System.arraycopy(matrix, 0, tmp, 0, matrix.length);
tmp[tmp.length - 1] = new InternalBloomFilter(vectorSize, nbHash, hashType);
matrix = tmp;
}
+ protected void addRows(int size) {
Review Comment:
added TestInternalDynamicBloomFilter#testInternalDynamicBloomFilterRescale which cover `rescaleFromTarget`, `getMatrixLength` and `addRows`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2970138603
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2149190686
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/ParquetBinaryCopyChecker.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY;
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_BLOOM_FILTER_TYPE_CODE;
+import static org.apache.hudi.common.bloom.BloomFilterTypeCode.SIMPLE;
+
+public class ParquetBinaryCopyChecker {
+
+ private ParquetBinaryCopyChecker() {
+
+ }
+
+ /**
+ * Verify whether a set of files meet the conditions for binary stream copying
+ * 1. All input parquet file schema support binary copy
+ * 2. This set of files contains only one type of BloomFilterTypeCode, including null
+ * 3. The same column across these files has only one repetition type
+ * @param files
+ * @return
+ */
+ public static boolean verifyFiles(List<ParquetFileInfo> files) {
+ boolean schemaSupportBinaryCopy = files.stream().allMatch(ParquetFileInfo::isSchemaSupport);
+ if (!schemaSupportBinaryCopy) {
+ return false;
+ }
+
+ boolean hasSameFilterCodeType = files.stream()
+ .map(ParquetFileInfo::getBloomFilterTypeCode)
+ .distinct()
+ .count() <= 1;
+ if (!hasSameFilterCodeType) {
+ return false;
+ }
+
+ Map<String, Set<String>> fieldsMap = collectRepetitions(files);
+ return fieldsMap.values().stream().allMatch(reps -> reps.size() == 1);
+ }
+
+ private static Map<String, Set<String>> collectRepetitions(List<ParquetFileInfo> files) {
+ Map<String, Set<String>> fieldsMap = new HashMap<>();
+ for (ParquetFileInfo file : files) {
+ Map<String, String> fields = file.getRepetitions();
+ fields.forEach((field, repetition) -> {
+ Set<String> repetitions = fieldsMap.computeIfAbsent(field, f -> new HashSet<>());
+ repetitions.add(repetition);
+ fieldsMap.put(field, repetitions);
+ });
+ }
+ return fieldsMap;
+ }
+
+ /**
+ * Preliminary inspection of a single file
+ * 1. verify file schema support binary copy or not
+ * 2. get hoodie bloom filter type code
+ * 3. get repetition of every field
+ * @param conf
+ * @param file
+ * @return
+ */
+ public static ParquetFileInfo verifyFile(Configuration conf, String file) {
+ Path path = new Path(file);
+ ParquetMetadata footer = readMetadata(conf, path, ParquetMetadataConverter.SKIP_ROW_GROUPS);
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ MessageType fileSchema = fileMetaData.getSchema();
+ List<Type> fields = fileSchema.getFields();
+ if (schemaNotSupportBinaryCopy(fields)) {
+ return new ParquetFileInfo(false, null, Collections.EMPTY_MAP);
+ }
+
+ Map<String, String> keyValueMetaData = fileMetaData.getKeyValueMetaData();
+ String bloomFileTypeCode = keyValueMetaData.get(HOODIE_BLOOM_FILTER_TYPE_CODE);
+ if (bloomFileTypeCode == null && keyValueMetaData.get(HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY) != null) {
+ bloomFileTypeCode = SIMPLE.name();
+ }
+ Map<String, String> repetitionMap = fields.stream()
+ .collect(Collectors.toMap(Type::getName, type -> type.getRepetition().name()));
+ return new ParquetFileInfo(true, bloomFileTypeCode, repetitionMap);
+ }
+
+ private static ParquetMetadata readMetadata(
+ Configuration conf,
+ Path parquetFilePath,
+ ParquetMetadataConverter.MetadataFilter filter) {
+ ParquetMetadata footer;
+ try {
+ footer = ParquetFileReader.readFooter(conf, parquetFilePath, filter);
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to read footer for parquet " + parquetFilePath, e);
+ }
+ return footer;
+ }
+
+ /**
+ * Check whether input schema not supports binary copy
+ * Following two case can not support
+ * 1. two level List structure, because the result of parquet rewrite is three level List structure
+ * 2. Decimal types stored via INT32/INT64/INT96, because it can not be read by parquet-avro
+ *
+ * @param parquetFields
+ * @return
+ */
+ private static boolean schemaNotSupportBinaryCopy(List<Type> parquetFields) {
+ for (Type type : parquetFields) {
+ if (type.getOriginalType() == OriginalType.DECIMAL) {
+ PrimitiveType primitiveType = type.asPrimitiveType();
+ PrimitiveType.PrimitiveTypeName typeName = primitiveType.getPrimitiveTypeName();
+ if (typeName == PrimitiveType.PrimitiveTypeName.INT32
+ || typeName == PrimitiveType.PrimitiveTypeName.INT64
+ || typeName == PrimitiveType.PrimitiveTypeName.INT96) {
+ return true;
+ }
+ }
+ if (!type.isPrimitive()) {
+ GroupType groupType = type.asGroupType();
+ OriginalType originalType = groupType.getOriginalType();
+ if (originalType == OriginalType.LIST
+ && groupType.getType(0).getName().equals("array")) {
+ return true;
+ }
+ if (schemaNotSupportBinaryCopy(groupType.getFields())) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ public static class ParquetFileInfo implements Serializable {
+ private final boolean isSchemaSupport;
+ private final String bloomFilterTypeCode;
+ private final Map<String, String> repetitions;
+
+ public ParquetFileInfo(boolean isSchemaSupport, String hoodieBloomFilterTypeCode, Map<String, String> repetitions) {
+ this.isSchemaSupport = isSchemaSupport;
+ this.bloomFilterTypeCode = hoodieBloomFilterTypeCode;
+ this.repetitions = repetitions;
+ }
+
+ public boolean isSchemaSupport() {
Review Comment:
added.
The verification of binary copy is divided into two parts.
First, internal file verification does not support binary copy in the two scenarios mentioned.
```
/**
* Current file can use binary copy or not
* Following two case can not support
* 1. two level List structure, because the result of parquet rewrite is three level List structure
* 2. Decimal types stored via INT32/INT64/INT96, because it can not be read by parquet-avro
*
* @param parquetFields
* @return
*/
public boolean canBinaryCopy() {
return canBinaryCopy;
}
```
Second, cross-file verification also does not support binary copy in these scenarios.
```
/**
* Verify whether a set of files meet the conditions for binary stream copying
* 1. All input parquet file schema support binary copy
* 2. This set of files contains only one type of BloomFilterTypeCode, including null
* 3. The same column across these files has only one repetition type
* @param files
* @return
*/
public static boolean verifyFiles(List<ParquetFileInfo> files) {
```
In summary, strict verification is required for historical stock data (since the schema of historical stock data is not strictly aligned).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2975336242
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190)
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3165642992
@shangxinli @zhangyue19921010 : you both got the follow ups?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2934712992
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8e88905c1ee696a98750b86c9b09edd2c19eb98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803)
* 6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124190795
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.util.CompressionConverter;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.schema.MessageType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+
+public class HoodieParquetFileRewriter extends HoodieParquetRewriterBase implements HoodieFileRewriter {
Review Comment:
Yes, it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124143964
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/rewrite/HoodieFileRewriter.java:
##########
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage.rewrite;
+
+import java.io.IOException;
+
+public interface HoodieFileRewriter {
Review Comment:
Maybe this interface is useful here. Since parquet and orc both can implement Binary Copy
1. Binary copy parquet at row group level
2. Binary copy orc at stripe level.
Also as VC mentioned. Change the function to
` long binaryCopy(List<StoragePath> inputFilePaths, List<StoragePath> outputFilePath, MessageType writeSchema, Properties props) throws IOException;
`
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938909665
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3f1ff569f1b1857ec1635febc2733f56581e4896 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974)
* 3327ebc046921fff138b889b8110a45b63b04ee2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2949315619
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 7d5efa6090bbca6e5d479106f05060cac4be9089 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "xiarixiaoyao (via GitHub)" <gi...@apache.org>.
xiarixiaoyao commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2958992893
@zhangyue19921010 @danny0405 pls check the match of table_schema and parquet schema before do binary copy.
Spark BulkInsert will produce parquets with required attribute； however those columns in hudi itself are optional。
Do binary copy will corrupt parquet file which is a serious problem https://github.com/apache/spark/blob/1a3ae66c6c48bb319f0798826085e694fa7d0b58/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java#L389。
Suggest: throw exception directly when found above mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3018901372
> Sorry for the late comments. The PR has been merged. Maybe consider these points for future improvements:
>
> Handling schema evolution and masking columns involves writing a lot of low-level Parquet code and introduces some complexity. If we can group files by schema, then we can merge only files with the same schema, which would help avoid this complexity.
>
> There was a file merging functionality in [parquet-java](https://github.com/apache/parquet-java/blob/parquet-1.11.x/parquet-tools/src/main/java/org/apache/parquet/tools/command/MergeCommand.java) that we could potentially reuse. The code has been tested and used in production. This command was later removed when the entire parquet-tools was deprecated, but we could consider bringing it back. At least most of the core implementations, such as appendFile() and mergeMetadataFiles(), still exist.
Thanks @shangxinli groupby schema is a great idea, which save a lot of data compatibility verification work. Will have a deep
look give a new clustering plan based on that asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2998696708
Is the feature able to be used by Flink clustering? Any thing needs to be changed in hudi-flink-datasource?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914811642
> Please block one review from either me or @yihua .. to ensure the changes play well with how we are thinking about changing the spark writer path
Okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2111554197
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieCreateRewriteHandle.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriter;
+import org.apache.hudi.io.storage.rewrite.HoodieFileRewriterFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.hadoop.conf.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.List;
+
+public class HoodieCreateRewriteHandle<T, I, K, O> extends HoodieCreateHandle<T, I, K, O> {
Review Comment:
Also add some doc to this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2146292970
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/ParquetBinaryCopyChecker.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.OriginalType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY;
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_BLOOM_FILTER_TYPE_CODE;
+import static org.apache.hudi.common.bloom.BloomFilterTypeCode.SIMPLE;
+
+public class ParquetBinaryCopyChecker {
+
+ private ParquetBinaryCopyChecker() {
+
+ }
+
+ /**
+ * Verify whether a set of files meet the conditions for binary stream copying
+ * 1. All input parquet file schema support binary copy
+ * 2. This set of files contains only one type of BloomFilterTypeCode, including null
+ * 3. The same column across these files has only one repetition type
+ * @param files
+ * @return
+ */
+ public static boolean verifyFiles(List<ParquetFileInfo> files) {
+ boolean schemaSupportBinaryCopy = files.stream().allMatch(ParquetFileInfo::isSchemaSupport);
+ if (!schemaSupportBinaryCopy) {
+ return false;
+ }
+
+ boolean hasSameFilterCodeType = files.stream()
+ .map(ParquetFileInfo::getBloomFilterTypeCode)
+ .distinct()
+ .count() <= 1;
+ if (!hasSameFilterCodeType) {
+ return false;
+ }
+
+ Map<String, Set<String>> fieldsMap = collectRepetitions(files);
+ return fieldsMap.values().stream().allMatch(reps -> reps.size() == 1);
+ }
+
+ private static Map<String, Set<String>> collectRepetitions(List<ParquetFileInfo> files) {
+ Map<String, Set<String>> fieldsMap = new HashMap<>();
+ for (ParquetFileInfo file : files) {
+ Map<String, String> fields = file.getRepetitions();
+ fields.forEach((field, repetition) -> {
+ Set<String> repetitions = fieldsMap.computeIfAbsent(field, f -> new HashSet<>());
+ repetitions.add(repetition);
+ fieldsMap.put(field, repetitions);
+ });
+ }
+ return fieldsMap;
+ }
+
+ /**
+ * Preliminary inspection of a single file
+ * 1. verify file schema support binary copy or not
+ * 2. get hoodie bloom filter type code
+ * 3. get repetition of every field
+ * @param conf
+ * @param file
+ * @return
+ */
+ public static ParquetFileInfo verifyFile(Configuration conf, String file) {
+ Path path = new Path(file);
+ ParquetMetadata footer = readMetadata(conf, path, ParquetMetadataConverter.SKIP_ROW_GROUPS);
+ FileMetaData fileMetaData = footer.getFileMetaData();
+ MessageType fileSchema = fileMetaData.getSchema();
+ List<Type> fields = fileSchema.getFields();
+ if (schemaNotSupportBinaryCopy(fields)) {
+ return new ParquetFileInfo(false, null, Collections.EMPTY_MAP);
+ }
+
+ Map<String, String> keyValueMetaData = fileMetaData.getKeyValueMetaData();
+ String bloomFileTypeCode = keyValueMetaData.get(HOODIE_BLOOM_FILTER_TYPE_CODE);
+ if (bloomFileTypeCode == null && keyValueMetaData.get(HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY) != null) {
+ bloomFileTypeCode = SIMPLE.name();
+ }
+ Map<String, String> repetitionMap = fields.stream()
+ .collect(Collectors.toMap(Type::getName, type -> type.getRepetition().name()));
+ return new ParquetFileInfo(true, bloomFileTypeCode, repetitionMap);
+ }
+
+ private static ParquetMetadata readMetadata(
+ Configuration conf,
+ Path parquetFilePath,
+ ParquetMetadataConverter.MetadataFilter filter) {
+ ParquetMetadata footer;
+ try {
+ footer = ParquetFileReader.readFooter(conf, parquetFilePath, filter);
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to read footer for parquet " + parquetFilePath, e);
+ }
+ return footer;
+ }
+
+ /**
+ * Check whether input schema not supports binary copy
+ * Following two case can not support
+ * 1. two level List structure, because the result of parquet rewrite is three level List structure
+ * 2. Decimal types stored via INT32/INT64/INT96, because it can not be read by parquet-avro
+ *
+ * @param parquetFields
+ * @return
+ */
+ private static boolean schemaNotSupportBinaryCopy(List<Type> parquetFields) {
+ for (Type type : parquetFields) {
+ if (type.getOriginalType() == OriginalType.DECIMAL) {
+ PrimitiveType primitiveType = type.asPrimitiveType();
+ PrimitiveType.PrimitiveTypeName typeName = primitiveType.getPrimitiveTypeName();
+ if (typeName == PrimitiveType.PrimitiveTypeName.INT32
+ || typeName == PrimitiveType.PrimitiveTypeName.INT64
+ || typeName == PrimitiveType.PrimitiveTypeName.INT96) {
+ return true;
+ }
+ }
+ if (!type.isPrimitive()) {
+ GroupType groupType = type.asGroupType();
+ OriginalType originalType = groupType.getOriginalType();
+ if (originalType == OriginalType.LIST
+ && groupType.getType(0).getName().equals("array")) {
+ return true;
+ }
+ if (schemaNotSupportBinaryCopy(groupType.getFields())) {
+ return true;
+ }
+ }
+ }
+ return false;
+ }
+
+ public static class ParquetFileInfo implements Serializable {
+ private final boolean isSchemaSupport;
+ private final String bloomFilterTypeCode;
+ private final Map<String, String> repetitions;
+
+ public ParquetFileInfo(boolean isSchemaSupport, String hoodieBloomFilterTypeCode, Map<String, String> repetitions) {
+ this.isSchemaSupport = isSchemaSupport;
+ this.bloomFilterTypeCode = hoodieBloomFilterTypeCode;
+ this.repetitions = repetitions;
+ }
+
+ public boolean isSchemaSupport() {
Review Comment:
can we add some doc, what does this mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2975344236
> @zhangyue19921010 @danny0405 pls check the match of table_schema and parquet schema before do binary copy. Spark BulkInsert will produce parquets with required attribute； however those columns in hudi itself are optional。 Do binary copy will corrupt parquet file [https://github.com/apache/spark/blob/1a3ae66c6c48bb319f0798826085e694fa7d0b58/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java#L389。](https://github.com/apache/spark/blob/1a3ae66c6c48bb319f0798826085e694fa7d0b58/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java#L389%E3%80%82) Suggest: throw exception directly when found above mismatch.
Hi @xiarixiaoyao added all necessary checks in `supportBinaryStreamCopy`including
1. Check if all input parquet files schema support binary copy
1) two level List structure check
2) Decimal types stored via INT32/INT64/INT96 check
2. Check if set of files contains only one type of BloomFilterTypeCode, including null
3. Check if the same column across these files has only one repetition type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2975622188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* adfc05d336dce06c43e3b458b0079894b92eaf33 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938823485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3f1ff569f1b1857ec1635febc2733f56581e4896 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124149945
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkStreamCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieCreateRewriteHandle;
+import org.apache.hudi.io.SingleFileHandleRewriteFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkStreamCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkStreamCopyClusteringExecutionStrategy.class);
+
+ public SparkStreamCopyClusteringExecutionStrategy(
Review Comment:
added `hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestClusteringBinaryCopyStrategy.scala`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2936060166
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963)
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2936327696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948833294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 26825f4bd4101deee97aacb1d106943e9515605f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031)
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938732462
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2948486594
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3327ebc046921fff138b889b8110a45b63b04ee2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981)
* 26825f4bd4101deee97aacb1d106943e9515605f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2126521473
##########
pom.xml:
##########
@@ -2604,6 +2609,7 @@
<orc.flink.version>1.5.6</orc.flink.version>
<flink.avro.version>1.11.4</flink.avro.version>
<flink.format.parquet.version>1.12.2</flink.format.parquet.version>
+ <parquet.version>${flink.format.parquet.version}</parquet.version>
Review Comment:
I'm concerned about the parquet version conflict like this, Spark and Flink does not really share the same parquet version before. Not sure how the flink profile would impact the `parquet.version` anyway, because the flink profile does not override it previously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2988087261
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3006841294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
}, {
"hash" : "237c232ad21117e76d82db253db93e86a23d302e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6466",
"triggerID" : "237c232ad21117e76d82db253db93e86a23d302e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449)
* 237c232ad21117e76d82db253db93e86a23d302e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005200547
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 16e3ed7ebb88a99ca55d879b367e0eaf630744c5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354)
* 7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448)
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3005578941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3006931686
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
}, {
"hash" : "237c232ad21117e76d82db253db93e86a23d302e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6466",
"triggerID" : "237c232ad21117e76d82db253db93e86a23d302e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* 237c232ad21117e76d82db253db93e86a23d302e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6466)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2136750924
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
Is this a enforcement for nexted data types?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
Is this an enforcement for nexted data types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2136750206
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/AvroConversionUtils.scala:
##########
@@ -268,4 +267,106 @@ object AvroConversionUtils {
}
schema
}
+
+ /**
+ * Recursively aligns the nullable property of hoodie table schema, supporting nested structures
+ */
+ def alignFields(sourceSchema: StructType, avroSchema: Schema): StructType = {
Review Comment:
alignFields -> alignFieldsNullability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2934707096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8e88905c1ee696a98750b86c9b09edd2c19eb98 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803)
* 6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2934974471
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2935678335
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960)
* 2ed9776f0277beb11e8e7087a6a0ece5c81b0c67 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124133565
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/HoodieDynamicBoundedBloomFilter.java:
##########
@@ -125,5 +126,42 @@ private void extractAndSetInternalBloomFilter(DataInputStream dis) throws IOExce
internalDynamicBloomFilter = new InternalDynamicBloomFilter();
internalDynamicBloomFilter.readFields(dis);
}
+
+ @Override
+ public void or(BloomFilter other) {
+ if (other != null) {
+ ValidationUtils.checkArgument(other instanceof HoodieDynamicBoundedBloomFilter,
+ "HoodieDynamicBoundedBloomFilter can only perform OR operations with other HoodieDynamicBoundedBloomFilter.");
+ HoodieDynamicBoundedBloomFilter otherFilter = (HoodieDynamicBoundedBloomFilter) other;
+ int sourceMatrixLength = this.getMatrixLength();
+ int targetMatrixLength = otherFilter.getMatrixLength();
+ if (targetMatrixLength > sourceMatrixLength) {
+ this.rescaleFromTarget(targetMatrixLength);
+ } else {
+ otherFilter.rescaleFromTarget(sourceMatrixLength);
+ }
+ this.internalDynamicBloomFilter.or(otherFilter.internalDynamicBloomFilter);
+ }
+ }
+
+ /**
+ * rescale the internal dynamic bloom filter by length
+ * @param targetMatrixLength
+ * @return
+ */
+ private HoodieDynamicBoundedBloomFilter rescaleFromTarget(int targetMatrixLength) {
+ int initMatrixLength = this.internalDynamicBloomFilter.getMatrixLength();
+ int needAddRowNum = targetMatrixLength - initMatrixLength;
+ if (needAddRowNum > 0) {
+ for (int i = 0; i < needAddRowNum; i++) {
+ this.internalDynamicBloomFilter.addRow();
Review Comment:
Nice catch here. add a new func addRows to improve performance.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/TypedProperties.java:
##########
@@ -39,7 +39,7 @@ public TypedProperties() {
super(null);
}
- protected TypedProperties(Properties defaults) {
+ public TypedProperties(Properties defaults) {
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124162047
##########
hudi-common/src/main/java/org/apache/hudi/common/bloom/BloomFilter.java:
##########
@@ -54,4 +54,10 @@ public interface BloomFilter {
* @return the bloom index type code.
**/
BloomFilterTypeCode getBloomFilterTypeCode();
+
+ /**
+ * Performs a logical OR operations with other BloomFilter.
+ * @param other
+ */
+ void or(BloomFilter other);
Review Comment:
In TestHoodieFileMetadataMerger#testBloomFilter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2124178775
##########
hudi-hadoop-common/src/test/java/org/apache/hudi/parquet/io/TestHoodieParquetFileRewriter.java:
##########
@@ -0,0 +1,459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.rewrite.HoodieFileMetadataMerger;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Version;
+import org.apache.parquet.column.ParquetProperties;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.format.DataPageHeader;
+import org.apache.parquet.format.DataPageHeaderV2;
+import org.apache.parquet.format.PageHeader;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.hadoop.util.CompressionConverter.TransParquetFileReader;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.internal.column.columnindex.ColumnIndex;
+import org.apache.parquet.internal.column.columnindex.OffsetIndex;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieRecord.FILENAME_METADATA_FIELD;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.BINARY;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.INT64;
+import static org.apache.parquet.schema.Type.Repetition.OPTIONAL;
+import static org.apache.parquet.schema.Type.Repetition.REPEATED;
+import static org.apache.parquet.schema.Type.Repetition.REQUIRED;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class TestHoodieParquetFileRewriter {
Review Comment:
Sure will add more UTs here, working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2125160501
##########
pom.xml:
##########
@@ -2604,6 +2609,7 @@
<orc.flink.version>1.5.6</orc.flink.version>
<flink.avro.version>1.11.4</flink.avro.version>
<flink.format.parquet.version>1.12.2</flink.format.parquet.version>
+ <parquet.version>${flink.format.parquet.version}</parquet.version>
Review Comment:
not sure why we need this, the `parquet.version` will finally be overridden in https://github.com/apache/hudi/blob/2db999a1a253fb222f1e4a397e46f92f2b1a846d/packaging/hudi-flink-bundle/pom.xml#L39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2938846929
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* 3679b9dd42afc403fd24d4ca392c9b6092ae8b61 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972)
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 3f1ff569f1b1857ec1635febc2733f56581e4896 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2126208177
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SingleSparkJobExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+ if (!supportBinaryStreamCopy()) {
+ String message = "1. Only support for CoW table 2. Not support for sort function";
+ throw new UnsupportedOperationException(message);
+
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ @Override
+ protected List<WriteStatus> performClusteringForGroup(ReaderContextFactory<T> readerContextFactory, ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ return new ArrayList<>(handler.close());
+ }
+
+ private boolean supportBinaryStreamCopy() {
+ return this.getHoodieTable().getMetaClient().getTableType() == COPY_ON_WRITE;
Review Comment:
Technically, Mor + insert is also applicable. However, during the constructor phase, there is no guarantee that the files to be clustered do not contain log files. Therefore, strong validation is implemented here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2126207603
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/parquet/io/HoodieParquetFileBinaryCopier.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.parquet.io;
+
+import org.apache.hudi.io.storage.HoodieFileMetadataMerger;
+import org.apache.hudi.io.storage.HoodieFileBinaryCopier;
+import org.apache.hudi.storage.StoragePath;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.HadoopReadOptions;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.FileMetaData;
+import org.apache.parquet.hadoop.util.CompressionConverter;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+import org.apache.parquet.schema.MessageType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Queue;
+import java.util.Set;
+
+/**
+ * HoodieParquetFileBinaryCopier is a high-performance utility designed for efficient merging of Parquet files at the binary level.
+ * Unlike conventional Parquet writers, it bypasses costly data processing operations through a block-based approach:
+ *
+ * Core Capabilities:
+ * 1. Zero-Processing Merge
+ * Directly concatenates raw Parquet data blocks (row groups) from input files
+ * Avoids:
+ * 1) Data serialization/deserialization
+ * 2) Compression/decompression cycles
+ * 3) Record-level reprocessing
+ *
+ * 2. Metadata Reconstruction
+ * Dynamically rebuilds file metadata:
+ * 1) New footer with merged statistics
+ * 2) Updated row group offsets
+ * 3) Validated schema consistency
+ */
+public class HoodieParquetFileBinaryCopier extends HoodieParquetBinaryCopyBase implements HoodieFileBinaryCopier {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HoodieParquetFileBinaryCopier.class);
+ private final CompressionCodecName codecName;
+
+ // Reader and relevant states of the in-processing input file
+ private Queue<CompressionConverter.TransParquetFileReader> inputFiles = new LinkedList<>();
+
+ private Map<String, String> extraMetaData = new HashMap<>();
+
+ // The reader for the current input file
+ private CompressionConverter.TransParquetFileReader reader = null;
+
+ private HoodieFileMetadataMerger metadataMerger;
+
+ public HoodieParquetFileBinaryCopier(
+ Configuration conf,
+ CompressionCodecName codecName,
+ HoodieFileMetadataMerger metadataMerger) {
+ super(conf);
+ this.metadataMerger = metadataMerger;
+ this.codecName = codecName;
+ }
+
+ @Override
+ protected Map<String, String> finalizeMetadata() {
+ return this.extraMetaData;
+ }
+
+ /**
+ * Merge all inputFilePaths to outputFilePath at block level
+ */
+ @Override
+ public long binaryCopy(List<StoragePath> inputFilePaths,
+ List<StoragePath> outputFilePath,
+ MessageType writeSchema,
+ Properties props) throws IOException {
+ openInputFiles(inputFilePaths, conf);
+ initFileWriter(new Path(outputFilePath.get(0).toUri()), codecName, writeSchema);
+ initNextReader();
+
+ Set<String> allOriginalCreatedBys = new HashSet<>();
+ while (reader != null) {
+ List<BlockMetaData> rowGroups = reader.getRowGroups();
+ FileMetaData fileMetaData = reader.getFooter().getFileMetaData();
+ String createdBy = fileMetaData.getCreatedBy();
+ allOriginalCreatedBys.add(createdBy);
+ Map<String, String> metaMap = fileMetaData.getKeyValueMetaData();
+ metadataMerger.mergeMetaData(metaMap);
+
+ for (BlockMetaData block : rowGroups) {
+ processBlocksFromReader(reader, reader.readNextRowGroup(), block, createdBy);
+ }
+ initNextReader();
+ }
+ extraMetaData.putAll(metadataMerger.getMergedMetaData());
+ extraMetaData.put(ORIGINAL_CREATED_BY_KEY, String.join("\n", allOriginalCreatedBys));
+ return totalRecordsWritten;
+ }
+
+ // Open all input files to validate their schemas are compatible to merge
+ private void openInputFiles(List<StoragePath> inputFiles, Configuration conf) {
+ Preconditions.checkArgument(inputFiles != null && !inputFiles.isEmpty(), "No input files");
+
+ for (StoragePath inputFile : inputFiles) {
+ try {
+ CompressionConverter.TransParquetFileReader reader = new CompressionConverter.TransParquetFileReader(
+ HadoopInputFile.fromPath(new Path(inputFile.toUri()), conf),
+ HadoopReadOptions.builder(conf).build());
+ this.inputFiles.add(reader);
+ } catch (IOException e) {
+ throw new IllegalArgumentException("Failed to open input file: " + inputFile, e);
+ }
+ }
+ }
+
+ // Routines to get reader of next input file and set up relevant states
+ private void initNextReader() {
+ if (reader != null) {
Review Comment:
Close former reader before init next reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2949000033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* 90115910fb597ab25f3d4111d4f184727c5e1b10 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033)
* 7d5efa6090bbca6e5d479106f05060cac4be9089 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "gudladona (via GitHub)" <gi...@apache.org>.
gudladona commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2134867004
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SingleSparkJobExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+ if (!supportBinaryStreamCopy()) {
+ String message = "1. Only support for CoW table 2. Not support for sort function";
+ throw new UnsupportedOperationException(message);
+
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ @Override
+ protected List<WriteStatus> performClusteringForGroup(ReaderContextFactory<T> readerContextFactory, ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ return new ArrayList<>(handler.close());
+ }
+
+ private boolean supportBinaryStreamCopy() {
+ return this.getHoodieTable().getMetaClient().getTableType() == COPY_ON_WRITE;
Review Comment:
Could you advise on what is needed to bring this capability to MOR with INSERT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13365:
URL: https://github.com/apache/hudi/pull/13365#discussion_r2163741826
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
Review Comment:
nts: where should this live.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkConsistentBucketClusteringExecutionStrategy.java:
##########
@@ -81,7 +80,7 @@ public HoodieData<WriteStatus> performClusteringWithRecordsAsRow(Dataset<Row> in
Dataset<Row> repartitionedRecords = partitioner.repartitionRecords(inputRecords, numOutputGroups);
return HoodieDatasetBulkInsertHelper.bulkInsert(repartitionedRecords, instantTime, getHoodieTable(), newConfig,
- partitioner.arePartitionRecordsSorted(), shouldPreserveHoodieMetadata, WriteOperationType.CLUSTER);
Review Comment:
any behavior changes from this? there may be older code that relies on having this operation type
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:
##########
@@ -150,14 +149,8 @@ object HoodieDatasetBulkInsertHelper
table: HoodieTable[_, _, _, _],
writeConfig: HoodieWriteConfig,
arePartitionRecordsSorted: Boolean,
- shouldPreserveHoodieMetadata: Boolean,
- operation: WriteOperationType): HoodieData[WriteStatus] = {
- val schema = operation match {
- case WriteOperationType.CLUSTER =>
- alignNotNullFields(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
- case _ =>
- dataset.schema
- }
+ shouldPreserveHoodieMetadata: Boolean): HoodieData[WriteStatus] = {
+ val schema = AvroConversionUtils.alignFieldsNullability(dataset.schema, new Schema.Parser().parse(writeConfig.getSchema))
Review Comment:
this causes a behavior change for bulk insert as well ? (not just clustering). Please help me understand why this is ok
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.parquet.io.ParquetBinaryCopyChecker;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+import static org.apache.hudi.config.HoodieClusteringConfig.PLAN_STRATEGY_SORT_COLUMNS;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+
+ List<ClusteringGroupInfo> clusteringGroupInfos = clusteringPlan.getInputGroups()
+ .stream()
+ .map(ClusteringGroupInfo::create)
+ .collect(Collectors.toList());
+ if (!supportBinaryStreamCopy(clusteringGroupInfos, clusteringPlan.getStrategy().getStrategyParams())) {
+ LOG.info("Required conditions for binary stream copy are currently not satisfied, falling back to default clustering behavior");
+ HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder().withProperties(writeConfig.getProps())
+ .withStorageConfig(HoodieStorageConfig.newBuilder().parquetWriteLegacyFormat("false").build()).build();
+ resetWriteConfig(newConfig);
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+ TaskContextSupplier taskContextSupplier = getEngineContext().getTaskContextSupplier();
+ SerializableSchema serializableSchema = new SerializableSchema(schema);
+ boolean shouldPreserveMetadata = Option.ofNullable(clusteringPlan.getPreserveHoodieMetadata()).orElse(false);
+ JavaRDD<ClusteringGroupInfo> groupInfoJavaRDD = engineContext.parallelize(clusteringGroupInfos, clusteringGroupInfos.size());
+ LOG.info("number of partitions for clustering " + groupInfoJavaRDD.getNumPartitions());
+ JavaRDD<WriteStatus> writeStatusRDD = groupInfoJavaRDD
+ .mapPartitions(clusteringOps -> {
+ Iterable<ClusteringGroupInfo> clusteringOpsIterable = () -> clusteringOps;
+ return StreamSupport.stream(clusteringOpsIterable.spliterator(), false)
+ .flatMap(clusteringOp ->
+ runClusteringForGroup(
+ clusteringOp,
+ clusteringPlan.getStrategy().getStrategyParams(),
+ shouldPreserveMetadata,
+ serializableSchema,
+ taskContextSupplier,
+ instantTime))
+ .iterator();
+ });
+
+ HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = new HoodieWriteMetadata<>();
+ writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
+ return writeMetadata;
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ private Stream<WriteStatus> runClusteringForGroup(ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<WriteStatus> statuses = new ArrayList<>();
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ statuses.addAll(handler.close());
+ return statuses.stream();
+ }
+
+ /**
+ * 1. Check Table type
+ * 2. Check bloom filter type code
+ * 3. Check Array Type Schema consistency affected by hoodie.parquet.writelegacyformat.enabled and spark.hadoop.parquet.avro.write-old-list-structure
+ * 4. Check Schema Optional or Required consistency for the same field
+ */
+ public boolean supportBinaryStreamCopy(List<ClusteringGroupInfo> inputGroups, Map<String, String> strategyParams) {
+ if (getHoodieTable().getMetaClient().getTableType() != COPY_ON_WRITE) {
+ LOG.warn("Only support CoW table. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+ Option<String[]> orderByColumnsOpt =
+ Option.ofNullable(strategyParams.get(PLAN_STRATEGY_SORT_COLUMNS.key()))
+ .map(listStr -> listStr.split(","));
+
+ if (orderByColumnsOpt.isPresent()) {
+ LOG.warn("Not support to sort input records. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+
+ List<ParquetBinaryCopyChecker.ParquetFileInfo> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
+ .flatMap(group -> group.getOperations().iterator())
+ .map(op -> {
+ String filePath = op.getDataFilePath();
+ if (!filePath.endsWith(PARQUET.getFileExtension())) {
+ return new ParquetBinaryCopyChecker.ParquetFileInfo(false, null, null);
Review Comment:
should n't we fall back and return `false` if not parquet file.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/strategy/ClusteringExecutionStrategy.java:
##########
@@ -96,6 +96,10 @@ protected HoodieWriteConfig getWriteConfig() {
return this.writeConfig;
}
+ protected void resetWriteConfig(HoodieWriteConfig config) {
Review Comment:
could we avoid mutating the config like this
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/SparkBinaryCopyClusteringExecutionStrategy.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.HoodieStorageConfig;
+import org.apache.hudi.common.config.SerializableSchema;
+import org.apache.hudi.common.data.HoodieData;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.engine.TaskContextSupplier;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.ClusteringGroupInfo;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.io.HoodieBinaryCopyHandle;
+import org.apache.hudi.io.BinaryCopyHandleFactory;
+import org.apache.hudi.parquet.io.ParquetBinaryCopyChecker;
+import org.apache.hudi.storage.StoragePath;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+import static org.apache.hudi.common.model.HoodieFileFormat.PARQUET;
+import static org.apache.hudi.common.model.HoodieTableType.COPY_ON_WRITE;
+import static org.apache.hudi.config.HoodieClusteringConfig.PLAN_STRATEGY_SORT_COLUMNS;
+
+/**
+ * Clustering strategy to submit single spark jobs using streaming copy
+ * PAY ATTENTION!!!
+ * IN THIS STRATEGY
+ * 1. Only support clustering for cow table.
+ * 2. Sort function is not supported yet.
+ * 3. Each clustering group only has one task to write.
+ */
+public class SparkBinaryCopyClusteringExecutionStrategy<T> extends SparkSortAndSizeExecutionStrategy<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(SparkBinaryCopyClusteringExecutionStrategy.class);
+
+ public SparkBinaryCopyClusteringExecutionStrategy(
+ HoodieTable table,
+ HoodieEngineContext engineContext,
+ HoodieWriteConfig writeConfig) {
+ super(table, engineContext, writeConfig);
+ }
+
+ @Override
+ public HoodieWriteMetadata<HoodieData<WriteStatus>> performClustering(
+ HoodieClusteringPlan clusteringPlan,
+ Schema schema,
+ String instantTime) {
+
+ List<ClusteringGroupInfo> clusteringGroupInfos = clusteringPlan.getInputGroups()
+ .stream()
+ .map(ClusteringGroupInfo::create)
+ .collect(Collectors.toList());
+ if (!supportBinaryStreamCopy(clusteringGroupInfos, clusteringPlan.getStrategy().getStrategyParams())) {
+ LOG.info("Required conditions for binary stream copy are currently not satisfied, falling back to default clustering behavior");
+ HoodieWriteConfig newConfig = HoodieWriteConfig.newBuilder().withProperties(writeConfig.getProps())
+ .withStorageConfig(HoodieStorageConfig.newBuilder().parquetWriteLegacyFormat("false").build()).build();
+ resetWriteConfig(newConfig);
+ return super.performClustering(clusteringPlan, schema, instantTime);
+ }
+ LOG.info("Required conditions are currently satisfied, enabling the optimization of using binary stream copy ");
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+ TaskContextSupplier taskContextSupplier = getEngineContext().getTaskContextSupplier();
+ SerializableSchema serializableSchema = new SerializableSchema(schema);
+ boolean shouldPreserveMetadata = Option.ofNullable(clusteringPlan.getPreserveHoodieMetadata()).orElse(false);
+ JavaRDD<ClusteringGroupInfo> groupInfoJavaRDD = engineContext.parallelize(clusteringGroupInfos, clusteringGroupInfos.size());
+ LOG.info("number of partitions for clustering " + groupInfoJavaRDD.getNumPartitions());
+ JavaRDD<WriteStatus> writeStatusRDD = groupInfoJavaRDD
+ .mapPartitions(clusteringOps -> {
+ Iterable<ClusteringGroupInfo> clusteringOpsIterable = () -> clusteringOps;
+ return StreamSupport.stream(clusteringOpsIterable.spliterator(), false)
+ .flatMap(clusteringOp ->
+ runClusteringForGroup(
+ clusteringOp,
+ clusteringPlan.getStrategy().getStrategyParams(),
+ shouldPreserveMetadata,
+ serializableSchema,
+ taskContextSupplier,
+ instantTime))
+ .iterator();
+ });
+
+ HoodieWriteMetadata<HoodieData<WriteStatus>> writeMetadata = new HoodieWriteMetadata<>();
+ writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
+ return writeMetadata;
+ }
+
+ /**
+ * Submit job to execute clustering for the group.
+ */
+ private Stream<WriteStatus> runClusteringForGroup(ClusteringGroupInfo clusteringOps, Map<String, String> strategyParams,
+ boolean preserveHoodieMetadata, SerializableSchema schema,
+ TaskContextSupplier taskContextSupplier, String instantTime) {
+ List<WriteStatus> statuses = new ArrayList<>();
+ List<HoodieFileGroupId> inputFileIds = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new HoodieFileGroupId(op.getPartitionPath(), op.getFileId()))
+ .collect(Collectors.toList());
+ List<StoragePath> inputFilePaths = clusteringOps.getOperations()
+ .stream()
+ .map(op -> new StoragePath(op.getDataFilePath()))
+ .collect(Collectors.toList());
+
+ BinaryCopyHandleFactory factory = new BinaryCopyHandleFactory(inputFilePaths);
+ HoodieBinaryCopyHandle handler = factory.create(
+ getWriteConfig(),
+ instantTime,
+ getHoodieTable(),
+ inputFileIds.get(0).getPartitionPath(),
+ FSUtils.createNewFileIdPfx(),
+ taskContextSupplier);
+
+ handler.write();
+ statuses.addAll(handler.close());
+ return statuses.stream();
+ }
+
+ /**
+ * 1. Check Table type
+ * 2. Check bloom filter type code
+ * 3. Check Array Type Schema consistency affected by hoodie.parquet.writelegacyformat.enabled and spark.hadoop.parquet.avro.write-old-list-structure
+ * 4. Check Schema Optional or Required consistency for the same field
+ */
+ public boolean supportBinaryStreamCopy(List<ClusteringGroupInfo> inputGroups, Map<String, String> strategyParams) {
+ if (getHoodieTable().getMetaClient().getTableType() != COPY_ON_WRITE) {
+ LOG.warn("Only support CoW table. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+ Option<String[]> orderByColumnsOpt =
+ Option.ofNullable(strategyParams.get(PLAN_STRATEGY_SORT_COLUMNS.key()))
+ .map(listStr -> listStr.split(","));
+
+ if (orderByColumnsOpt.isPresent()) {
+ LOG.warn("Not support to sort input records. Will fall back to common clustering execution strategy.");
+ return false;
+ }
+
+ JavaSparkContext engineContext = HoodieSparkEngineContext.getSparkContext(getEngineContext());
+
+ List<ParquetBinaryCopyChecker.ParquetFileInfo> fileStatus = engineContext.parallelize(inputGroups, inputGroups.size())
+ .flatMap(group -> group.getOperations().iterator())
+ .map(op -> {
+ String filePath = op.getDataFilePath();
Review Comment:
Should we use the table config i.e base file format .. instead of file path name
##########
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieFileMetadataMerger.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io.storage;
+
+import org.apache.hudi.avro.HoodieBloomFilterWriteSupport;
+import org.apache.hudi.common.bloom.BloomFilter;
+import org.apache.hudi.common.bloom.BloomFilterFactory;
+import org.apache.hudi.common.util.StringUtils;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hudi.avro.HoodieBloomFilterWriteSupport.HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY;
+
+public class HoodieFileMetadataMerger {
+
+ private final HashMap<String, String> mergedMateData = new HashMap<>();
+
+ private BloomFilter bloomFilter;
+
+ private String minRecordKey;
+
+ private String maxRecordKey;
+
+ public HoodieFileMetadataMerger() {
+
+ }
+
+ public Map<String, String> mergeMetaData(Map<String, String> metaMap) {
+ mergedMateData.putAll(metaMap);
+
+ String minRecordKey = metaMap.get(HoodieBloomFilterWriteSupport.HOODIE_MIN_RECORD_KEY_FOOTER);
+ String maxRecordKey = metaMap.get(HoodieBloomFilterWriteSupport.HOODIE_MAX_RECORD_KEY_FOOTER);
+ if (this.minRecordKey == null || (minRecordKey != null && this.minRecordKey.compareTo(minRecordKey) > 0)) {
+ this.minRecordKey = minRecordKey;
+ }
+ if (this.maxRecordKey == null || (maxRecordKey != null && this.maxRecordKey.compareTo(maxRecordKey) < 0)) {
+ this.maxRecordKey = maxRecordKey;
+ }
+
+ String bloomFilterType = metaMap.get(HoodieBloomFilterWriteSupport.HOODIE_BLOOM_FILTER_TYPE_CODE);
+ String avroBloom = metaMap.get(HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY);
+ if (!StringUtils.isNullOrEmpty(avroBloom)) {
+ BloomFilter targetBloomFilter = BloomFilterFactory.fromString(avroBloom, bloomFilterType);
+ if (this.bloomFilter == null) {
+ this.bloomFilter = targetBloomFilter;
+ } else {
+ this.bloomFilter.or(targetBloomFilter);
+ }
+ }
+
+ if (this.minRecordKey != null) {
+ mergedMateData.put(HoodieBloomFilterWriteSupport.HOODIE_MIN_RECORD_KEY_FOOTER, this.minRecordKey);
+ }
+ if (this.maxRecordKey != null) {
+ mergedMateData.put(HoodieBloomFilterWriteSupport.HOODIE_MAX_RECORD_KEY_FOOTER, this.maxRecordKey);
+ }
+ if (this.bloomFilter != null) {
+ mergedMateData.put(HOODIE_AVRO_BLOOM_FILTER_METADATA_KEY, this.bloomFilter.serializeToString());
+ }
+
+ return mergedMateData;
Review Comment:
typo: mergedMetadata
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/functional/TestHoodieClientOnCopyOnWriteStorage.java:
##########
@@ -1017,6 +1017,14 @@ public void testSimpleClustering(boolean populateMetaFields) throws Exception {
false, SqlQueryEqualityPreCommitValidator.class.getName(), COUNT_SQL_QUERY_FOR_VALIDATION, "");
}
+ @Test
+ public void testBinaryCopyClustering() throws Exception {
+ String strategy = "org.apache.hudi.client.clustering.run.strategy.SparkBinaryCopyClusteringExecutionStrategy";
+ HoodieClusteringConfig config = createClusteringBuilder(true, 1).withClusteringExecutionStrategyClass(strategy).build();
+ testInsertAndClustering(config, true, true,
+ false, SqlQueryEqualityPreCommitValidator.class.getName(), COUNT_SQL_QUERY_FOR_VALIDATION, "");
Review Comment:
can we also validate the contents of the files..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-3006772484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
}, {
"hash" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5803",
"triggerID" : "b8e88905c1ee696a98750b86c9b09edd2c19eb98",
"triggerType" : "PUSH"
}, {
"hash" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5960",
"triggerID" : "6b3e7e8dcd3b4ff74c4d907ec923800d2ea23231",
"triggerType" : "PUSH"
}, {
"hash" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5963",
"triggerID" : "2ed9776f0277beb11e8e7087a6a0ece5c81b0c67",
"triggerType" : "PUSH"
}, {
"hash" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7fda08485d1e4d0f15014574d6c922fccba959b9",
"triggerType" : "PUSH"
}, {
"hash" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002d23e5a51e5afacbc4adb5d12cc337a79ef71b",
"triggerType" : "PUSH"
}, {
"hash" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5965",
"triggerID" : "8183f6d7cf06d3d80f0dc97c00159b5a6e233bc8",
"triggerType" : "PUSH"
}, {
"hash" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5972",
"triggerID" : "3679b9dd42afc403fd24d4ca392c9b6092ae8b61",
"triggerType" : "PUSH"
}, {
"hash" : "f71312bf500e234f7361c8598d876556e91c323f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f71312bf500e234f7361c8598d876556e91c323f",
"triggerType" : "PUSH"
}, {
"hash" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5974",
"triggerID" : "3f1ff569f1b1857ec1635febc2733f56581e4896",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5976",
"triggerID" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"triggerType" : "PUSH"
}, {
"hash" : "3327ebc046921fff138b889b8110a45b63b04ee2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5981",
"triggerID" : "2939576681",
"triggerType" : "MANUAL"
}, {
"hash" : "26825f4bd4101deee97aacb1d106943e9515605f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6031",
"triggerID" : "26825f4bd4101deee97aacb1d106943e9515605f",
"triggerType" : "PUSH"
}, {
"hash" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "02ad712a93647d4269c66dadc1ad2ab9ce928f0a",
"triggerType" : "PUSH"
}, {
"hash" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6033",
"triggerID" : "90115910fb597ab25f3d4111d4f184727c5e1b10",
"triggerType" : "PUSH"
}, {
"hash" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6035",
"triggerID" : "7d5efa6090bbca6e5d479106f05060cac4be9089",
"triggerType" : "PUSH"
}, {
"hash" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6100",
"triggerID" : "79d067dc6055365a678997b8a6d3a0abccc41122",
"triggerType" : "PUSH"
}, {
"hash" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6103",
"triggerID" : "40d3cbc23a2ab560fc760fd92145630cf8cad0e2",
"triggerType" : "PUSH"
}, {
"hash" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6190",
"triggerID" : "0d96b1c7f39ecf1eb548e33cc3fd04e792a688a0",
"triggerType" : "PUSH"
}, {
"hash" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d9051043412d2b96a7fe998cbe43fb640c37ed71",
"triggerType" : "PUSH"
}, {
"hash" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6228",
"triggerID" : "adfc05d336dce06c43e3b458b0079894b92eaf33",
"triggerType" : "PUSH"
}, {
"hash" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57c1bf35efb0257a8792c33fb72833a9dd9894a3",
"triggerType" : "PUSH"
}, {
"hash" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6352",
"triggerID" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"triggerType" : "PUSH"
}, {
"hash" : "16e3ed7ebb88a99ca55d879b367e0eaf630744c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6354",
"triggerID" : "2987870492",
"triggerType" : "MANUAL"
}, {
"hash" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6448",
"triggerID" : "7cfd82b0779b3664acc9ccefdb8c3fcb784d1b90",
"triggerType" : "PUSH"
}, {
"hash" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449",
"triggerID" : "c35fd3e42538512ed0dac606eacf19ae8ec9d491",
"triggerType" : "PUSH"
}, {
"hash" : "237c232ad21117e76d82db253db93e86a23d302e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "237c232ad21117e76d82db253db93e86a23d302e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7fda08485d1e4d0f15014574d6c922fccba959b9 UNKNOWN
* 002d23e5a51e5afacbc4adb5d12cc337a79ef71b UNKNOWN
* f71312bf500e234f7361c8598d876556e91c323f UNKNOWN
* 02ad712a93647d4269c66dadc1ad2ab9ce928f0a UNKNOWN
* d9051043412d2b96a7fe998cbe43fb640c37ed71 UNKNOWN
* 57c1bf35efb0257a8792c33fb72833a9dd9894a3 UNKNOWN
* 338fa5c46c4e3d35d1a045c9ab496fb9abf1bd94 UNKNOWN
* c35fd3e42538512ed0dac606eacf19ae8ec9d491 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6449)
* 237c232ad21117e76d82db253db93e86a23d302e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914945470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46aa20c12464c31375fadbe8c396adb5f4b008dd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793)
* bf2a61d993d8784144df7d28f80f0610aaad8bb8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2914942158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46aa20c12464c31375fadbe8c396adb5f4b008dd Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793)
* bf2a61d993d8784144df7d28f80f0610aaad8bb8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9468] Parquet Binary Copy at Rowgroup Level [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13365:
URL: https://github.com/apache/hudi/pull/13365#issuecomment-2915130514
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5793",
"triggerID" : "46aa20c12464c31375fadbe8c396adb5f4b008dd",
"triggerType" : "PUSH"
}, {
"hash" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800",
"triggerID" : "bf2a61d993d8784144df7d28f80f0610aaad8bb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bf2a61d993d8784144df7d28f80f0610aaad8bb8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org