You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "lucasfang (via GitHub)" <gi...@apache.org> on 2026/06/23 02:31:18 UTC
[PR] feat(mergetree): add compact manager, task abstractions and file rewr… [paimon-cpp]
lucasfang opened a new pull request, #105:
URL: https://github.com/apache/paimon-cpp/pull/105
### Purpose
Linked issue: No linked issue
This change adds merge tree compact manager, compact task abstractions, and file rewrite compact task support for the mergetree compaction subsystem.
Included changes:
- **MergeTreeCompactManager**:
- Adds `merge_tree_compact_manager.h` and `merge_tree_compact_manager.cpp` to orchestrate merge tree compaction workflows.
- Adds test coverage in `merge_tree_compact_manager_test.cpp`.
- **MergeTreeCompactManagerFactory**:
- Adds `merge_tree_compact_manager_factory.h` and `merge_tree_compact_manager_factory.cpp` to provide factory-based construction of compact managers.
- Adds test coverage in `merge_tree_compact_manager_factory_test.cpp`.
- **MergeTreeCompactTask**:
- Adds `merge_tree_compact_task.h` and `merge_tree_compact_task.cpp` to define the base compact task abstraction for merge tree operations.
- **FileRewriteCompactTask**:
- Adds `file_rewrite_compact_task.h` to define file-rewrite specific compact task logic.
- Adds test coverage in `file_rewrite_compact_task_test.cpp`.
### Tests
Not run. Local compile, CMake, and gtest environment checks are not part of this PR description.
Test coverage included in this change:
- `MergeTreeCompactManagerTest`
- `MergeTreeCompactManagerFactoryTest`
- `FileRewriteCompactTaskTest`
### API and Format
No public API, storage format, or protocol changes.
### Documentation
No documentation changes required.
### Generative AI tooling
Migrate-by: Aone Copilot (Qwen-3.7-Max)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(mergetree): add compact manager, task abstractions and file rewr… [paimon-cpp]
Posted by "lszskye (via GitHub)" <gi...@apache.org>.
lszskye commented on PR #105:
URL: https://github.com/apache/paimon-cpp/pull/105#issuecomment-4777063291
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(mergetree): add compact manager, task abstractions and file rewr… [paimon-cpp]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #105:
URL: https://github.com/apache/paimon-cpp/pull/105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(mergetree): add compact manager, task abstractions and file rewr… [paimon-cpp]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #105:
URL: https://github.com/apache/paimon-cpp/pull/105#discussion_r3456665965
##########
src/paimon/core/mergetree/compact/merge_tree_compact_manager.cpp:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/core/mergetree/compact/merge_tree_compact_manager.h"
+
+#include <cassert>
+#include <iterator>
+
+#include "fmt/format.h"
+#include "paimon/common/executor/future.h"
+#include "paimon/core/compact/compact_deletion_file.h"
+#include "paimon/core/compact/compact_task.h"
+#include "paimon/core/mergetree/compact/file_rewrite_compact_task.h"
+#include "paimon/core/mergetree/compact/interval_partition.h"
+#include "paimon/core/mergetree/compact/merge_tree_compact_task.h"
Review Comment:
`MergeTreeCompactManager` uses `StringUtils::VectorToString(...)` but this translation unit does not include `paimon/common/utils/string_utils.h`, making compilation dependent on transitive includes. Also `interval_partition.h` is included but not used in this file.
##########
src/paimon/core/mergetree/compact/merge_tree_compact_manager_factory.cpp:
##########
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/core/mergetree/compact/merge_tree_compact_manager_factory.h"
+
+#include <vector>
+
+#include "paimon/common/data/serializer/row_compacted_serializer.h"
Review Comment:
This file calls `DeletionVector::CreateFactory(...)` but doesn't include the header that declares `DeletionVector`. Relying on indirect inclusion through other headers makes the build fragile to include-order changes.
##########
src/paimon/core/mergetree/compact/merge_tree_compact_manager.cpp:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/core/mergetree/compact/merge_tree_compact_manager.h"
+
+#include <cassert>
+#include <iterator>
+
+#include "fmt/format.h"
+#include "paimon/common/executor/future.h"
+#include "paimon/core/compact/compact_deletion_file.h"
+#include "paimon/core/compact/compact_task.h"
+#include "paimon/core/mergetree/compact/file_rewrite_compact_task.h"
+#include "paimon/core/mergetree/compact/interval_partition.h"
+#include "paimon/core/mergetree/compact/merge_tree_compact_task.h"
+
+namespace paimon {
+
+std::string FilesToString(const std::vector<std::shared_ptr<DataFileMeta>>& files) {
Review Comment:
This helper is only used within this `.cpp` file; giving it internal linkage avoids exporting an unnecessary symbol from the object file.
##########
src/paimon/core/mergetree/compact/merge_tree_compact_task.h:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <vector>
+
+#include "paimon/common/utils/fields_comparator.h"
+#include "paimon/core/compact/compact_deletion_file.h"
+#include "paimon/core/compact/compact_task.h"
+#include "paimon/core/compact/compact_unit.h"
+#include "paimon/core/mergetree/compact/compact_rewriter.h"
+#include "paimon/core/mergetree/compact/interval_partition.h"
Review Comment:
`interval_partition.h` isn't referenced by this header (only used in the `.cpp`). Keeping it here increases transitive compile cost for every includer. Consider removing it from the header and including it in `merge_tree_compact_task.cpp` instead.
##########
src/paimon/core/mergetree/compact/merge_tree_compact_manager.cpp:
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/core/mergetree/compact/merge_tree_compact_manager.h"
+
+#include <cassert>
+#include <iterator>
+
+#include "fmt/format.h"
+#include "paimon/common/executor/future.h"
+#include "paimon/core/compact/compact_deletion_file.h"
+#include "paimon/core/compact/compact_task.h"
+#include "paimon/core/mergetree/compact/file_rewrite_compact_task.h"
+#include "paimon/core/mergetree/compact/interval_partition.h"
+#include "paimon/core/mergetree/compact/merge_tree_compact_task.h"
+
+namespace paimon {
+
+std::string FilesToString(const std::vector<std::shared_ptr<DataFileMeta>>& files) {
+ fmt::memory_buffer buffer;
+ bool first = true;
+ for (const auto& file : files) {
+ if (!first) {
+ fmt::format_to(std::back_inserter(buffer), ", ");
+ }
+ fmt::format_to(std::back_inserter(buffer), "({}, {}, {})", file->file_name, file->level,
+ file->file_size);
+ first = false;
+ }
+ return fmt::to_string(buffer);
+}
+
+std::string FilesToString(const std::vector<LevelSortedRun>& runs) {
Review Comment:
This helper is only used within this `.cpp` file; giving it internal linkage avoids exporting an unnecessary symbol from the object file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org