You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/02/09 08:49:19 UTC
[GitHub] [flink] pltbkd opened a new pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
pltbkd opened a new pull request #18680:
URL: https://github.com/apache/flink/pull/18680
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
This pull request makes the FileSink to support compacting small files, based on the new extended sink API.
## Brief change log
- Users may enable compaction when building FileSink by calling enableCompact.
- Committables emitted from FileWriter will be collected by CompactCoordinator and reorgnized into CompactRequests according to the user defined FileCompactStrategy.
- CompactRequests will be sent to CompactorOperators then executed, and the results will be sent to the committers, as if it's created by the writers.
- FileCompactor is introduced to execute the compaction, and several basic implementations are provided.
## Verifying this change
This change added some unit tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
- The serializers: (yes)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804334220
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/ConcatFileCompactor.java
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+
+/**
+ * A {@link OutputStreamBasedFileCompactor} implementation that simply concat the compacting files.
+ * The fileDelimiter will be added between neighbouring files if provided.
+ */
+public class ConcatFileCompactor extends OutputStreamBasedFileCompactor {
+
+ private static final int CHUNK_SIZE = 4 * 1024 * 1024;
+
+ private final byte[] fileDelimiter;
+
+ public ConcatFileCompactor() {
Review comment:
ConcatFileCompactor is created by the users and I'd like to provide a non-parameter constructor for users who want to concat files without a delimiter. Passing a null by the user is more or less confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r803476471
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
Review comment:
Why we need to add it to the head of the list? It should be quite inefficient to insert elements to an array.
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java
##########
@@ -170,15 +177,38 @@ public WriterProperties getProperties() {
implements PendingFileRecoverable {
private final RecoverableWriter.CommitRecoverable commitRecoverable;
+ private final Path targetPath;
Review comment:
@Nullable
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedPartFileWriter.java
##########
@@ -189,15 +219,36 @@ public OutputStreamBasedPendingFileRecoverable(
implements InProgressFileRecoverable {
private final RecoverableWriter.ResumeRecoverable resumeRecoverable;
+ private final Path targetPath;
Review comment:
@Nullable
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/DecoderBasedReader.java
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.Path;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.Serializable;
+
+/**
+ * A {@link RecordWiseFileCompactor.Reader} implementation that reads the file as an {@link
+ * FSDataInputStream} and decodes the record with the {@link Decoder}.
+ */
+public class DecoderBasedReader<T> implements RecordWiseFileCompactor.Reader<T> {
+ private final Decoder<T> decoder;
+ private final FSDataInputStream input;
+
+ public DecoderBasedReader(Path path, Decoder<T> decoder) throws IOException {
+ this.decoder = decoder;
+ this.input = path.getFileSystem().open(path);
+ }
+
+ @Override
+ public T read() throws IOException {
+ if (input.available() > 0) {
+ return decoder.decodeNext(input);
+ }
+ return null;
+ }
+
+ @Override
+ public void close() throws Exception {
+ input.close();
+ }
+
+ /**
+ * A {@link Decoder} to decode the file content into the actual records.
+ *
+ * <p>A {@link Decoder} is generally the reverse of a {@link
+ * org.apache.flink.api.common.serialization.Encoder}.
+ *
+ * @param <T> Thy type of the records the reader is reading.
+ */
+ public interface Decoder<T> extends Serializable {
+ /**
+ * @return The next record that decoded from the InputStream, or null if no more available.
+ */
+ T decodeNext(InputStream input) throws IOException;
Review comment:
Might split to `open` and `decodeNext` to allow the users to cache a reader easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1041157963
> Thanks @pltbkd for the update! I try to fixed some tests. Also, could you have a look that if the two `*StateHandle` needs `snapshotState` ?
Cleaning up of the state is added when the remaining state is drained in the handlers. Nothing special is necessary while snapshotting the state. I suppose we don't need to implement a customized `snapshotState`. I have added some comments to explain this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006)
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804489417
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
Review comment:
It seems that we can do nothing at present. I'll leave a comment here and maybe we can modify this when the relating issues in the Sink api is solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804316721
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/FileCompactStrategy.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.connector.file.sink.FileSink;
+
+import java.io.Serializable;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+
+/** Strategy for compacting the files written in {@link FileSink} before committing. */
+public class FileCompactStrategy implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ // Compaction triggering strategies.
+ private final long sizeThreshold;
+ private final boolean compactOnCheckpoint;
+
+ // Compaction executing strategies.
+ private final int compactThread;
+
+ private FileCompactStrategy(
+ long sizeThreshold, boolean compactOnCheckpoint, int compactThread) {
+ this.sizeThreshold = sizeThreshold;
+ this.compactOnCheckpoint = compactOnCheckpoint;
+ this.compactThread = compactThread;
+ }
+
+ public long getSizeThreshold() {
+ return sizeThreshold;
+ }
+
+ public boolean isCompactOnCheckpoint() {
Review comment:
If possible we might support compact for each `n` checkpoints.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/ConcatFileCompactor.java
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+
+/**
+ * A {@link OutputStreamBasedFileCompactor} implementation that simply concat the compacting files.
+ * The fileDelimiter will be added between neighbouring files if provided.
+ */
+public class ConcatFileCompactor extends OutputStreamBasedFileCompactor {
+
+ private static final int CHUNK_SIZE = 4 * 1024 * 1024;
+
+ private final byte[] fileDelimiter;
+
+ public ConcatFileCompactor() {
Review comment:
This method is not used?
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/OutputStreamBasedFileCompactor.java
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.OutputStreamBasedCompactingFileWriter;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.List;
+
+/**
+ * Base class for {@link FileCompactor} implementations that use the {@link
+ * OutputStreamBasedCompactingFileWriter}.
+ */
+public abstract class OutputStreamBasedFileCompactor implements FileCompactor {
+ @Override
+ public final CompactingFileWriter.Type getWriterType() {
+ return CompactingFileWriter.Type.OUTPUT_STREAM;
+ }
+
+ @Override
+ public void compact(List<Path> inputFiles, CompactingFileWriter writer) throws Exception {
+ // The outputStream returned by OutputStreamBasedCompactingFileWriter#asOutputStream should
+ // not be closed here.
+ CloseShieldOutputStream outputStream =
+ new CloseShieldOutputStream(
+ ((OutputStreamBasedCompactingFileWriter) writer).asOutputStream());
+ doCompact(inputFiles, outputStream);
+ }
+
+ protected abstract void doCompact(List<Path> inputFiles, OutputStream outputStream)
+ throws Exception;
+
+ // A utility class to prevent the enclosing OutputStream being closed.
+ private static class CloseShieldOutputStream extends OutputStream {
Review comment:
There is already a class `CloseShieldOutputStream`, could you move this to a public position and reuse across the two positions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
* de9803e3453639e9073d6f6151575e95592621e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii closed pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii closed pull request #18680:
URL: https://github.com/apache/flink/pull/18680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
* de9803e3453639e9073d6f6151575e95592621e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
* 2ceef38711255737eba93cdb7803b41fcf87c49a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598)
* 1e885d354bb156b1598601a9fa6d06cb7972e15c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804323075
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter.java
##########
@@ -39,6 +39,32 @@
InProgressFileWriter<IN, BucketID> openNewInProgressFile(
final BucketID bucketID, final Path path, final long creationTime) throws IOException;
+ /**
+ * Used to create a new {@link CompactingFileWriter} of the requesting type. A {@link
+ * InProgressFileWriter} will be created by default, which supports only the RECORD_WISE type.
+ * Requesting a writer of an unsupported type will result in an UnsupportedOperationException.
Review comment:
I think it is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1041597793
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
* de9803e3453639e9073d6f6151575e95592621e4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de9803e3453639e9073d6f6151575e95592621e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
* d6511df39e333c9f57e2dee94504d7769c66a1a8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ceef38711255737eba93cdb7803b41fcf87c49a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598)
* 1e885d354bb156b1598601a9fa6d06cb7972e15c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e885d354bb156b1598601a9fa6d06cb7972e15c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602)
* Unknown: [CANCELED](TBD)
* 2ffd741d316346745bd8fe754bb02a73cbf0d0ba UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643",
"triggerID" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614)
* 48412828133309fc4f4cddecfedcdfcbbbee8ebf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643",
"triggerID" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1041597793",
"triggerType" : "MANUAL"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643",
"triggerID" : "1041597793",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 48412828133309fc4f4cddecfedcdfcbbbee8ebf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033509904
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 (Wed Feb 09 08:53:09 UTC 2022)
**Warnings:**
* **1 pom.xml files were touched**: Check for build and licensing issues.
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r803584223
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
+ for (Path f : compactingFiles) {
+ // cleanup compacted files
+ results.add(new FileSinkCommittable(request.getBucketId(), f));
+ }
+
+ return results;
+ }
+
+ // results: side output pass through committable
+ private List<Path> getCompactingPath(
+ CompactorRequest request, List<FileSinkCommittable> results) throws IOException {
+ List<FileSinkCommittable> compactingCommittable = request.getCommittableToCompact();
+ List<Path> compactingFiles = new ArrayList<>();
+
+ for (FileSinkCommittable committable : compactingCommittable) {
+ PendingFileRecoverable pendingFile = committable.getPendingFile();
+ if (pendingFile == null
+ || pendingFile.getPath() == null
+ || !pendingFile.getPath().getName().startsWith(".")) {
+ // the file may be written with writer of elder version, or
+ // the file will be visible once committed, so it can not be compacted.
+ // pass through, add to results, do not add to compacting files
+ results.add(committable);
+ } else {
+ // commit the pending file and compact the committed file
+ bucketWriter.recoverPendingFile(committable.getPendingFile()).commitAfterRecovery();
+ compactingFiles.add(committable.getPendingFile().getPath());
+ }
+ }
+ return compactingFiles;
+ }
+
+ private static Path assembleCompactedFilePath(Path uncompactedPath) {
+ String uncompactedName = uncompactedPath.getName();
+ if (uncompactedName.startsWith(".")) {
+ uncompactedName = uncompactedName.substring(1);
+ }
+ return new Path(uncompactedPath.getParent(), COMPACTED_PREFIX + uncompactedName);
Review comment:
Files to compact and the compacted file will both exist for a while, so the names should not conflict.
Part file path is assembled as prefix-subtaskIndex-partIndex-suffix, where partIndex is from a counter in the writer. Shall we assemble like this but add the compacted_prefix tag in somewhere between the prefix and the suffix to distinguish them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804337678
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
Review comment:
I intended to ensure that the compacted file is committed before the pending files being cleanup. But the atomicity should have been guaranteed by the committer. I'll change this to add at the tail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804460785
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/FileCompactStrategy.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.connector.file.sink.FileSink;
+
+import java.io.Serializable;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+
+/** Strategy for compacting the files written in {@link FileSink} before committing. */
+public class FileCompactStrategy implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ // Compaction triggering strategies.
+ private final long sizeThreshold;
+ private final boolean compactOnCheckpoint;
+
+ // Compaction executing strategies.
+ private final int compactThread;
+
+ private FileCompactStrategy(
+ long sizeThreshold, boolean compactOnCheckpoint, int compactThread) {
+ this.sizeThreshold = sizeThreshold;
+ this.compactOnCheckpoint = compactOnCheckpoint;
+ this.compactThread = compactThread;
+ }
+
+ public long getSizeThreshold() {
+ return sizeThreshold;
+ }
+
+ public boolean isCompactOnCheckpoint() {
Review comment:
The property is named numCheckpointsBeforeCompaction now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "cb3873dad3173b8e340d01efa9ef17168602b91a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb3873dad3173b8e340d01efa9ef17168602b91a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd437dcf44a04f54c59f8ded28ad345b4d62d620 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523)
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
* cb3873dad3173b8e340d01efa9ef17168602b91a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1034821469
@gaoyunhaii
Thanks for the suggestions.
The commits are reorgnized to be more clear, and about 2/3 comments are resolved. I'm still working on the remaining ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033636620
The branch is force pushed to add a hotfix before all commits, to fix the bug in SinkTransformationTranslator that sets parallelism of transformations added in the per-commit topology without checking if the parallelism is set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de9803e3453639e9073d6f6151575e95592621e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233)
* fd437dcf44a04f54c59f8ded28ad345b4d62d620 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de9803e3453639e9073d6f6151575e95592621e4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233)
* fd437dcf44a04f54c59f8ded28ad345b4d62d620 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 304fc4e10db023e486ad39fd191d150793c7d6c5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575)
* d6511df39e333c9f57e2dee94504d7769c66a1a8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804491562
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
+ .transform(
Review comment:
The uid is set now. Since the SinkExpander will append a user-specific prefix to all transformations added here, we only need to use the fixed uid here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
* de9803e3453639e9073d6f6151575e95592621e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1036024964
All comments so far are resolved now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd437dcf44a04f54c59f8ded28ad345b4d62d620 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523)
* 304fc4e10db023e486ad39fd191d150793c7d6c5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804458284
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactCoordinator.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactStrategy;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinator that coordinates file compaction for the {@link FileSink}.
+ *
+ * <p>All committable emitted from the writers are collected and packed into {@link
+ * CompactorRequest}s. The {@link FileCompactStrategy} defines when the requests can be fired. When
+ * a firing condition is met, the requests will be sent to the {@link CompactorOperator}.
+ *
+ * <p>The {@link CompactCoordinator} stores the non-fired committable as its state, and may emit a
+ * request at any time. A {@link CompactorOperator} must ensure that the ownership of the
+ * committable in a compact request is successfully handed from the coordinator, before it can
+ * actually perform the compaction.
+ */
+public class CompactCoordinator extends AbstractStreamOperator<CompactorRequest>
+ implements OneInputStreamOperator<
+ CommittableMessage<FileSinkCommittable>, CompactorRequest>,
+ BoundedOneInput {
+
+ private static final ListStateDescriptor<byte[]> REMAINING_COMMITTABLE_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_compact_commt_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final FileCompactStrategy strategy;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final Map<String, CompactorRequest> packing = new HashMap<>();
+ private final Map<String, CompactTrigger> triggers = new HashMap<>();
+
+ private ListState<FileSinkCommittable> remainingCommittableState;
+
+ public CompactCoordinator(
+ FileCompactStrategy strategy,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer) {
+ this.strategy = strategy;
+ this.committableSerializer = checkNotNull(committableSerializer);
+ }
+
+ @Override
+ public void processElement(StreamRecord<CommittableMessage<FileSinkCommittable>> element)
+ throws Exception {
+ CommittableMessage<FileSinkCommittable> message = element.getValue();
+ if (message instanceof CommittableWithLineage) {
+ FileSinkCommittable committable =
+ ((CommittableWithLineage<FileSinkCommittable>) element.getValue())
+ .getCommittable();
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ // or message instanceof CommittableSummary
+ // info in CommittableSummary is not necessary for compacting at present, ignore it
+ }
+
+ private boolean packAndTrigger(String bucketId, FileSinkCommittable committable) {
+ CompactorRequest bucketRequest = packing.computeIfAbsent(bucketId, CompactorRequest::new);
+ if (committable.hasInProgressFileToCleanup() || committable.hasCompactedFileToCleanup()) {
+ // cleanup request, pass through directly
+ bucketRequest.addToPassthrough(committable);
+ return false;
+ }
+
+ if (!committable.hasPendingFile()) {
+ throw new RuntimeException("Committable to compact has no content.");
+ }
+
+ CompactTrigger trigger =
+ triggers.computeIfAbsent(bucketId, id -> new CompactTrigger(strategy));
+ CompactTriggerResult triggerResult = trigger.onElement(committable);
+ switch (triggerResult) {
+ case PASS_THROUGH:
+ bucketRequest.addToPassthrough(committable);
+ return false;
+ case CONTINUE:
+ bucketRequest.addToCompact(committable);
+ return false;
+ case FIRE_AND_PURGE:
+ bucketRequest.addToCompact(committable);
+ return true;
+ default:
+ throw new RuntimeException("Unexpected trigger result:" + triggerResult);
+ }
+ }
+
+ private void fireAndPurge(String bucketId) {
+ triggers.remove(bucketId);
+ CompactorRequest request = packing.remove(bucketId);
+ if (request != null) {
+ output.collect(new StreamRecord<>(request));
+ }
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // emit all requests remained
+ for (CompactorRequest request : packing.values()) {
+ output.collect(new StreamRecord<>(request));
+ }
+ packing.clear();
+ triggers.clear();
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ super.prepareSnapshotPreBarrier(checkpointId);
+
+ // trigger on checkpoint
+ List<String> bucketsToFire = new ArrayList<>(triggers.size());
+ for (Map.Entry<String, CompactTrigger> e : triggers.entrySet()) {
+ String bucketId = e.getKey();
+ CompactTrigger trigger = e.getValue();
+ if (trigger.onCheckpoint(checkpointId) == CompactTriggerResult.FIRE_AND_PURGE) {
+ bucketsToFire.add(bucketId);
+ }
+ }
+ bucketsToFire.forEach(this::fireAndPurge);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ List<FileSinkCommittable> remainingCommittable =
+ packing.values().stream()
+ .flatMap(r -> r.getCommittableToCompact().stream())
+ .collect(Collectors.toList());
+ packing.values().stream()
+ .flatMap(r -> r.getCommittableToPassthrough().stream())
+ .forEach(remainingCommittable::add);
+ remainingCommittableState.update(remainingCommittable);
+
+ // triggers will be recomputed when restoring so it's not necessary to store
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingCommittableState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_COMMITTABLE_RAW_STATES_DESC),
+ committableSerializer);
+
+ Iterable<FileSinkCommittable> stateRemaining = remainingCommittableState.get();
+ if (stateRemaining != null) {
+ for (FileSinkCommittable committable : stateRemaining) {
+ // restore and redistribute
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ }
+ }
+
+ enum CompactTriggerResult {
+ CONTINUE,
+ FIRE_AND_PURGE,
+ PASS_THROUGH
+ }
+
+ private static class CompactTrigger {
+ private final long threshold;
+ private final boolean compactOnCheckpoint;
+
+ private long size;
+
+ CompactTrigger(FileCompactStrategy strategy) {
+ this.threshold = strategy.getSizeThreshold();
+ this.compactOnCheckpoint = strategy.isCompactOnCheckpoint();
+ }
+
+ public CompactTriggerResult onElement(FileSinkCommittable committable) {
Review comment:
Yes, that was handled in CompactorOperator previously, and is now moved here. Pending files without a path or with a path that not starts with '.' will be passed through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
* de9803e3453639e9073d6f6151575e95592621e4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643",
"triggerID" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643",
"triggerID" : "1041597793",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041597793",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614)
* 48412828133309fc4f4cddecfedcdfcbbbee8ebf Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31643)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
* 2ceef38711255737eba93cdb7803b41fcf87c49a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1041207282
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
}, {
"hash" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31602",
"triggerID" : "1e885d354bb156b1598601a9fa6d06cb7972e15c",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "1041207282",
"triggerType" : "MANUAL"
}, {
"hash" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614",
"triggerID" : "2ffd741d316346745bd8fe754bb02a73cbf0d0ba",
"triggerType" : "PUSH"
}, {
"hash" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48412828133309fc4f4cddecfedcdfcbbbee8ebf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 2ffd741d316346745bd8fe754bb02a73cbf0d0ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31614)
* 48412828133309fc4f4cddecfedcdfcbbbee8ebf UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804325406
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
+ for (Path f : compactingFiles) {
+ // cleanup compacted files
+ results.add(new FileSinkCommittable(request.getBucketId(), f));
+ }
+
+ return results;
+ }
+
+ // results: side output pass through committable
+ private List<Path> getCompactingPath(
+ CompactorRequest request, List<FileSinkCommittable> results) throws IOException {
+ List<FileSinkCommittable> compactingCommittable = request.getCommittableToCompact();
+ List<Path> compactingFiles = new ArrayList<>();
+
+ for (FileSinkCommittable committable : compactingCommittable) {
+ PendingFileRecoverable pendingFile = committable.getPendingFile();
+ if (pendingFile == null
+ || pendingFile.getPath() == null
+ || !pendingFile.getPath().getName().startsWith(".")) {
+ // the file may be written with writer of elder version, or
+ // the file will be visible once committed, so it can not be compacted.
+ // pass through, add to results, do not add to compacting files
+ results.add(committable);
+ } else {
+ // commit the pending file and compact the committed file
+ bucketWriter.recoverPendingFile(committable.getPendingFile()).commitAfterRecovery();
+ compactingFiles.add(committable.getPendingFile().getPath());
+ }
+ }
+ return compactingFiles;
+ }
+
+ private static Path assembleCompactedFilePath(Path uncompactedPath) {
+ String uncompactedName = uncompactedPath.getName();
+ if (uncompactedName.startsWith(".")) {
+ uncompactedName = uncompactedName.substring(1);
+ }
+ return new Path(uncompactedPath.getParent(), COMPACTED_PREFIX + uncompactedName);
Review comment:
Thanks @pltbkd for the explanation! If so I think we might first keep the current logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r803478421
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter.java
##########
@@ -39,6 +39,32 @@
InProgressFileWriter<IN, BucketID> openNewInProgressFile(
final BucketID bucketID, final Path path, final long creationTime) throws IOException;
+ /**
+ * Used to create a new {@link CompactingFileWriter} of the requesting type. A {@link
+ * InProgressFileWriter} will be created by default, which supports only the RECORD_WISE type.
+ * Requesting a writer of an unsupported type will result in an UnsupportedOperationException.
Review comment:
Used to create a new {@link CompactingFileWriter} of the requesting type. Requesting a writer of an unsupported type will result in UnsupportedOperationException. By default, only RECORD_WISE type is supported, for which a {@link InProgressFileWriter} will be created.
How about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r803265948
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/OutputStreamBasedCompactingFileWriter.java
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.sink.filesystem;
+
+import java.io.IOException;
+import java.io.OutputStream;
+
+/**
+ * The compactors use the {@link OutputStreamBasedCompactingFileWriter} to directly write a
+ * compacting file as an {@link OutputStream}.
+ */
+public interface OutputStreamBasedCompactingFileWriter extends CompactingFileWriter {
+ /**
+ * Get the output stream underlying the writer. The close method of the returned stream should
Review comment:
`Get` -> `Gets`
##########
File path: flink-connectors/flink-file-sink-common/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/BucketWriter.java
##########
@@ -39,6 +39,32 @@
InProgressFileWriter<IN, BucketID> openNewInProgressFile(
final BucketID bucketID, final Path path, final long creationTime) throws IOException;
+ /**
+ * Used to create a new {@link CompactingFileWriter} of the requesting type. A {@link
+ * InProgressFileWriter} will be created by default, which supports only the RECORD_WISE type.
+ * Requesting a writer of an unsupported type will result in an UnsupportedOperationException.
Review comment:
`an UnsupportedOperationException` -> `UnsupportedOperationException`?
Also I think we do not create the `InProgressFileWriter` by default, might change to `A {@link
* InProgressFileWriter} will be created by default` for RECORD_WISE type`.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactCoordinator.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactStrategy;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinator that coordinates file compaction for the {@link FileSink}.
+ *
+ * <p>All committable emitted from the writers are collected and packed into {@link
+ * CompactorRequest}s. The {@link FileCompactStrategy} defines when the requests can be fired. When
+ * a firing condition is met, the requests will be sent to the {@link CompactorOperator}.
+ *
+ * <p>The {@link CompactCoordinator} stores the non-fired committable as its state, and may emit a
+ * request at any time. A {@link CompactorOperator} must ensure that the ownership of the
+ * committable in a compact request is successfully handed from the coordinator, before it can
+ * actually perform the compaction.
+ */
+public class CompactCoordinator extends AbstractStreamOperator<CompactorRequest>
+ implements OneInputStreamOperator<
+ CommittableMessage<FileSinkCommittable>, CompactorRequest>,
+ BoundedOneInput {
+
+ private static final ListStateDescriptor<byte[]> REMAINING_COMMITTABLE_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_compact_commt_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final FileCompactStrategy strategy;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final Map<String, CompactorRequest> packing = new HashMap<>();
+ private final Map<String, CompactTrigger> triggers = new HashMap<>();
+
+ private ListState<FileSinkCommittable> remainingCommittableState;
+
+ public CompactCoordinator(
+ FileCompactStrategy strategy,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer) {
+ this.strategy = strategy;
+ this.committableSerializer = checkNotNull(committableSerializer);
+ }
+
+ @Override
+ public void processElement(StreamRecord<CommittableMessage<FileSinkCommittable>> element)
+ throws Exception {
+ CommittableMessage<FileSinkCommittable> message = element.getValue();
+ if (message instanceof CommittableWithLineage) {
+ FileSinkCommittable committable =
+ ((CommittableWithLineage<FileSinkCommittable>) element.getValue())
+ .getCommittable();
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ // or message instanceof CommittableSummary
+ // info in CommittableSummary is not necessary for compacting at present, ignore it
+ }
+
+ private boolean packAndTrigger(String bucketId, FileSinkCommittable committable) {
+ CompactorRequest bucketRequest = packing.computeIfAbsent(bucketId, CompactorRequest::new);
+ if (committable.hasInProgressFileToCleanup() || committable.hasCompactedFileToCleanup()) {
Review comment:
Perhaps first assert here that such committables must not has pendingFiles, checkState(!committable.hasPendingFIle()).
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactCoordinator.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactStrategy;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinator that coordinates file compaction for the {@link FileSink}.
+ *
+ * <p>All committable emitted from the writers are collected and packed into {@link
+ * CompactorRequest}s. The {@link FileCompactStrategy} defines when the requests can be fired. When
+ * a firing condition is met, the requests will be sent to the {@link CompactorOperator}.
+ *
+ * <p>The {@link CompactCoordinator} stores the non-fired committable as its state, and may emit a
+ * request at any time. A {@link CompactorOperator} must ensure that the ownership of the
+ * committable in a compact request is successfully handed from the coordinator, before it can
+ * actually perform the compaction.
+ */
+public class CompactCoordinator extends AbstractStreamOperator<CompactorRequest>
+ implements OneInputStreamOperator<
+ CommittableMessage<FileSinkCommittable>, CompactorRequest>,
+ BoundedOneInput {
+
+ private static final ListStateDescriptor<byte[]> REMAINING_COMMITTABLE_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_compact_commt_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final FileCompactStrategy strategy;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final Map<String, CompactorRequest> packing = new HashMap<>();
+ private final Map<String, CompactTrigger> triggers = new HashMap<>();
+
+ private ListState<FileSinkCommittable> remainingCommittableState;
+
+ public CompactCoordinator(
+ FileCompactStrategy strategy,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer) {
+ this.strategy = strategy;
+ this.committableSerializer = checkNotNull(committableSerializer);
+ }
+
+ @Override
+ public void processElement(StreamRecord<CommittableMessage<FileSinkCommittable>> element)
+ throws Exception {
+ CommittableMessage<FileSinkCommittable> message = element.getValue();
+ if (message instanceof CommittableWithLineage) {
+ FileSinkCommittable committable =
+ ((CommittableWithLineage<FileSinkCommittable>) element.getValue())
+ .getCommittable();
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ // or message instanceof CommittableSummary
+ // info in CommittableSummary is not necessary for compacting at present, ignore it
+ }
+
+ private boolean packAndTrigger(String bucketId, FileSinkCommittable committable) {
+ CompactorRequest bucketRequest = packing.computeIfAbsent(bucketId, CompactorRequest::new);
+ if (committable.hasInProgressFileToCleanup() || committable.hasCompactedFileToCleanup()) {
+ // cleanup request, pass through directly
+ bucketRequest.addToPassthrough(committable);
+ return false;
+ }
+
+ if (!committable.hasPendingFile()) {
+ throw new RuntimeException("Committable to compact has no content.");
+ }
+
+ CompactTrigger trigger =
+ triggers.computeIfAbsent(bucketId, id -> new CompactTrigger(strategy));
+ CompactTriggerResult triggerResult = trigger.onElement(committable);
+ switch (triggerResult) {
+ case PASS_THROUGH:
+ bucketRequest.addToPassthrough(committable);
+ return false;
+ case CONTINUE:
+ bucketRequest.addToCompact(committable);
+ return false;
+ case FIRE_AND_PURGE:
+ bucketRequest.addToCompact(committable);
+ return true;
+ default:
+ throw new RuntimeException("Unexpected trigger result:" + triggerResult);
+ }
+ }
+
+ private void fireAndPurge(String bucketId) {
+ triggers.remove(bucketId);
+ CompactorRequest request = packing.remove(bucketId);
+ if (request != null) {
+ output.collect(new StreamRecord<>(request));
+ }
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // emit all requests remained
+ for (CompactorRequest request : packing.values()) {
+ output.collect(new StreamRecord<>(request));
+ }
+ packing.clear();
+ triggers.clear();
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ super.prepareSnapshotPreBarrier(checkpointId);
+
+ // trigger on checkpoint
+ List<String> bucketsToFire = new ArrayList<>(triggers.size());
+ for (Map.Entry<String, CompactTrigger> e : triggers.entrySet()) {
+ String bucketId = e.getKey();
+ CompactTrigger trigger = e.getValue();
+ if (trigger.onCheckpoint(checkpointId) == CompactTriggerResult.FIRE_AND_PURGE) {
+ bucketsToFire.add(bucketId);
+ }
+ }
+ bucketsToFire.forEach(this::fireAndPurge);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ List<FileSinkCommittable> remainingCommittable =
+ packing.values().stream()
+ .flatMap(r -> r.getCommittableToCompact().stream())
+ .collect(Collectors.toList());
+ packing.values().stream()
+ .flatMap(r -> r.getCommittableToPassthrough().stream())
+ .forEach(remainingCommittable::add);
+ remainingCommittableState.update(remainingCommittable);
+
+ // triggers will be recomputed when restoring so it's not necessary to store
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingCommittableState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_COMMITTABLE_RAW_STATES_DESC),
+ committableSerializer);
+
+ Iterable<FileSinkCommittable> stateRemaining = remainingCommittableState.get();
+ if (stateRemaining != null) {
+ for (FileSinkCommittable committable : stateRemaining) {
+ // restore and redistribute
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ }
+ }
+
+ enum CompactTriggerResult {
+ CONTINUE,
+ FIRE_AND_PURGE,
+ PASS_THROUGH
+ }
+
+ private static class CompactTrigger {
+ private final long threshold;
+ private final boolean compactOnCheckpoint;
+
+ private long size;
+
+ CompactTrigger(FileCompactStrategy strategy) {
+ this.threshold = strategy.getSizeThreshold();
+ this.compactOnCheckpoint = strategy.isCompactOnCheckpoint();
+ }
+
+ public CompactTriggerResult onElement(FileSinkCommittable committable) {
Review comment:
Has we considered the case that when starting from an existing file sink some committable might not target at a temporary file? namely path does not start with '.'
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
Review comment:
Also, in this case we might not need to use the iterator directly:
```
for (Map.Entry<Long, List<CompactorRequest>> entry : canSubmit.entrySet()) {
}
canSubmit.clear();
```
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
+ .transform(
Review comment:
We would need to set the uid in someway for these operators since they have state.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
Review comment:
The default `rebalance` would be the same with `rescale` here?
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -275,6 +339,18 @@ public T withOutputFileConfig(final OutputFileConfig outputFileConfig) {
return self();
}
+ public T enableCompact(final FileCompactStrategy strategy, final FileCompactor compactor) {
+ this.compactStrategy = strategy;
Review comment:
checkNotNull, also for the following fileCompactor.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
Review comment:
This service is not shut down?
We might need to try to close the service on `close()` and `finalize()`. See `SystemProcessingTimeService`.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
+ .transform(
+ "CompactorCoordinator",
+ new CompactorRequestTypeInfo(
+ bucketsBuilder::getCommittableSerializer),
+ new CompactCoordinatorFactory(this, strategy))
+ .setParallelism(1)
+ .setMaxParallelism(1);
+
+ TypeInformation<CommittableMessage<FileSinkCommittable>> committableType =
+ committableStream.getType();
+ return coordinatorOp
+ .shuffle()
Review comment:
Do we have special reason to use shuffle? `shuffle` might incurs creating random numbers, which might cause additional cpu usage.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactCoordinator.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactStrategy;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinator that coordinates file compaction for the {@link FileSink}.
+ *
+ * <p>All committable emitted from the writers are collected and packed into {@link
+ * CompactorRequest}s. The {@link FileCompactStrategy} defines when the requests can be fired. When
+ * a firing condition is met, the requests will be sent to the {@link CompactorOperator}.
+ *
+ * <p>The {@link CompactCoordinator} stores the non-fired committable as its state, and may emit a
+ * request at any time. A {@link CompactorOperator} must ensure that the ownership of the
+ * committable in a compact request is successfully handed from the coordinator, before it can
+ * actually perform the compaction.
+ */
+public class CompactCoordinator extends AbstractStreamOperator<CompactorRequest>
+ implements OneInputStreamOperator<
+ CommittableMessage<FileSinkCommittable>, CompactorRequest>,
+ BoundedOneInput {
+
+ private static final ListStateDescriptor<byte[]> REMAINING_COMMITTABLE_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_compact_commt_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final FileCompactStrategy strategy;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final Map<String, CompactorRequest> packing = new HashMap<>();
Review comment:
`packing` sounds like a verb? Might rename to compactRequests?
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -275,6 +339,18 @@ public T withOutputFileConfig(final OutputFileConfig outputFileConfig) {
return self();
}
+ public T enableCompact(final FileCompactStrategy strategy, final FileCompactor compactor) {
+ this.compactStrategy = strategy;
+ // we always commit before compacting, so hide the file written by writer
+ this.outputFileConfig =
Review comment:
We might not directly modify the outputFileConfig here since users could still override the config after enableCompact. We might move the change to the prefix to when we create writers.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
Review comment:
I tend to not expose new methods in `FileSink`.
Is it possible to pass `SerializableSupplier<FileCompactor>` and so on for the fields required to create the `CompactorOperatorFactory` ? Or directly pass a `SerializableSupplier<CompactorOperator>`
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactCoordinator.java
##########
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactStrategy;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinator that coordinates file compaction for the {@link FileSink}.
+ *
+ * <p>All committable emitted from the writers are collected and packed into {@link
+ * CompactorRequest}s. The {@link FileCompactStrategy} defines when the requests can be fired. When
+ * a firing condition is met, the requests will be sent to the {@link CompactorOperator}.
+ *
+ * <p>The {@link CompactCoordinator} stores the non-fired committable as its state, and may emit a
+ * request at any time. A {@link CompactorOperator} must ensure that the ownership of the
+ * committable in a compact request is successfully handed from the coordinator, before it can
+ * actually perform the compaction.
+ */
+public class CompactCoordinator extends AbstractStreamOperator<CompactorRequest>
+ implements OneInputStreamOperator<
+ CommittableMessage<FileSinkCommittable>, CompactorRequest>,
+ BoundedOneInput {
+
+ private static final ListStateDescriptor<byte[]> REMAINING_COMMITTABLE_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_compact_commt_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final FileCompactStrategy strategy;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final Map<String, CompactorRequest> packing = new HashMap<>();
+ private final Map<String, CompactTrigger> triggers = new HashMap<>();
+
+ private ListState<FileSinkCommittable> remainingCommittableState;
+
+ public CompactCoordinator(
+ FileCompactStrategy strategy,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer) {
+ this.strategy = strategy;
+ this.committableSerializer = checkNotNull(committableSerializer);
+ }
+
+ @Override
+ public void processElement(StreamRecord<CommittableMessage<FileSinkCommittable>> element)
+ throws Exception {
+ CommittableMessage<FileSinkCommittable> message = element.getValue();
+ if (message instanceof CommittableWithLineage) {
+ FileSinkCommittable committable =
+ ((CommittableWithLineage<FileSinkCommittable>) element.getValue())
+ .getCommittable();
+ String bucketId = committable.getBucketId();
+ if (packAndTrigger(bucketId, committable)) {
+ fireAndPurge(bucketId);
+ }
+ }
+ // or message instanceof CommittableSummary
+ // info in CommittableSummary is not necessary for compacting at present, ignore it
+ }
+
+ private boolean packAndTrigger(String bucketId, FileSinkCommittable committable) {
Review comment:
nit: I think it might not need to pass `bukcetId` separately, it is a part of `committable`
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
Review comment:
nit: might not use part of word as variable name.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
Review comment:
Might add some comments to these collections?
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
+ for (Path f : compactingFiles) {
+ // cleanup compacted files
+ results.add(new FileSinkCommittable(request.getBucketId(), f));
+ }
+
+ return results;
+ }
+
+ // results: side output pass through committable
+ private List<Path> getCompactingPath(
+ CompactorRequest request, List<FileSinkCommittable> results) throws IOException {
+ List<FileSinkCommittable> compactingCommittable = request.getCommittableToCompact();
+ List<Path> compactingFiles = new ArrayList<>();
+
+ for (FileSinkCommittable committable : compactingCommittable) {
+ PendingFileRecoverable pendingFile = committable.getPendingFile();
+ if (pendingFile == null
+ || pendingFile.getPath() == null
+ || !pendingFile.getPath().getName().startsWith(".")) {
Review comment:
Why don't we pass-through these pending files in coordinator? Since it might cause the `CompactStrategy` not fulfilled if we consider these files here.
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/operator/CompactorOperator.java
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor.operator;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.state.CheckpointListener;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeutils.base.array.BytePrimitiveArraySerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.sink.FileSinkCommittable;
+import org.apache.flink.connector.file.sink.compactor.FileCompactor;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerialization;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.runtime.state.StateSnapshotContext;
+import org.apache.flink.runtime.util.Hardware;
+import org.apache.flink.streaming.api.connector.sink2.CommittableMessage;
+import org.apache.flink.streaming.api.connector.sink2.CommittableSummary;
+import org.apache.flink.streaming.api.connector.sink2.CommittableWithLineage;
+import org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.CompactingFileWriter;
+import org.apache.flink.streaming.api.functions.sink.filesystem.InProgressFileWriter.PendingFileRecoverable;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.util.SimpleVersionedListState;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.stream.Collectors;
+
+/**
+ * An operator that perform compaction for the {@link FileSink}.
+ *
+ * <p>Requests received from the {@link CompactCoordinator} will firstly be held in memory, and
+ * snapshot into the state of a checkpoint. When the checkpoint is successfully completed, all
+ * requests received before can be submitted. The results can be emitted at the next {@link
+ * #prepareSnapshotPreBarrier} invoking after the compaction is finished, to ensure that committers
+ * can receive only one CommittableSummary and the corresponding number of Committable for a single
+ * checkpoint.
+ */
+public class CompactorOperator
+ extends AbstractStreamOperator<CommittableMessage<FileSinkCommittable>>
+ implements OneInputStreamOperator<
+ CompactorRequest, CommittableMessage<FileSinkCommittable>>,
+ BoundedOneInput,
+ CheckpointListener {
+
+ private static final String COMPACTED_PREFIX = "compacted-";
+ private static final long SUBMITTED_ID = -1L;
+
+ private static final ListStateDescriptor<byte[]> REMAINING_REQUESTS_RAW_STATES_DESC =
+ new ListStateDescriptor<>(
+ "remaining_requests_raw_state", BytePrimitiveArraySerializer.INSTANCE);
+
+ private final int compactThreads;
+ private final SimpleVersionedSerializer<FileSinkCommittable> committableSerializer;
+
+ private final FileCompactor fileCompactor;
+ private final BucketWriter<?, String> bucketWriter;
+
+ private transient ExecutorService compactService;
+
+ private List<CompactorRequest> collectingRequests = new ArrayList<>();
+ private final TreeMap<Long, List<CompactorRequest>> snapshotRequests = new TreeMap<>();
+ private final List<Tuple2<CompactorRequest, CompletableFuture<Iterable<FileSinkCommittable>>>>
+ compactingRequests = new LinkedList<>();
+
+ private ListState<Map<Long, List<CompactorRequest>>> remainingRequestsState;
+
+ public CompactorOperator(
+ int compactThreads,
+ SimpleVersionedSerializer<FileSinkCommittable> committableSerializer,
+ FileCompactor fileCompactor,
+ BucketWriter<?, String> bucketWriter) {
+ this.compactThreads = compactThreads;
+ this.committableSerializer = committableSerializer;
+ this.fileCompactor = fileCompactor;
+ this.bucketWriter = bucketWriter;
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ this.compactService =
+ Executors.newFixedThreadPool(
+ Math.max(1, Math.min(compactThreads, Hardware.getNumberCPUCores())),
+ new ExecutorThreadFactory("compact-executor"));
+ }
+
+ @Override
+ public void processElement(StreamRecord<CompactorRequest> element) throws Exception {
+ collectingRequests.add(element.getValue());
+ }
+
+ @Override
+ public void endInput() throws Exception {
+ // add collecting requests into the final snapshot
+ snapshotRequests.put(Long.MAX_VALUE, collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // submit all requests and wait until they are done
+ submitUntil(Long.MAX_VALUE);
+ assert snapshotRequests.isEmpty();
+
+ getAllTasksFuture().join();
+ emitCompacted(null);
+ assert compactingRequests.isEmpty();
+ }
+
+ @Override
+ public void notifyCheckpointComplete(long checkpointId) throws Exception {
+ super.notifyCheckpointComplete(checkpointId);
+ submitUntil(checkpointId);
+ }
+
+ @Override
+ public void prepareSnapshotPreBarrier(long checkpointId) throws Exception {
+ emitCompacted(checkpointId);
+ }
+
+ @Override
+ public void snapshotState(StateSnapshotContext context) throws Exception {
+ super.snapshotState(context);
+
+ // add collecting requests during the checkpoint into the snapshot
+ snapshotRequests.put(context.getCheckpointId(), collectingRequests);
+ collectingRequests = new ArrayList<>();
+
+ // snapshot all compacting requests as well, including the requests that are not finished
+ // when invoking prepareSnapshotPreBarrier but finished now, since they are not emitted yet
+ Map<Long, List<CompactorRequest>> requests = new HashMap<>(snapshotRequests);
+ requests.computeIfAbsent(SUBMITTED_ID, id -> new ArrayList<>())
+ .addAll(compactingRequests.stream().map(r -> r.f0).collect(Collectors.toList()));
+ remainingRequestsState.update(Collections.singletonList(requests));
+ }
+
+ @Override
+ public void initializeState(StateInitializationContext context) throws Exception {
+ super.initializeState(context);
+
+ remainingRequestsState =
+ new SimpleVersionedListState<>(
+ context.getOperatorStateStore()
+ .getListState(REMAINING_REQUESTS_RAW_STATES_DESC),
+ new RemainingRequestsSerializer(
+ new CompactorRequestSerializer(committableSerializer)));
+
+ Iterable<Map<Long, List<CompactorRequest>>> stateRemaining = remainingRequestsState.get();
+ if (stateRemaining != null) {
+ for (Map<Long, List<CompactorRequest>> requests : stateRemaining) {
+ // elements can be more than one when redistributed after parallelism changing
+ for (Map.Entry<Long, List<CompactorRequest>> e : requests.entrySet()) {
+ List<CompactorRequest> list =
+ snapshotRequests.computeIfAbsent(e.getKey(), id -> new ArrayList<>());
+ list.addAll(e.getValue());
+ }
+ }
+ }
+ // submit all requests that is already submitted before this checkpoint
+ submitUntil(SUBMITTED_ID);
+ }
+
+ private void submitUntil(long checkpointId) {
+ NavigableMap<Long, List<CompactorRequest>> canSubmit =
+ snapshotRequests.subMap(Long.MIN_VALUE, true, checkpointId, true);
+ Iterator<Entry<Long, List<CompactorRequest>>> iter = canSubmit.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<CompactorRequest>> requestEntry = iter.next();
+ for (CompactorRequest req : requestEntry.getValue()) {
+ CompletableFuture<Iterable<FileSinkCommittable>> resultFuture =
+ new CompletableFuture<>();
+ compactingRequests.add(new Tuple2<>(req, resultFuture));
+ compactService.submit(
+ () -> {
+ try {
+ Iterable<FileSinkCommittable> result = compact(req);
+ resultFuture.complete(result);
+ } catch (Exception e) {
+ resultFuture.completeExceptionally(e);
+ }
+ });
+ }
+ iter.remove();
+ }
+ }
+
+ private Iterable<FileSinkCommittable> compact(CompactorRequest request) throws Exception {
+ List<FileSinkCommittable> results = new ArrayList<>(request.getCommittableToPassthrough());
+
+ List<Path> compactingFiles = getCompactingPath(request, results);
+ if (compactingFiles.isEmpty()) {
+ return results;
+ }
+
+ Path targetPath = assembleCompactedFilePath(compactingFiles.get(0));
+ CompactingFileWriter compactingFileWriter =
+ bucketWriter.openNewCompactingFile(
+ fileCompactor.getWriterType(),
+ request.getBucketId(),
+ targetPath,
+ System.currentTimeMillis());
+ fileCompactor.compact(compactingFiles, compactingFileWriter);
+ PendingFileRecoverable compactedPendingFile = compactingFileWriter.closeForCommit();
+
+ FileSinkCommittable compacted =
+ new FileSinkCommittable(request.getBucketId(), compactedPendingFile);
+ results.add(0, compacted);
+ for (Path f : compactingFiles) {
+ // cleanup compacted files
+ results.add(new FileSinkCommittable(request.getBucketId(), f));
+ }
+
+ return results;
+ }
+
+ // results: side output pass through committable
+ private List<Path> getCompactingPath(
+ CompactorRequest request, List<FileSinkCommittable> results) throws IOException {
+ List<FileSinkCommittable> compactingCommittable = request.getCommittableToCompact();
+ List<Path> compactingFiles = new ArrayList<>();
+
+ for (FileSinkCommittable committable : compactingCommittable) {
+ PendingFileRecoverable pendingFile = committable.getPendingFile();
+ if (pendingFile == null
+ || pendingFile.getPath() == null
+ || !pendingFile.getPath().getName().startsWith(".")) {
+ // the file may be written with writer of elder version, or
+ // the file will be visible once committed, so it can not be compacted.
+ // pass through, add to results, do not add to compacting files
+ results.add(committable);
+ } else {
+ // commit the pending file and compact the committed file
+ bucketWriter.recoverPendingFile(committable.getPendingFile()).commitAfterRecovery();
+ compactingFiles.add(committable.getPendingFile().getPath());
+ }
+ }
+ return compactingFiles;
+ }
+
+ private static Path assembleCompactedFilePath(Path uncompactedPath) {
+ String uncompactedName = uncompactedPath.getName();
+ if (uncompactedName.startsWith(".")) {
+ uncompactedName = uncompactedName.substring(1);
+ }
+ return new Path(uncompactedPath.getParent(), COMPACTED_PREFIX + uncompactedName);
Review comment:
Do we need to add the prefix? Perhaps we could keep user's outputFileNameConfig?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804373669
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/compactor/FileCompactStrategy.java
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.file.sink.compactor;
+
+import org.apache.flink.connector.file.sink.FileSink;
+
+import java.io.Serializable;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+
+/** Strategy for compacting the files written in {@link FileSink} before committing. */
+public class FileCompactStrategy implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ // Compaction triggering strategies.
+ private final long sizeThreshold;
+ private final boolean compactOnCheckpoint;
+
+ // Compaction executing strategies.
+ private final int compactThread;
+
+ private FileCompactStrategy(
+ long sizeThreshold, boolean compactOnCheckpoint, int compactThread) {
+ this.sizeThreshold = sizeThreshold;
+ this.compactOnCheckpoint = compactOnCheckpoint;
+ this.compactThread = compactThread;
+ }
+
+ public long getSizeThreshold() {
+ return sizeThreshold;
+ }
+
+ public boolean isCompactOnCheckpoint() {
Review comment:
I'd change the compactOnCheckpoint strategy to compactEveryNCheckpoints, which is equals to compactOnCheckpoint when N=1, and <=0 to disable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990)
* 90f534147b92bd17e9a653590c8cd1e98b6709aa UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e50fb694ce8cf0ae952973693bcbf17b1cb75b25 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006)
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
gaoyunhaii commented on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1041032799
Thanks @pltbkd for the update! I try to fixed some tests. Also, could you have a look that if the two `*StateHandle` needs `snapshotState` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3ecea69aa562e4be5e92a68bfb2617dde45e4e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114)
* 1327927cedcfbb3c4435e4bc5f625bee093c438b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] pltbkd commented on a change in pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
pltbkd commented on a change in pull request #18680:
URL: https://github.com/apache/flink/pull/18680#discussion_r804443649
##########
File path: flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/sink/FileSink.java
##########
@@ -177,6 +189,45 @@ private FileSink(BucketsBuilder<IN, ? extends BucketsBuilder<IN, ?>> bucketsBuil
basePath, bulkWriterFactory, new DateTimeBucketAssigner<>());
}
+ public BucketWriter<IN, String> createBucketWriter() throws IOException {
+ return bucketsBuilder.createBucketWriter();
+ }
+
+ public FileCompactor getFileCompactor() {
+ return bucketsBuilder.getFileCompactor();
+ }
+
+ @Override
+ public DataStream<CommittableMessage<FileSinkCommittable>> addPreCommitTopology(
+ DataStream<CommittableMessage<FileSinkCommittable>> committableStream) {
+ FileCompactStrategy strategy = bucketsBuilder.getCompactStrategy();
+ if (strategy == null) {
+ // not enabled
+ return committableStream;
+ }
+
+ SingleOutputStreamOperator<CompactorRequest> coordinatorOp =
+ committableStream
+ .rescale()
Review comment:
I found that we have to set a partition strategy here, or the graph will be generated with "forward". Rebalance is ok to replace rescale, of course.
The problem is, the partition transformation is already generated before expanding the sink, and the partitioner is forward because the parallelism of writer and committer are the same. Since the parallelism of coordinator is fixed to 1, which is probably not same with that of the writers, the partition transformation must be overwritten.
By the way, I found another issue relating to this. The parallelism of compactor operators and committers also have to be the same now. In the current FileSink design, committableSummary and committables are sent sequentially from the writer to the committer, that means the receiver of a sequence must be a fixed one. So the parallelism of writers must be equals to (when using forward partitioner) or more than (when using rescale partitioner) that of the committers. For the same reason, the parallelism of compactor operators is also be restricted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18680: [FLINK-25583] Support compacting small files for FileSink.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18680:
URL: https://github.com/apache/flink/pull/18680#issuecomment-1033512635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30990",
"triggerID" : "2ff898d8b0c5ba46e32e0ea26ac021cbaf29cb66",
"triggerType" : "PUSH"
}, {
"hash" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30994",
"triggerID" : "90f534147b92bd17e9a653590c8cd1e98b6709aa",
"triggerType" : "PUSH"
}, {
"hash" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31006",
"triggerID" : "e50fb694ce8cf0ae952973693bcbf17b1cb75b25",
"triggerType" : "PUSH"
}, {
"hash" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31114",
"triggerID" : "d3ecea69aa562e4be5e92a68bfb2617dde45e4e7",
"triggerType" : "PUSH"
}, {
"hash" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31224",
"triggerID" : "1327927cedcfbb3c4435e4bc5f625bee093c438b",
"triggerType" : "PUSH"
}, {
"hash" : "de9803e3453639e9073d6f6151575e95592621e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31233",
"triggerID" : "de9803e3453639e9073d6f6151575e95592621e4",
"triggerType" : "PUSH"
}, {
"hash" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31523",
"triggerID" : "fd437dcf44a04f54c59f8ded28ad345b4d62d620",
"triggerType" : "PUSH"
}, {
"hash" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31575",
"triggerID" : "304fc4e10db023e486ad39fd191d150793c7d6c5",
"triggerType" : "PUSH"
}, {
"hash" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578",
"triggerID" : "d6511df39e333c9f57e2dee94504d7769c66a1a8",
"triggerType" : "PUSH"
}, {
"hash" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598",
"triggerID" : "2ceef38711255737eba93cdb7803b41fcf87c49a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d6511df39e333c9f57e2dee94504d7769c66a1a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31578)
* 2ceef38711255737eba93cdb7803b41fcf87c49a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31598)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org